How Harvey Built a Research Lab on a Budget | Gabe Pereyra

以下是内容的中文翻译： Harvey 的联合创始人兼总裁 Gabe 分享了他们公司如何建立一个强大的研究实验室的见解，有效地展示了应用层公司如何在预算有限的情况下，通过利用更广泛的AI生态系统，与前沿AI实验室竞争。他题为《以有限预算建立研究实验室》的演讲，强调了在没有大型研究机构庞大资源的情况下，开发前沿智能的关键策略。 Gabe 首先阐述了应用层公司面临的固有挑战：与拥有更优越资金、人才、算力、基础设施和数据的“富裕团队”竞争。Harvey 的方法论诞生于许多基础AI公司刚刚起步的环境中，他们专注于产品和市场推广。然而，他认为，如今成熟的前沿生态系统使得应用公司能够建立有竞争力的智能。 Harvey 策略的第一步也是关键一步是**建立基准**。如果没有强大的基准，有效训练和部署模型将是不可能的。Harvey 已经开源了三个重要的法律数据集： 1. **法律代理基准 (Legal Agent Bench)**：一个针对大型律师事务所律师的复杂任务分类体系，涵盖了基金设立文件起草、判例法研究等多样化的执业领域。 2. **合同数据集 (Contracting Data Set)**：旨在教导智能体为企业内部法务部门进行谈判。 3. **大型尽职调查数据集 (Large Diligence Data Set)**：最令人兴奋的一个，包含高达8000万token的数据室，允许研究长上下文、高度复杂的任务。 Harvey 面临的一个主要挑战是，由于处理来自顶级律师事务所高度敏感和受保护的法律数据，他们无法在客户数据上进行训练。他们的创新解决方案涉及**由领域专家指导合成数据生成**。例如，Gabe 的律师兄弟学会了使用AI模型进行“氛围编码”，以创建极其逼真的法律数据集。这个过程始于定义一个问题细则和场景，然后围绕这些预设问题生成数据室。Mercore 和 Snorkel 等工具随后帮助扩展此过程，并使生成的合同看起来真实。这些合成数据集解决了“先有鸡还是先有蛋”的问题，即需要数据来证明模型能力，又需要证明才能获取数据。 这些数据集随后被转化为高效的强化学习（RL）环境，通常与 LangChain 等工具集成，以管理高昂的评估成本，特别是在处理大型模型和数千个单元测试时。开源这些数据集是一个有争议但具有战略意义的举动，它邀请社区参与、发现问题，并鼓励实验室在 Harvey 的数据上进行基准测试，Gabe 认为这为他们带来了建议、拉取请求，甚至得到了埃隆·马斯克的转发。 接下来，Gabe 讨论了**模型后训练**。随着 Kim E3、GLM 5.2、Nemotron 和 Inkling 等开源模型达到有竞争力的水平，后训练已成为一种可行的策略。Harvey 与提供专业知识和基础设施的“新实验室”（例如 Fireworks、Base10、Graham、Applied Compute、Trajectory）合作。这种多供应商方法使 Harvey 能够探索多样化的研究方向，训练各种开源模型，并获取有价值的见解，同时补充他们日益增长的内部后训练工作，例如使用 Tinker 等 API。目标是将这些工作打包进 Harvey 专有的‘Composer One’模型。 **在生产环境中部署模型**是一个非同小可但至关重要的一步。Harvey 在60个国家运营，拥有多个产品领域和不同的客户模型偏好，这需要一个复杂的模型部署基础设施。这个复杂体系包括处理跨供应商的故障回退机制以满足服务水平协议（SLA），现在也扩展到包括开源模型。 在任何模型（开源、闭源或后训练模型）投入生产之前，都需经过严格的**预生产评估**： * **自动化**：实验室基准评估前沿状态和法律领域实力。 * **人工测试**：特定于产品界面的并排比较和关键用户旅程测试。 * **启发式评估**：成本、延迟、区域和可用性。 一旦投入生产，持续的**监控**至关重要，包括 A/B 测试、参与度跟踪、正常运行时间、token 效率以及直接产品反馈（“愤怒的客户邮件”）。Gabe 强调，这个基础设施在考虑后训练之前就至关重要，因为它提供了关键的反馈循环。 他还建议从“简单的开源切换”开始，识别产品中可以简单地用开源模型替换以获得成本或性能优势的部分（例如，引文生成），或者为特定查询实施模型路由。这培养了管理开源模型和专有模型的技能，最终推动后训练的飞轮效应。 Gabe 最后重申，每家公司都将成为一家AI公司，而 Harvey 的策略提供了一条“改变游戏规则”的道路。他将此与《点球成金》（Moneyball）进行了类比，在这部电影中，以有限预算和独特团队取得胜利最终改变了这项运动。 在问答环节中，Gabe 进一步阐述了几点： * **合成数据生成**：他兄弟的方法是将问题植入细则中，然后生成数据室，从而有效验证模型是否能发现这些预设问题。 * **招聘**：虽然最初难以与顶级实验室竞争，但生态系统的成熟和专业API的可用性降低了对深层基础设施人才的需求，从而扩大了合适研究人员的范围。 * **基准哲学**：开源在社区参与和战略差异化之间取得了平衡。真正的长期价值在于帮助律师事务所训练*他们自己*的私有敏感数据。 * **未决问题**：主要挑战包括：在不使用客户数据的情况下弥合合成数据与生产数据之间的分布差距；提高模型在长上下文复杂任务（例如8000万token数据室）上的表现；以及为企业客户开发强大的持续学习机制，同时保护数据隐私。 * **与通用产品竞争**：Harvey 的策略从个人生产力（如 Codex Co-work）转向大型律师事务所的**组织生产力**，通过在数千个客户项目中协调人类和AI智能体，有效地在法律领域实现“超垂直化”。 Gabe 的演讲为旨在构建复杂AI能力的应用层公司提供了一份务实且鼓舞人心的路线图，强调了战略性地利用前沿生态系统、创新的数据生成方法和强大的运营基础设施。

Gabe, co-founder and president of Harvey, shared insights on how their company built a robust research lab, effectively demonstrating how application-layer companies can compete with frontier AI labs on a budget by leveraging the broader AI ecosystem. His talk, aptly titled "Building a Research Lab on a Budget," highlighted key strategies for developing frontier intelligence without the vast resources of major research institutions. Gabe began by outlining the inherent challenges faced by application-layer companies: competing with "rich teams" possessing superior money, talent, compute, infrastructure, and data. Harvey's approach, born out of a landscape where many foundational AI companies were nascent, was to focus on product and go-to-market. Today, however, he argued that the mature frontier ecosystem enables application companies to build competitive intelligence. The first critical step in Harvey's playbook is **building benchmarks**. Without robust benchmarks, training and serving models effectively is impossible. Harvey has open-sourced three significant legal datasets: 1. **Legal Agent Bench:** A taxonomy of complex tasks for associates at large law firms, covering diverse practice areas like drafting fund formation documents and case law research. 2. **Contracting Data Set:** Designed to teach agents negotiation skills for in-house legal departments. 3. **Large Diligence Data Set:** The most exciting, featuring data rooms up to 80 million tokens, allowing research into long-context, highly complex tasks. A major challenge for Harvey, working with highly sensitive and privileged legal data from top law firms, is the inability to train on customer data. Their innovative solution involves **domain experts guiding synthetic data generation**. Gabe's lawyer brother, for instance, learned to "vibe code" using AI models to create incredibly realistic legal datasets. This process starts with defining a rubric of issues and a scenario, then generating the data room around these planted issues. Tools like Mercore and Snorkel then help scale this process and make the generated contracts look realistic. These synthetic datasets solve the "chicken and egg" problem of needing data to prove model capability and needing proof to get data. These datasets are then turned into efficient RL environments, often integrated with tools like LangChain, to manage the high cost of evaluation, especially with large models and thousands of unit tests. Open-sourcing these datasets was a controversial but strategic move, inviting community input, identifying issues, and encouraging labs to benchmark on Harvey's data, which Gabe credits for receiving suggestions, pull requests, and even a retweet from Elon Musk. Next, Gabe discussed **post-training models**. With open-source models like Kim E3, GLM 5.2, Nemotron, and Inkling reaching competitive levels, post-training has become a viable strategy. Harvey collaborates with "neo labs" (e.g., Fireworks, Base10, Graham, Applied Compute, Trajectory) which offer expertise and infrastructure. This multi-vendor approach allows Harvey to explore diverse research bets, train various open-source models, and gain valuable insights, complementing their increasing internal post-training efforts using APIs like Tinker. The goal is to package this work into Harvey's proprietary "Composer One" model. **Serving models in production** is a non-trivial but crucial step. Harvey operates in 60 countries with multiple product areas and varying customer model preferences, necessitating a complex model serving infrastructure. This matrix includes handling fallback mechanisms across providers to meet SLAs, now extended to include open-source models. Before any model (open-source, closed-source, or post-trained) goes into production, it undergoes rigorous **pre-production evaluation**: * **Automated:** Lab benchmarks assess frontier status and legal domain strength. * **Human Testing:** Side-by-side comparisons and critical user journey tests specific to product surfaces. * **Heuristics:** Cost, latency, region, and availability. Once in production, continuous **monitoring** is vital, using A/B testing, engagement tracking, uptime, token efficiency, and direct product feedback ("angry customer emails"). Gabe stressed that this infrastructure is essential *before* even considering post-training, as it provides the critical feedback loop. He also recommended starting with "simple open-source switches" by identifying parts of the product where open-source models can be naively swapped for cost or performance benefits (e.g., citation generation), or by implementing model routing for specific queries. This builds the muscle for managing open-source models alongside proprietary ones, ultimately feeding into the post-training flywheel. Gabe concluded by reiterating that every company will become an AI company, and Harvey's playbook offers a path to "change the game." He drew an analogy to Moneyball, where winning on a budget with a unique team ultimately transforms the sport. During the Q&A, Gabe expanded on several points: * **Synthetic Data Generation:** His brother's method involved planting issues into a rubric and then generating the data room, allowing for effective validation of whether a model catches those planted problems. * **Hiring:** While initially difficult to compete with top labs, the maturation of the ecosystem and availability of specialized APIs reduce the need for deep infrastructure talent, broadening the pool of suitable researchers. * **Benchmark Philosophy:** Open-sourcing creates a balance between community engagement and strategic differentiation. The true long-term value lies in helping law firms train on *their* private, sensitive data. * **Remaining Open Questions:** Key challenges include bridging the distribution gap between synthetic and production data without using customer data, improving model performance on long-context complex tasks (like 80M token data rooms), and developing robust continual learning mechanisms for enterprise clients while preserving data privacy. * **Competing with General Products:** Harvey's strategy shifts from individual productivity (like Codex Co-work) to **organizational productivity** for large law firms, orchestrating humans and AI agents across thousands of client projects, effectively "hyper-verticalizing" into the legal domain. Gabe's talk provided a pragmatic and inspiring roadmap for application-layer companies aiming to build sophisticated AI capabilities, emphasizing strategic leveraging of the frontier ecosystem, innovative data generation, and robust operational infrastructure.