LangChain联合创始人兼首席执行官Harrison（哈里森）发表了一场演讲，重点阐述了AI代理的“编排层”（harnesses）和“评估”（evals）在实现AI“拥有自己的智能”方面的关键作用。他将代理定义为由模型、上下文和编排层三个主要部分组成，强调了控制这三者的重要性。 **编排层**至关重要，它负责编排并在恰当的时机向模型提供正确的上下文。其核心是，代理架构涉及一个循环运行的大型语言模型（LLM），在循环中生成内容、调用工具并将观察结果处理回循环。这种看似简单的结构具有高度可定制性。编排层可以通过“中间件”、“钩子”或“插件”进行扩展，这些扩展可以在模型和工具调用之前或之后修改行为。例如，集成沙箱、访问文件系统、管理子代理、处理内存、执行摘要或卸载上下文。对于高度特定或可预测的任务，可以构建更明确的、具有定义好顺序步骤的“认知架构”。 关于**何时构建自定义编排层以及何时使用现成解决方案**，Harrison建议，为快速获得初步价值，可以从通用编排层（如Deep Agents、Claude Code或Codex）开始。当代理任务偏离模型训练数据分布越远时，定制化就变得至关重要。即使在像编码这样的通用任务中，不同的模型实验室（例如OpenAI和Anthropic）可能针对不同的文件编辑方法优化了其模型，这就需要在自定义编排层中包含“模型配置文件”以进行适应。 **评估与可观测性**对于代理的持续改进至关重要。Harrison引用了Satiya的见解，强调自定义评估能够定义组织成功，保留机构知识，并建立一个“持续学习循环”，从而叠加AI投资的价值。 **评估**涉及为关键任务代理构建基准测试，以发现回归并推动改进。Harbor作为一个开源评估运行器，正逐渐成为行业标准。它定义了任务，每个任务都有自己的环境（通常是Docker化），一个黄金解决方案，灵活的“测试脚本”（可以是代码、单元测试，甚至可以是作为评判者的LLM），以及清晰的说明。像LangSmith这样的平台允许用户可视化和比较不同代理编排层和模型之间的评估结果，不仅追踪准确性，还追踪延迟和成本。 **可观测性**对于调试代理至关重要，因为故障往往源于上下文不足或管理不善，而非模型本身的固有弱点。有效的可观测性工具必须揭示哪些内容进入了模型的上下文窗口，上下文是如何累积的，以及哪些步骤和工具被调用。LangSmith提供了高层次的“轨迹”视图（如对话轮次）和详细的“追踪”视图，用于深度调试。 这些要素共同构成了**持续改进的数据飞轮**：构建代理、运行代理、收集追踪数据、整理追踪数据（通过用户反馈或微调的SLM、LLM作为评判者等合成评估器），然后运行实验。这个迭代过程允许更新代理的三个核心组件中的任何一个：编排层（编排层工程）、模型（微调）或上下文（内存管理）。 Harrison展示了**LangSmith Engine**，一个体现该数据飞轮的自动化代理。Engine分析追踪数据，识别问题，生成“问题看板”，并提出针对编排层改进的具体代码变更建议，旨在自动化迭代优化过程。 总而言之，Harrison强调，尽管通用编排层是一个很好的起点，但随着任务特异性和可预测性需求的增加，对自定义编排层的需求也会增长。评估和可观测性不仅是工具，更是基础实践，使组织能够驾驭不断演变的AI格局，理解代理性能，并通过系统改进持续叠加其AI投资。

Harrison, co-founder and CEO of LangChain, delivered a presentation focusing on the critical roles of AI agent "harnesses" and "evaluations" (evals) in achieving "owning your own intelligence" with AI. He defined an agent as comprising three main parts: the model, the context, and the harness, stressing the importance of controlling all three. The **harness** is paramount, serving to orchestrate and bring the right context to the model at the right time. At its core, an agent architecture involves a Large Language Model (LLM) running in a loop, making generations, calling tools, and processing observations back into the loop. This seemingly simple structure is highly customizable. Harnesses can be extended through "middleware," "hooks," or "plugins" that modify behavior before or after model and tool calls. Examples include integrating a sandbox, accessing file systems, managing sub-agents, handling memory, performing summarization, or offloading context. For highly specific or predictable tasks, more explicit "cognitive architectures" with defined, sequential steps can be built. Regarding **when to build a custom harness versus using an off-the-shelf solution**, Harrison advised starting with general harnesses (like Deep Agents, Claude Code, or Codex) for quick initial value. Customization becomes essential as an agent's task deviates further from the model's training distribution. Even within general tasks like coding, different model labs (e.g., OpenAI and Anthropic) might have optimized their models for distinct methods of file editing, necessitating "model profiles" in custom harnesses to adapt. **Evaluations and Observability** are crucial for continuous agent improvement. Harrison cited Satiya's insights, highlighting that custom evals define organizational success, preserve institutional knowledge, and establish a "continuous learning loop" that compounds AI investments. **Evals** involve building benchmarks for mission-critical agents to catch regressions and drive improvement. Harbor, an open-source eval runner, is emerging as an industry standard. It defines tasks, each with its own environment (often Dockerized), a golden solution, flexible "test scripts" (which can be code, unit tests, or even LLMs as judges), and clear instructions. Platforms like LangSmith allow users to visualize and compare eval results across different agent harnesses and models, tracking not only accuracy but also latency and cost. **Observability** is vital for debugging agents, as failures often stem from insufficient or poorly managed context rather than inherent model weaknesses. Effective observability tools must reveal what enters the model's context window, how context accumulates, and which steps and tools are invoked. LangSmith provides both high-level "trajectory" views (like conversational turns) and detailed "trace" views for in-depth debugging. These elements feed into a **data flywheel for continuous improvement**: build an agent, run it, collect traces, curate the trace data (via user feedback or synthetic evaluators like fine-tuned SLMs or LLM-as-a-judge), and then run experiments. This iterative process allows for updates to any of the three core agent components: the harness (harness engineering), the model (fine-tuning), or the context (memory management). Harrison demonstrated **LangSmith Engine**, an automated agent that embodies this flywheel. Engine analyzes traces, identifies issues, generates an "issue board," and suggests specific code changes for harness improvements, aiming to automate the iterative refinement process. In summary, Harrison emphasized that while general-purpose harnesses are a good starting point, the need for custom harnesses grows with task specificity and the demand for predictability. Evals and observability are not just tools but foundational practices that enable organizations to navigate the evolving AI landscape, understand agent performance, and continuously compound their AI investments through systematic improvement.