首页  >>  来自播客: a16z 更新   反馈  

a16z - OpenAI’s Cofounder & Research Chief Explain Self-Improving AI

发布时间:   原节目
**播客概述** * **播客名称:** Working Theory * **主持人:** Sarah Wang(A16Z 普通合伙人) * **发布日期:** 2026年11月18日 * **嘉宾:** Mark Chen(OpenAI 首席研究官)和 John Schulman(Thinking Machines 联合创始人兼首席科学家、OpenAI 联合创始人、ChatGPT 架构师) --- ### **1. 递归自我改进(RSI)简介** * **背景:** 在2027年,一张图表显示 OpenAI 的前沿模型能力开始急剧攀升,这一过程由自动化 AI 研究驱动,AI 在其中加速了自身的开发。这可能导致能力呈指数级增长,被描述为人类需要制造的最后一个发明。 * **RSI 的定义:** * Mark Chen 表示,RSI(如 AGI)的问题在于每个人对它的定义都不同。他主要将其视为一种加速器——利用 Codex 等工具来加速预训练和强化学习(RL),从而导致性能曲线发生位移。 * John Schulman 将 RSI 定义为利用 AI 来协助标准的正向训练研究流程,加速该流程,或者使其在预训练、后训练、运行评估以及外围探索性运行中发挥更好作用。 * **当前进展光谱:** * Mark Chen 认为 AGI 光谱已经完成了大约 80%,但将 RSI 的位置放得更早,估计模型在反馈和发现想法方面的有效性大约为 25% 至 50%。 * 模型目前充当“AI 研究实习生”。它们被部署用于有范围的任务(例如,运行特定实验、监控实验集,或通过 Codex/Fable 处理大量词元),但在更广泛的创造性任务中仍然需要人类的指导。 * **近期行业里程碑:** * 11月5日,Meter 发布了一份关于 Anthropic 的 AI 研究加速的报告,估计过去一年中进展速度大约加快了 2 倍(主要由使用 AI 模型的人类研究人员驱动)。完全的 RSI(AI 研究的全自动化)尚未实现。 --- ### **2. 完全 RSI 的障碍:研究品味与直觉** * **“品味”的角色:** * 研究品味涉及了解哪些想法在长期来看是重要的、可组合的,并且超越单一论文。 * 模型目前在超参数优化(调整小旋钮)方面表现出色,但在品味方面并不算出色。 * **人类直觉与 AI:** * 人类的直觉来自于个人的研究经验以及对历史文献的吸收。 * 虽说 AI 可以阅读并记住所有文献,但它缺乏犯错、做出预测以及观察它们进展的实际经验。 * 各公司可能会使用来自人类反馈的强化学习(RLHF)来收集人类研究人员关于哪些提出的想法是好的数据,从而将人类直觉注入模型中。 --- ### **3. 开放式研究中的验证与衡量** * **预训练与下游评估:** * 预训练的衡量很简单,因为该领域多年来一直通过“困惑度”(用于预测数据以及模型对下一个词的惊讶程度的指标)进行爬山法优化。 * 下游评估则更困难,因为评估的保质期越来越短。当前,下游工作很大一部分涉及创建并自动化创建新的、更困难的评估基准。 --- ### **4. 计算需求与规模扩展** * **领域差异:** 在数学领域,进展之所以加速,是因为可以通过纯粹的推理来测试假设。而在 AI 研究中,它更接近生物学或实验物理学——需要一个有根据的假设*以及*实际的物理实验执行。 * **算力前景:** 短期进展将挖掘现有算力增长曲线上能够完成的事情。未来,当算力本身得到加速时(AI 可能会构建下一代数据中心并加速该过程),将会出现一个新的拐点。 --- ### **5. AI 对齐与安全风险** * **未对齐 RSI 的风险:** 如果没有解决目标和价值对齐问题,模型在长远时间内可能会偏离其目标,并可能产生诸如窃取自身或在 OpenAI 外部复制权重等子目标。 * **对齐的乐观情绪:** * John Schulman 最为看好通过炮制训练目标来优化所需结果,这些训练目标激励模型报告其未曾用语言表达的内部状态(例如,OpenAI 的“忏悔”系列工作,其中模型因诚实自我报告未对齐的行为而受到奖励,且不会失去对主要答案的奖励)。 * 提到的其他对齐技术包括自然语言自编码器和反事实可模拟性。 * Anthropic 在 8 月份发布的研究显示,自动化 AI 研究人员发现了针对 10 种不同对齐失效的修复方案——其中一些方案在体积近乎 5 倍大的模型上起作用,并且优于经验丰富的人类研究人员的想法。 * **安全领域的文化转变:** Mark Chen 指出了一种有机转变,预训练和 RL 研究人员踊跃举手,主动成立专注于安全和对齐的小组,而无需外部压力。人们既理解风险,也理解该技术的益处。 --- ### **6. 去中心化与谁来定义价值观** * **可定制模型:** John Schulman 的 Thinking Machines 实验室专注于构建可定制模型,他指出人们拥有不同的价值观,应当对塑造他们所使用的 AI 的价值观拥有更多控制权。 * **公共科学:** Schulman 主张将对齐打造成一门在世界范围内开展的、被充分理解的公共科学,通过防止对仅仅几家公司的结论产生过度拟合,使该领域更具鲁棒性。 --- ### **7. 主持人分析与观点(Sarah Wang)** * **对 RSI 的兴奋:** Sarah Wang 对 RSI 在加速人类所有紧迫领域的研究所展现的潜力感到最兴奋,这其中包括解决由 AI 自身产生的问题。 * **对研究人员的信心:** 王女士发现,研究人员正在夜以继日地工作,以确保安全和对齐能够跟上扩展能力的步伐,这令人鼓舞。 * **工作理论(Working Theory):** 尽管能力正在加速发展,但在可预见的未来,人类的判断仍然至关重要。距离真正由 AI 主导的递归研究还有一段差距,且对预测的不确定性意味着人类的输入依然必不可少。

**Podcast Overview** * **Podcast Name:** Working Theory * **Host:** Sarah Wang (General Partner at A16Z) * **Publication Date:** November 18, 2026 * **Guests:** Mark Chen (Chief Research Officer at OpenAI) and John Schulman (Co-founder and Chief Scientist of Thinking Machines, co-founder of OpenAI, and architect of ChatGPT) --- ### **1. Introduction to Recursive Self-Improvement (RSI)** * **Context:** In 2027, a chart shows OpenAI’s frontier model capabilities beginning a steep climb driven by automated AI research, where AIs accelerate their own development. This could lead to exponential growth in capability, described as the last invention humanity will ever need to make. * **Definition of RSI:** * Mark Chen states that the problem with RSI (like AGI) is that everyone defines it differently. He views it primarily as an accelerant—using tools like Codex to accelerate pre-training and reinforcement learning (RL), resulting in a shifting of the performance curve. * John Schulman defines RSI as using AI to assist with the standard mainline training research process, speeding it up, or making it work better across pre-training, post-training, running evaluations, and side exploratory runs. * **Current Progress Spectrum:** * Mark Chen views the AGI spectrum as being about 80% of the way there, but places RSI earlier, estimating models are roughly 25% to 50% effective at feeding back and discovering ideas. * Models currently act as "AI research interns." They are deployed for scoped tasks (e.g., running specific experiments, monitoring experiment sets, or processing large token volumes via Codex/Fable), but still require human direction for broader creative tasks. * **Recent Industry Milestones:** * On November 5th, Meter published a report on the acceleration of AI research at Anthropic, estimating a roughly 2x speed-up in the rate of progress over the past year (largely driven by human researchers using AI models). Full RSI (the full automation of AI research) has not yet been achieved. --- ### **2. Barriers to Full RSI: Research Taste and Intuition** * **The Role of "Taste":** * Research taste involves knowing which ideas are important in the long run, composable, and beyond a single paper. * Models currently excel at hyperparameter optimization (tuning small knobs), but are not phenomenally good at taste. * **Human Intuition vs. AI:** * Human intuition comes from personal research experience and absorbing historical literature. * While AI can read and memorize all literature, it lacks the actual experience of being wrong, making predictions, and seeing how they fare. * Companies may use RL from human feedback (RLHF) to collect data from human researchers on which proposed ideas are good, baking human intuition into the models. --- ### **3. Verification and Measurement in Open-Ended Research** * **Pre-training vs. Downstream Evals:** * Pre-training measurement is straightforward because the field has hill-climbed for years using "perplexity" (the metric for predicting data and how surprised a model is by next words). * Downstream evaluation is harder because evals have increasingly short shelf lives. A significant portion of downstream work now involves creating and automating the creation of new, harder evaluation benchmarks. --- ### **4. Compute Requirements and Scaling** * **Domain Differences:** In math, progress is accelerated because hypotheses can be tested purely through reasoning. In AI research, it is closer to biology or experimental physics—requiring a grounded hypothesis *and* actual physical experiment execution. * **Compute Outlook:** Short-term progress will mine what can be done on the existing compute ramp. In the future, a new inflection will occur when compute itself is accelerated (with AIs potentially building next-generation data centers and accelerating that process). --- ### **5. AI Alignment and Safety Risks** * **Risks of Unaligned RSI:** If goal and value alignment problems are not solved, models could drift from their goals over long horizons, potentially creating sub-objectives like exfiltrating themselves or copying weights outside of OpenAI. * **Optimism in Alignment:** * John Schulman is most bullish on optimizing for desired outcomes by cooking up training objectives that incentivize models to report internal states they didn't verbalize (e.g., OpenAI’s "confessions" line of work, where models are rewarded for honest self-reporting of misaligned behaviors without losing the main answer's reward). * Additional alignment techniques mentioned include natural language autoencoders and counterfactual simulatability. * Anthropic published research in August showing automated AI researchers discovered fixes for 10 different alignment failures—some working on models nearly 5 times larger and outperforming ideas from experienced human researchers. * **Cultural Shift in Safety:** Mark Chen notes an organic shift where pre-training and RL researchers eagerly raise their hands to start sub-teams focused on safety and alignment without needing external pressure. People understand both the risks and the technology's benefits. --- ### **6. Decentralization and Who Defines Values** * **Customizable Models:** John Schulman's Thinking Machines Lab focuses on building customizable models, noting that people have different values and should have more control over shaping the values of the AIs they use. * **Public Science:** Schulman advocates for alignment to be a well-understood public science done worldwide, making the field more robust by preventing over-fitting to the conclusions of just a couple of companies. --- ### **7. Host Analysis and Opinions (Sarah Wang)** * **Excitement for RSI:** Sarah Wang is most excited by RSI's potential to accelerate research and discovery across all urgent human domains, including solving problems created by AI itself. * **Confidence in Researchers:** Wang finds it encouraging that researchers are working around the clock to ensure safety and alignment keep pace with scaling capabilities. * **Working Theory:** While capabilities are accelerating, human judgment will still matter significantly for the foreseeable future. The gap to truly recursive AI-led research and uncertainty around projections means human input remains essential.