How Scaling Laws Will Determine AI's Future | YC Decoded

这段视频的文字稿探讨了大型语言模型 (LLM) 中规模定律的演变和潜在未来。它首先突出了 YC Spring Batch 的吸引力，这证明了人工智能的力量和前景，然后深入探讨了核心问题：仅仅扩大 LLM 的规模是否已经达到了极限，或者一种新的范式正在出现？ 视频回顾了 LLM 扩展的历史，从 2019 年 OpenAI 的 GPT-2 和 2020 年具有突破性的 GPT-3 开始。GPT-3 的规模是其前身 GPT-2 的 100 多倍，它表明增加模型大小、数据和计算能力可以显著提高性能，从而开启了规模定律的时代。Jared Kaplan 和 OpenAI 的同事在一篇论文中正式提出了这个概念，该论文显示，当这三个因素增加时，模型性能会遵循幂律平稳、持续地提高。性能变得更依赖于规模，而不是算法创新。随后的研究证实了这些规模定律在各种模态中的有效性，包括文本到图像和图像到文本任务。 视频赞扬了匿名研究员 "Wern" 将规模定律带入主流，他提倡这样一种观点，即智能可能仅仅是大量计算、数据和参数的产物。这种观念迅速转变为人工智能开发中的一项基本原则。 然而，叙事在 Google DeepMind 于 2022 年的研究中发生了微妙的转变。DeepMind 的研究强调了在足够的数据上训练模型的重要性，而不仅仅是增加其大小。他们的研究表明，像 GPT-3 这样的模型可能训练不足。这导致了 Chinchilla 的开发，Chinchilla 是一个比 GPT-3 小的 LLM，但使用四倍以上的数据进行训练。Chinchilla 的表现优于更大的模型，表明优化模型大小和训练数据之间的平衡至关重要。Chinchilla 规模定律极大地影响了当前前沿模型（如 GPT-4 和 Cloud 3.5 Sonnet）的开发。 视频随后提出了一个关键问题：人工智能的未来是否仅仅是构建越来越大的模型？人工智能界最近的争论表明，规模定律可能正在接近其极限。一些人认为，最新一代的模型虽然更大、更昂贵，但在能力方面却表现出收益递减的趋势。有传言称训练运行失败，并担心缺乏高质量的数据来训练新模型，这可能会导致瓶颈。 考虑到传统扩展方式可能存在的局限性，视频探索了一个潜在的新领域：OpenAI 的推理模型类别，以 O1 及其后续版本 O3 为例。这些模型表明了一种转变，即在模型的思维链过程中扩展可用的计算量，即所谓的“测试时计算”。通过让模型思考更长时间，LLM 可以按需利用更多的计算资源，从而提高它们在复杂问题上的表现。O3 在基准测试中表现出色，超过了软件工程和科学等领域先前的最先进结果，这证明了这种新方法的潜力。 虽然预训练的扩展可能正在趋于平缓，但视频认为，扩展测试时计算可能会释放全新的能力，从而可能为实现通用人工智能 (AGI) 铺平道路。视频最后强调，规模定律的原理不仅限于语言模型，还影响着图像扩散模型、蛋白质折叠、化学模型和机器人技术等领域。视频表明，虽然 LLM 可能正在进入“中场”阶段，但其他模态的扩展仍处于早期阶段，这预示着人工智能领域将持续存在兴奋和创新。

This video transcript explores the evolution and potential future of scaling laws in large language models (LLMs). It begins by highlighting the allure of the YC Spring Batch, a testament to the power and promise of AI, before diving into the central question: has the era of simply scaling up LLMs reached its limit, or is a new paradigm emerging? The video traces the history of scaling LLMs, starting with OpenAI's GPT-2 in 2019 and the groundbreaking GPT-3 in 2020. GPT-3, over 100 times larger than its predecessor, demonstrated that increasing model size, data, and compute significantly improved performance, ushering in the era of scaling laws. This concept was formalized in a paper by Jared Kaplan and colleagues at OpenAI, which showed a smooth, consistent improvement in model performance following a power law when these three factors were increased. Performance became more dependent on scale than on algorithmic innovation. Subsequent research confirmed the validity of these scaling laws across various modalities, including text-to-image and image-to-text tasks. The video credits the anonymous researcher "Wern" for bringing scaling laws to the mainstream, advocating the idea that intelligence might simply be a product of significant compute, data, and parameters. This notion quickly transformed into a fundamental principle in AI development. However, the narrative takes a nuanced turn with Google DeepMind's research in 2022. DeepMind's findings highlighted the importance of training models on sufficient data, not just increasing their size. Their research revealed that models like GPT-3 were likely undertrained. This led to the development of Chinchilla, an LLM smaller than GPT-3 but trained on four times more data. Chinchilla outperformed larger models, demonstrating that optimizing the balance between model size and training data was crucial. The Chinchilla scaling laws significantly influenced the development of current frontier models like GPT-4 and Cloud 3.5 Sonnet. The video then poses a critical question: is the future of AI simply about building increasingly larger models? Recent debates within the AI community suggest the possibility that scaling laws may be approaching their limits. Some argue that the latest generation of models, while larger and more expensive, are demonstrating diminishing returns in terms of capabilities. There are rumors of failed training runs and concerns about the lack of high-quality data to train new models, potentially leading to a bottleneck. Considering the possible limitations of traditional scaling, the video explores a potential new frontier: OpenAI's class of reasoning models, exemplified by O1 and its successor, O3. These models suggest a shift toward scaling the amount of compute available to the model during its chain of thought process, known as "test time compute." By allowing models to think for longer, LLMs can leverage more compute on demand, improving their performance on complex problems. O3's impressive performance on benchmarks, surpassing previous state-of-the-art results in areas like software engineering and science, demonstrates the potential of this new approach. While scaling pre-training may be plateauing, the video argues that scaling test time compute could unlock entirely new capabilities, potentially paving the way toward artificial general intelligence (AGI). The video concludes by emphasizing that the principles of scaling extend beyond language models, influencing areas such as image diffusion models, protein folding, chemical models, and robotics. The video suggests that while LLMs may be entering a "mid-game" phase, scaling other modalities is still in its early stages, signaling continued excitement and innovation in the field of artificial intelligence.