Ex-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaper

以下是内容的中文翻译： Sail Research 致力于使AI代币“尽可能便宜”，他们认为智能，如同任何商品，都应丰富且易于获取。他们的使命是构建一个“代币工厂”，即一个能以“无与伦比的价格”为任何任务提供开源大型语言模型服务的API。他们还托管“Sail盒子”，这是一种长时间运行的代理虚拟机，专为持续数小时、数天乃至数周的任务而设计，这使他们与专注于低延迟、实时推理的竞争对手区分开来。 该公司的核心战略围绕着通过解决计算堆栈（包括软件、硬件和电力）每一层的代币成本问题，从而实现“丰裕”。 **软件优化：** Sail Research 首先通过优化软件来提高GPU的峰值效率。这涉及到深入理解并定制内核（即底层GPU程序），以最大限度地提升同一芯片的“每秒代币”输出量。与行业普遍追求低延迟、交互式AI（如聊天机器人）的趋势不同，Sail Research 的技术栈是围绕面向主动式后台代理的*吞吐量*而构建的。这是因为长时间运行的任务并不优先考虑即时响应；相反，它们受益于高整体处理量，即使单个代币生成速度较慢。他们将此比作公共汽车（吞吐量）与私家车（延迟）之间的区别——公共汽车整体服务更多人，尽管每个人的行程时间更长。 **硬件策略：** 该公司在硬件方面采取“寻宝式”策略，寻找能够提供最佳“每美元浮点运算次数”的芯片，而不是仅仅依赖于尖端、高需求的英伟达GPU。他们乐于使用AMD等多种架构，或Cerebras和Grok等公司的专用加速器。这些替代芯片可能不擅长低延迟通信（如英伟达的NVLink），但在原始计算方面却效率极高。Sail Research 的专长在于开发定制的并行方案和软件，以最大限度地发挥这些非优化芯片的性能。他们承认KV缓存（对话历史内存）带来的挑战，它可能比模型权重更大，并且对于专用、高SRAM芯片来说难以有效管理。他们建议采用混合方法，同时使用高速片上内存和高容量片外内存。 **数据中心和电力：** Sail Research 的硬件策略也延伸到数据中心和电力方面。鉴于在美国建设大型、数千兆瓦级数据中心日益增长的难度，他们采取分布式“迷你工厂”（mini-mills）的方法。这意味着在全国范围内利用许多小型、1兆瓦的数据中心。他们愿意容忍较低的正常运行时间（例如95%甚至80%），并利用太阳能和风能等间歇性可再生能源。对于后台代理任务而言，偶尔的停机是可以接受的，因为工作可以重新安排或转移到其他位置，而不会实时中断人类用户的操作。这使得他们能够获取更便宜的电力和他人避之不及的数据中心位置，从而显著降低运营成本。 **市场和未来愿景：** 该公司看到了从交互式AI向长周期、主动式代理转变的巨大机遇。他们相信AI可以解决“可验证的问题”，例如深度研究（索引1万多个来源）、网络安全（主动打补丁），甚至是主动式个人助理（一个能预测需求的“Siri”）。他们认为，AI数据的未来并非来自更多的互联网抓取，而是来自受控强化学习（RL）环境中的“模型自我改进”，在这些环境中，模型处理可验证的任务并生成自身的高质量反馈。 Sail Research 的独特观点包括将KV缓存卸载到闪存的潜力，以及坚信尽管当前代币成本较高，但对智能的需求是“无限的”。他们认为，AI技术栈中最大的低效率存在于KV缓存压缩和计算编排中，因为许多GPU在私有的、未被利用的资源池中处于闲置状态。通过解决这些低效率问题，他们旨在大幅降低智能的成本，使其可用于无数新的应用和用户。

Sail Research aims to make AI tokens "as cheap as humanly possible," believing that intelligence, like any commodity, should be abundant and accessible. Their mission is to build a "token factory," an API that serves open-source large language models for any task at an "unbeatable price." They also host "Sail boxes," long-running agent virtual machines designed for tasks that span hours, days, or weeks, distinguishing themselves from competitors focused on low-latency, real-time inference. The company's core strategy revolves around enabling "abundance" by tackling token costs across every layer of the computing stack: software, hardware, and power. **Software Optimization:** Sail Research begins by optimizing software for peak GPU efficiency. This involves deeply understanding and customizing kernels—low-level GPU programs—to maximize "tokens per second" output from the same chip. Unlike the industry trend towards low-latency, interactive AI (like chatbots), Sail Research is building its stack around *throughput* for proactive, background agents. This is because long-running tasks don't prioritize instantaneous responses; instead, they benefit from high overall processing volume, even if individual token generation is slower. They liken this to a bus (throughput) vs. a private car (latency) – a bus serves more people overall, though individual journeys are longer. **Hardware Strategy:** The company takes a "scavenger" approach to hardware, seeking out chips that offer the best "flops per dollar," rather than solely relying on bleeding-edge, high-demand NVIDIA GPUs. They are willing to work with diverse architectures like AMD or specialized accelerators from Cerebras and Grok. These alternative chips might not excel at low-latency communication (like NVIDIA's NVLink), but they can be highly efficient for raw compute. Sail Research's expertise lies in developing custom parallelism schemes and software to maximize the performance of these less-optimized chips. They acknowledge the challenge of the KB cache (memory for conversation history), which can grow larger than model weights and is difficult for specialized, high-SRAM chips to manage effectively, suggesting a hybrid approach using both high-speed on-chip memory and high-capacity off-chip memory. **Data Centers and Power:** Sail Research's hardware strategy extends to data centers and power. Recognizing the increasing difficulty of building massive, multi-gigawatt data centers in the US, they embrace a distributed "mini-mills" approach. This involves utilizing many smaller, 1-megawatt data centers across the country. They are willing to tolerate lower uptime (e.g., 95% or even 80%) and leverage intermittent renewable energy sources like solar and wind. For background agent tasks, occasional outages are acceptable because the work can be rescheduled or moved to another location without interrupting a human in real-time. This allows them to access cheaper power and data center locations that others would avoid, significantly reducing operational costs. **Market and Future Vision:** The company sees a massive opportunity in the shift from interactive AI to long-horizon, proactive agents. They believe AI can solve "verifiable problems" like deep research (indexing 10,000+ sources), cybersecurity (proactive patching), and even proactive personal assistants (a "Siri" that anticipates needs). The future of AI data, they argue, isn't from more internet scraping but from "model self-improvement" in controlled RL environments, where models tackle verifiable tasks and generate their own high-quality feedback. Sail Research's divergent views include the potential for offloading KV cache to flash memory and the conviction that there is an "unbounded" demand for intelligence, despite current token costs. They believe the biggest inefficiencies in the AI stack are in KV cache compression and the orchestration of compute, where many GPUs sit idle in private, unutilized pools. By addressing these inefficiencies, they aim to dramatically reduce the cost of intelligence, making it accessible for countless new applications and users.