The one AI detector people actually trust

最新一期的《The VergeCast》播客由执行主编杰克·卡斯特罗纳基斯主持，深入探讨了AI文本检测这一长期存在的挑战，并介绍了Pangram作为一项潜在的突破性解决方案。卡斯特罗纳基斯强调了人们对AI检测器普遍存在的怀疑，这种情绪也得到了听众艾登的印证，艾登讲述了他的大学报社驳斥AI检测不可靠的说法。然而，随着Pangram崭露头角并成为备受信赖的名字，情况正在发生转变。 Pangram的首席执行官马克斯·斯皮罗受邀参加播客，讨论了他公司的发展历程和技术。斯皮罗承认，最初在对抗“AI检测器无效”这一说法时，面临着巨大挑战。他指出，Pangram由他和同样具备AI和机器学习背景的联合创始人创立，至今已有近三年。他们在大约一年的研究后取得了突破，最初实现了千分之一的“标志性误报率”，现已提升至令人印象深刻的万分之一（0.01%）。 Pangram的核心创新在于其“主动学习”方法。他们首先使用一个表现良好的AI模型，然后用它扫描大量人类撰写的文本语料库，识别模型难以区分人类文本和AI文本的例子。对于这些接近人工-AI边界的“边缘案例”，Pangram会创建“合成镜像”——即要求AI以与人类原始文本*相同风格*生成文本。通过这些人工-AI配对（特别是那些困难的例子）来训练模型，Pangram学习了区分人类写作的微妙“文体选择”。斯皮罗解释说，模型会“整体”地审视文档，做出许多“微观决策”，这些决策汇总后，能以高度置信度判断文本是否由AI生成。尽管模型本身有点像一个“黑箱”，但Pangram利用“可解释性”研究来理解哪些特征触发了其检测，甚至发现其模型可以根据生成文本的特定AI模型家族进行聚类。 斯皮罗将Pangram的方法与早期依赖“困惑度”（衡量一段文本对语言模型的“惊奇”程度）的AI检测器进行了对比。AI倾向于产生低困惑度、不令人意外的文本。然而，这种方法对记忆文本（如《独立宣言》，可能会被标记为AI生成）和简单语言（如英语学习者使用的语言）无效。Pangram则学习AI语言模型做出的一致“微观决策”。 关于可靠性，斯皮罗坚称Pangram的准确性，但也指出万分之一的误报率意味着偶尔会有错误。置信度随文本长度增加而增加；一条50字的推文若被标记为AI生成，很可能属实；但一本长篇小说被识别为90%由AI生成，则几乎是确凿无疑。Pangram还可以识别并评分文档的各个部分，显示人工、AI生成或辅助内容的细分。 对话还提到了最近的“英联邦奖”争议，其中一篇被Pangram评估为100%由AI生成的故事获得了奖项，尽管作者声称是人类创作。斯皮罗对作者的解释（包括使用语音转文本）表示怀疑，并指出了其采访中的不一致之处，这进一步证实了Pangram的检测结果。他还驳斥了使用Claude等通用大型语言模型来检测AI的效用，因为它们缺乏专业的检测能力。 Pangram服务于多样化的用户，包括教育工作者和大学（与Canvas等学习管理系统集成）、出版商（确保内容完整性），甚至其他AI公司（清理训练数据中无意的AI生成内容）。斯皮罗建议教授们将Pangram的积极结果作为对话的起点，将AI的使用视为更深层次学生问题的症状，而不仅仅是判零分的理由。 展望未来，斯皮罗谈到了AI检测的对抗性本质。他讨论了“拟人化工具”（humanizers）日益严重的问题——这些工具旨在润色AI文本以绕过检测器。Pangram正在积极开发一个专门用于更好检测这些“拟人化”文本并更准确衡量“AI辅助程度”的新模型。在ChatGPT问世后的世界中，获取新鲜、未受污染的人类手写数据用于训练仍然是一个重大挑战。斯皮罗提到正在探索诸如举办有监督的征文比赛，或从值得信赖的专业作家和历史作品中获取数据等方法。 尽管可能面临来自科技巨头的竞争，斯皮罗认为Pangram先进且持续改进的技术，使其成为解决这一长期而艰难问题的“赢家通吃”方案。他最后将自己的角色定义为“劣质内容清洁工”，表达了对AI写作的一种微妙看法。虽然AI生成文本有其用武之地，特别是如果公开说明其来源，但他强烈厌恶对其来源的欺骗。对斯皮罗来说，“劣质内容”指的是缺乏真正人类思想的AI内容，优先考虑易读性而非深度或有效沟通。“写作是思想的证明”这一核心理念是其观点的核心，他强调将思考任务卸载给AI会阻碍人们深入理解和个人表达。

The latest episode of The VergeCast, hosted by Executive Editor Jake Kastronakis, delves into the persistent challenge of AI text detection, introducing Pangram as a potential breakthrough solution. Kastronakis highlights the widespread skepticism surrounding AI detectors, a sentiment echoed by a listener named Aiden, who recounted his college newspaper dismissing AI detection claims as unreliable. However, a shift is occurring, with Pangram emerging as a trusted name. Max Spiro, CEO of Pangram, joins the podcast to discuss his company's journey and technology. Spiro acknowledges the initial struggle against the "AI detectors don't work" narrative, noting that Pangram, founded by himself and his co-founder with AI and machine learning backgrounds, has been around for nearly three years. Their breakthrough came after about a year of research, achieving a "flagship false positive rate" of one in 1,000, now improved to an impressive one in 10,000 (0.01%). Pangram's core innovation lies in its "active learning" approach. They start with a decent AI model, then use it to scan a vast corpus of human-written text, identifying examples where the model struggles to differentiate human from AI. For these "edge cases" close to the human-AI boundary, Pangram creates "synthetic mirrors" – asking an AI to generate text in the *same style* as the human original. By training their model on these human-AI pairs, particularly the difficult examples, Pangram learns the subtle "stylistic choices" that distinguish human writing. Spiro explains that the model looks "holistically" at a document, making many "micro-decisions" that, when aggregated, provide high confidence in AI generation. While the model itself is somewhat of a "black box," Pangram uses research in "interpretability" to understand what features trigger its detection, even discovering that its model can cluster texts by the specific AI model family that produced them. Spiro contrasts Pangram's method with earlier AI detectors that relied on "perplexity," a measure of how surprising a piece of text is to a language model. AI tends to produce low-perplexity, unsurprising text. However, this method fails with memorized texts (like the Declaration of Independence, which would be flagged as AI) and simple language (like that used by English language learners). Pangram, instead, learns the consistent "micro-decisions" AI language models make. Regarding reliability, Spiro asserts Pangram's accuracy but notes the 1 in 10,000 false positive rate means occasional errors. Confidence increases with text length; a 50-word tweet flagged as AI is likely correct, but a long novel identified as 90% AI is a near certainty. Pangram can also identify and score parts of a document, showing a breakdown of human, AI, or assisted content. The conversation touches on the recent "Commonwealth Prize" controversy, where a story assessed as 100% AI by Pangram won an award, despite the author's claims of human authorship. Spiro expresses skepticism about the author's explanations (including using voice-to-text) and points out inconsistencies in his interview, further validating Pangram's findings. He also dismisses the utility of using general LLMs like Claude to detect AI, as they lack specialized detection capabilities. Pangram serves diverse users, including educators and universities (integrating with learning management systems like Canvas), publishers (to ensure content integrity), and even other AI companies (to clean training data from unintended AI-generated content). Spiro advises professors to use a positive Pangram result as a starting point for conversation, viewing AI use as a symptom of deeper student issues rather than just a reason for a zero grade. Looking ahead, Spiro addresses the adversarial nature of AI detection. He discusses the growing problem of "humanizers" – tools designed to paraphrase AI text to bypass detectors. Pangram is actively developing a new model specifically designed to better detect these humanized texts and more accurately gauge the "degree of AI assistance." A significant challenge remains obtaining fresh, uncontaminated human-written data for training in a post-ChatGPT world. Spiro mentions exploring methods like essay contests with supervision or sourcing from trusted professional writers and historical works. Despite potential competition from tech giants, Spiro believes Pangram's advanced, continuously improving technology positions it as a "winner-takes-all" solution to a persistent and difficult problem. He concludes by defining his role as a "slop janitor," expressing a nuanced view on AI writing. While AI-generated text has its place, especially if disclosed, he strongly dislikes dishonesty about its origin. For Spiro, "slop" refers to AI content that lacks genuine human thought, prioritizing ease of reading over depth or effective communication. The core concept of "writing being proof of thought" is central to his perspective, emphasizing that offloading thinking to AI hinders deep understanding and personal expression.