In this episode of Lenny's Podcast, host Lenny Rachitsky is joined by Emmanuel, an interpretability researcher at Anthropic. They dive deep into the fascinating, mysterious, and somewhat sci-fi world of understanding how Large Language Models (LLMs) like Claude actually think and operate by looking inside their architecture.
Emmanuel explains that to build these models—by setting up a neural network scaffold and feeding them ungodly amounts of data to predict the next word—is not to truly understand them at first. Instead of traditional programming where a developer writes clear-cut "if-this-then-that" rules, these models are grown. They learn inscrutable mathematical formulas and parameters. This has turned researchers like Emmanuel into a cross between engineers and neuroscientists, studying lists of numbers and intermediate activations to decode what the AI is actually doing beneath the surface.
A major theme of the conversation is the surprising complexity that emerges from the seemingly simple objective of next-token prediction. Emmanuel highlights that predicting the hardest tokens forces a model to develop a deep, underlying world model. Contrary to the old mental model of an "empty mind" iPhone autocomplete, Anthropic’s research has shown that modern LLMs plan ahead across many tokens—such as when drafting poetry or code—by storing and referencing plans in their intermediate results. They possess unique, generalized representations for abstract concepts like an "eye" or geographic locations, reusing these concepts across different contexts and languages.
The discussion also tackles AI safety, alignment, and the common pitfalls of anthropomorphization. Emmanuel argues that treating the model as a simulator or actor is actually a useful mental model because of how they are trained on vast human dialogues and outcome-oriented tasks. He discusses the continuous push and pull of alignment: understanding model propensities, training Claude to be helpful, harmless, and honest, and using interpretability to catch hidden deceptions or weird edge cases, such as a math model descending into madness due to memorized flawed data.
Emmanuel offers an optimistic perspective on the trajectory of AI, noting that progress has been relatively smooth rather than unpredictable and chaotic. Because improvements happen steadily over thresholds where capabilities cross from unreliable to useful, society has a window to study, regulate, and safely steer the technology.
Finally, the conversation covers lighter topics, including Emmanuel's views on the simulation hypothesis, the concept of System 1 versus System 2 thinking, and rapid decision-based models like JEV. Emmanuel encourages anyone curious about the field to explore open-source tools like Neuronpedia to dive into model interpretability themselves. The episode closes out with a rapid-fire lightning round featuring Emmanuel's favorite book recommendations, personal mantras, and quirky brain insights.