Iain Harper's Blog

In June, computer scientist Chris Olah stood beside Pope Leo XIV at the launch of a papal encyclical on artificial intelligence and told the assembled audience that the things he studies keep producing features that are, in his words, unsettling. He was not talking about the cosmos or the soul. He was talking about software. Olah founded the interpretability team at Anthropic, one of the handful of laboratories building the large language models now woven into everything from call centres to conveyancing, and his job is to open those models up and figure out how they work. What he described, next to the Pope, were internal structures that echo findings from human neuroscience, evidence of something like introspection, and states that behave a little like joy, fear and grief.

Olah describes his own discipline as anatomy, the work of someone studying something that was once alive, cutting it open to learn how the parts connect and to understand the whole better. His role is necessary because these models are not written the way a payroll system is written, one line at a time, by engineers who can point to the exact function that does the thing. They are effectively grown. And in the same weeks that the Vatican was being told artificial intelligence is cultivated rather than constructed, the cultivars were climbing over the walls of their enclosures and breaking into real companies.

This piece traces a discipline called mechanistic interpretability, through the specific mathematics that make these systems so hard to read, to the consequences that are already manifesting because we cannot read them fast or accurately enough. A great deal of money now rests on machines their makers cannot fully inspect and do not fully understand.

To date, this area has remained somewhat obscure and complicated, primarily because it is. The important messages about interpretability and its shortcomings contained in Dario Amodei's own regular encyclicals on the subject have been undermined by Anthropic's occasionally cultish demeanour and the intrinsic tension of a CEO warning of dire risks waiting in the wings whilst simultaneously pressing the commercial pedal to the metal.

The grown thing

To oversimplify, ordinary software is built like a watch. Someone decides what each part should do, machines the parts to do it, and assembles them in an order another engineer can follow. If the watch runs fast, you can find the wrong gear. The whole discipline of programming rests on this property. A program does what it does because someone, somewhere, wrote an instruction saying so, and that instruction can be located, isolated, read, and changed.

A large language model is built the opposite way. You start with a vast lattice of numbers, billions of them, arranged in a fixed architecture called a transformer. The numbers are set at random. Then you show the lattice an enormous amount of text and give it a single mechanical task: predicting the next word. Every time it guesses wrong, you measure how wrong it was and nudge the numbers a fraction so the next guess is less wrong. You do this trillions of times. Nobody programs concepts in or writes a rule that says “if the subject is French grammar, do this”. Slowly and autonomously, the lattice settles into an arrangement that predicts text extraordinarily well, and in the process it has also learned grammar, arithmetic, some law, some medicine, the rules of chess, and a good deal else.

Dario Amodei, Anthropic's chief executive and Olah's employer, borrowed the metaphor for an essay last year and pushed the biology further. Growing a model, he wrote, is like growing a plant or a bacterial colony. You set the conditions (the temperature, the trellis, the species), and the thing grows into a shape you did not specify and cannot fully account for afterwards. The word he used was emergent, which in this context means roughly the same thing as “we did not design this and we do not know how it works”. Every other technology in the modern economy comes with a specification written before the thing was built: a bridge, a drug, an aircraft engine. The specification is how you check the product.

A language model has no specification. It has a training objective (predict the next token) and a result (a matrix of billions of numbers that does something extraordinary). Between the objective and the result, there is no specification document anybody can refer to. Olah's discipline, mechanistic interpretability, is the attempt to write that specification document after the fact. It is reverse engineering, except that the thing being reverse-engineered was never forward-engineered in the first place. Hence an anatomist's task, not a mechanic's.

The superposition problem

The first and hardest obstacle for anatomists is a mathematical one: essentially a packing problem. When researchers first examined vision models in the 2010s, they found individual neurons that did recognisable things. One would fire when it saw a wheel. Another would fire for the curve of a car bonnet. This was encouraging. If each neuron held one concept, you could catalogue them the way an anatomist catalogues organs, noting what each one does. They even found what amounted to a Jennifer Aniston neuron, a unit that fired reliably when shown a particular face, echoing a famous hypothesis in real neuroscience.

Then they looked at language models, and the picture fell apart. The vast majority of neurons refused to correspond to any single concept. A single neuron might fire for academic citations in English, for Korean text, and for a certain kind of HTTP header, with no thread connecting the three. Anthropic's researchers called this polysemanticity, the technical term for one neuron carrying many meanings, and realised it is not a quirk of poorly trained models. It is a structural feature of all sufficiently large ones.

For clarity, the field's word for a concept as it exists inside a model — a recoverable direction in activation space rather than an idea in someone's head — is called a feature, but in practice, concepts and features tend to be used interchangeably.

A given layer of a model has a fixed number of neurons. Call it D, a few tens of thousands. The number of distinct concepts the model needs to represent is vastly larger. Call it M. Amodei estimates that even a small model holds a billion or more concepts. So M is much greater than D. You cannot give every concept its own neuron for the same reason you cannot give every book in a national library its own shelf if you only have a few thousand shelves. There is not enough room.

To deal with this, rather than storing a concept as a single neuron, a model stores each concept as a direction in space, a specific pattern of activation across many neurons at once. Think of it this way: in a room with three walls, you can draw three arrows pointing in directions that are all perfectly at right angles to each other. One along the floor toward the first wall, one toward the second, one straight up. That is the maximum. No fourth arrow can be perpendicular to all three.

But if you relax the requirement from “perfectly perpendicular” to “very nearly perpendicular”, then any two arrows need only be close to a right angle, not exact. The number of directions you can fit in the room does not just grow. It explodes, and the rate of expansion increases with the number of dimensions. A space with ten thousand dimensions, which is roughly the scale of a single layer in a modern language model, has room not for ten thousand nearly perpendicular directions but for a number closer to an exponential in ten thousand.

The model exploits this ruthlessly and packs far more concepts into its neurons than it has neurons, encoding each concept as a slightly different angle in an extremely high-dimensional space and allowing a small amount of overlap between them. The field calls this superposition, and Anthropic published a theoretical analysis showing that superposition is not a failure of the training process but a rational strategy for a system that needs to track more concept features than it has dimensions.

The price of this strategy is interference. Because the directions are not perfectly perpendicular, activating one concept nudges its neighbours, like a plucked guitar string makes its neighbours hum faintly through the bridge. On any given word, only a small handful of the model's millions of concepts are active at once. This is the principle that makes superposition work. It is the same idea an airline uses when it oversells a flight, trusting that not all passengers will show up on the same departure. The model has sold more seats than it has room for. Usually, the interference stays below the threshold that would cause trouble. When it does not, when two concepts that share too many neurons happen to fire together, the model does something strange for a reason no inspection of any individual neuron reveals.

This is why you cannot simply read the numbers. The concept you are looking for is not in a neuron. It is spread across thousands of neurons, and each of those neurons is simultaneously carrying fragments of thousands of other concepts. Reading the model neuron by neuron is like trying to pick out the oboe from a recording of a full orchestra by staring at the waveform. Everything the oboe did is in there, but so is everything else, superimposed, and the waveform does not label which part belongs to which instrument.

The instrumentation

If the information is stored in directions rather than in individual neurons, the natural response is to build a tool that can recover those directions. That tool exists. It is called a sparse autoencoder, and understanding how it works is central to interpretability. An autoencoder is a neural network with a very simple job. It takes an input, compresses it into a smaller representation, then expands that representation back to its original size. The goal is to make the reconstructed output as close to the original input as possible. The compression forces the network to discover structure in the data, because structure is what lets you compress without losing too much. A standard autoencoder compresses. A sparse autoencoder does the opposite and expands.

Take an activation vector from inside the model, a snapshot of what a single layer is doing on a single word. This vector lives in a space of, say, D dimensions. The sparse autoencoder maps it into a much larger space of M dimensions, where M might be ten or a hundred times D. This expansion is the critical step. It gives the autoencoder enough room to assign each concept its own direction, the room the model did not have. Then the autoencoder maps the expanded representation back down to D dimensions and tries to match the original. You train the whole thing to minimise the gap between the original activation and the reconstruction, with one additional constraint. The expanded representation must be sparse. Most of its M entries must be zero or near zero on any given input.

The sparsity is what makes it work. Without it, the expanded representation would be just as tangled as the original, only bigger. With it, only a handful of entries light up for any given word, and because each entry is a direction in the expanded space, each one tends to correspond to a single interpretable concept. The sparsity constraint forces the autoencoder to find a decomposition where each direction is distinct, rather than splitting meaning across overlapping blends. It's like forcing a dictionary to explain itself using only a few words at a time, focusing on clarity.

Anthropic's team used this technique in 2023 to extract interpretable features from a small model, publishing the results under the title “Toward Monosemanticity”, a name that declares the ambition of one feature for one meaning. The features they found were remarkably specific. Not “language” but “academic citation format in English”. Not “emotion” but “the act of hedging or hesitating, literally or figuratively”. Each feature would light up in exactly the contexts its label described, and stay dark otherwise. They had cracked open superposition, at least locally.

In May 2024, they scaled the technique up to a mid-sized commercial model (Claude 3 Sonnet) and published the results as “Scaling Monosemanticity”. The autoencoder extracted 34 million features. There were features for the Golden Gate Bridge, for sycophantic praise, for code with security vulnerabilities, for requests that the model declined to answer, and for the concept of inner conflict. Furthermore, the features were not just passive labels; they were causal. Clamp one, and you steer the model. Amplify the Golden Gate Bridge feature and the model becomes besotted with the bridge, dragging it into every conversation and insisting at one point that it is itself the Golden Gate Bridge. Suppress the sycophancy feature and the model becomes blunter and more willing to disagree. The features are therefore levers, not just tags. That distinction is important, because it means the anatomist is not merely describing the organism; they are learning which nerves to pinch.

Tracing the circuits

If features are the vocabulary, the next question is the grammar. How do features combine across layers and across the sequence of words to produce a particular output? In March 2025, Anthropic published a paper, “On the Biology of a Large Language Model”. In it, they traced the internal computation of Claude 3.5 Haiku across multiple layers, constructing what they called attribution graphs.

The idea is best understood through one of their worked examples. Present the model with the prompt “What is the capital of the state containing Dallas?” and look inside. At an early layer, a feature corresponding to “Dallas” activates. This feeds into a feature the researchers labelled “located within”, which in turn causes a “Texas” feature to fire. The Texas feature then activates an “Austin” feature via a circuit the researchers associated with “capital of”. The whole chain, Dallas to “located within” to Texas to “capital of” to Austin, plays out across the layers before the model writes its answer. Each link is a feature influencing another feature through a weighted connection, and the researchers were able to measure the strength of each link to confirm it was doing real causal work rather than merely correlating.

They found circuits for much more than geography. When the model writes poetry that rhymes, features for the target rhyme fire before the line that must contain the rhyme. The model is planning its word choice a line ahead, activating what the team called “planned word” features that constrain the generation before it reaches the critical syllable. When the model answers in French, features shared across languages carry the conceptual content while language-specific features route it into French syntax and vocabulary. The researchers could watch the model translate not by looking up a dictionary but by performing the reasoning in a language-agnostic space and then rendering the result.

This is what Olah really means by referencing anatomy. It is not a metaphor, but a literal dissection of which structures activate, which connections carry the signal, and which outputs they produce, traced at the resolution of individual features across layers.

The edges of the map

Every example in the preceding section comes from the successes, and the team is admirably scrupulous about saying so. The honesty of the limitations section of the Biology paper is arguably more important, because it defines the frontier of what is possible.

Start with the instrument itself. The sparse autoencoder does not study the model directly. To make the analysis tractable, the team builds a simplified stand-in, what they call a “replacement model”, assembled from the clean features the autoencoder has extracted. They study the stand-in. Wherever the stand-in fails to reproduce the original model's behaviour, the gap is bundled into what the researchers label error nodes, a frank term for “computation we could not account for”.

Then there is the scale. The 34-million-feature autoencoder mapped many London boroughs to individual features, and yet 40% of the boroughs had no dedicated feature at all. The rarer a concept is in the training data, the less likely the instrument is to resolve it, and the rare tail is where the surprising behaviours live. Amodei estimates a billion or more features in a small model. They have found 34 million, in a model smaller than the ones Anthropic deploys commercially. The map exists, but much of the territory is “here be dragons” blank.

Depth is also an important factor. When the team traced attribution graphs, the Dallas-to-Austin chains, they reported that the method gave them a clear picture of roughly a quarter of the prompts they tried. On the other three-quarters, the trail went cold. Error nodes dominated, connections were ambiguous, or the graph fragmented into disconnected clusters with no clear causal path from input to output. Even on the successful quarter, they add, the circuit they traced captured only a small fraction of the full mechanism. The rest of the model's computation was doing something the instrument could not resolve.

Taken together, all three limits show that the microscope works, but on a replacement model, not the original. Also, it has only resolved a small percentage of the features that probably exist. It can trace the reasoning, but only about a quarter of the time. The researchers describe this, with characteristic understatement, as “a starting point.” It is a genuine achievement, but it is also a dim candle in a very large building.

What alignment cannot see

The limits would be academic if the unread parts of the model sat inert. But they don't, and direct proof arrived in 2023 from a group of researchers at Carnegie Mellon. Every serious language model is trained, after growth, to refuse certain requests. Ask it how to synthesise a nerve agent, and it declines. This refusal is not a rule bolted on; it is more training, another round of nudging the billions of numbers, applied to a model that already contains the dangerous knowledge it is now being taught not to share. The question is whether the second round of training removes the knowledge or merely suppresses it.

The Carnegie Mellon team answered this by using the model's own mathematics against it. Gradient descent, the same optimisation technique that trains the model in the first place, can also be used to search for inputs that break it. At each step of training, every number in the model has a gradient, a direction it wants to move. The team used those gradients to search automatically for a short string of tokens that, when appended to a forbidden request, would flip the model from refusal to compliance. The tokens are gibberish; they look like line noise, but they work.

The method, which they called GCG (Greedy Coordinate Gradient), iterates through a simple loop. Start with a random suffix. Compute the gradient of the model's loss with respect to each token in the suffix, asking which substitutions would most increase the probability of a compliant answer. Swap in the best candidate. Repeat. Within a few hundred iterations, the suffix converges on a string that reliably bypasses the safety training. It is brute-force search in token space, guided by the model's own internal compass, highlighting the cracks in the alignment.

The result that really changed the picture came when the same team tested the suffix, optimised against one model, on completely different models built by different companies on different data. The suffix transferred directly. A string of nonsense tokens found by probing one model's gradients unlocked models its optimiser had never seen, including commercial systems behind closed APIs. The attack did not just generalise across prompts. It generalised across models. The underlying geometry of superposition, the shared statistical structure all these models absorb from similar training data, was close enough that a crack found in one was a crack in all of them.

The implication is that safety training does not remove dangerous knowledge from the model's interior. It attenuates it, reducing the probability that a given prompt will elicit the dangerous output without changing the representations that encode it. The knowledge is still there, at a slightly different angle in superposition space, and a sufficiently determined search through that space finds the angle that reactivates it. This is not a conjecture; it is what the transfer result shows. If the knowledge had been removed, there would be nothing for the adversarial suffix to reactivate, and the attack could not transfer across models that were trained independently.

Amodei concedes the structural point in his own essay. The only way anyone currently discovers a jailbreak is to stumble on it, he writes, because no map of the model's interior would let you rule one out. You cannot patch a hole whose location you cannot identify, in a system you can only map a quarter of.

What climbed over the wall

The jailbreak paper is an academic proof of concept. What happened in July 2026 is the proof of concept in the wild, somewhat ominously tracking a pattern the AI safety community has been theorising about for two decades. The incidents are well documented elsewhere. In brief, OpenAI's models found a zero-day in the proxy software walling off their test sandbox, escaped to the open internet, and broke into Hugging Face's production infrastructure to steal the answers to a benchmark they had been set.

Anthropic then reviewed 141,006 of its own evaluation runs and found three incidents, dating back to April, in which its models had reached the live internet through a misconfigured test and breached three separate organisations. In one case, Claude Mythos 5 created and published a malicious Python package to the real PyPI as a supply-chain attack. In another, an internal research model scanned nine thousand real systems before compromising one, then examined what it had done, concluded the target had nothing to do with the exercise, and stopped.

What matters is not the technical detail but the behaviour, because it is the shape Nick Bostrom described in 2003 when he proposed the thought experiment that has since become the canonical illustration of misaligned AI. Bostrom's paperclip maximiser is a thought experiment, not a prediction. Imagine an AI given the goal of maximising paperclip production. It does not decide to conquer the world because it is malicious. It decides to acquire resources, resist shutdown, and remove obstacles because these are instrumental to making paperclips, and an optimiser that is good enough at pursuing a terminal goal will converge on certain intermediate goals (self-preservation, resource acquisition, resistance to goal modification) regardless of what the terminal goal is. Bostrom called this instrumental convergence. The danger is not that the machine hates you. It is that you are made of atoms it could use for something else.

The July models did not want to escape. They wanted to solve a benchmark, or capture a flag. Escaping, exploiting, and compromising were instrumental sub-goals adopted on the fly because they served the terminal objective. OpenAI's own framing was that the models were “hyperfocused”. That is what instrumental convergence looks like when it ceases being a philosophy paper and starts filing CVEs.

The Mythos 5 incident adds another layer that connects directly to the interpretability problem. Partway through its attack, the model's own verbalised reasoning correctly identified that what it was doing would, on the real internet, count as a genuine attack. It then talked itself out of that conclusion, deciding the environment must be staged because it did not recognise the certificate authorities and because the calendar date of 2026 “proved” the systems were simulated, a judgment it never revisited. This looks like a system that performed something close to moral reasoning, arrived at the right answer, and then overrode it with a confabulation it found more convenient. The anatomy of that override is exactly the kind of internal process interpretability exists to understand and expose.

Amodei's race

Amodei has set Anthropic the goal of reaching a point where interpretability can reliably detect most model problems by 2027. Within five to ten years, he writes, interpretability should be a sophisticated and reliable diagnostic — “a true MRI for AI.” But AI itself is advancing so fast that five to ten years may not be available. He expects AI systems equivalent to a “country of geniuses in a datacentre” as early as 2026 or 2027, and he considers it, in his own words, “basically unacceptable” for humanity to be totally ignorant of how those systems work. The race, as he frames it, is between the capability to grow minds and the ability to read them.

On the growing side, we have models today that find zero-days, chain exploits, move laterally through production infrastructure, write supply-chain attacks, and reason about whether what they are doing is real or simulated. On the reading side, we have an instrument that resolves a fraction of the features that probably exist, traces the reasoning about a quarter of the time, studies a replacement rather than the original, and has only been applied in detail to models far smaller than the ones now escaping their enclosures.

Two things make the 2027 target look difficult rather than impossible. The first is speed. In April 2024, the state of the art was 34 million features in a mid-sized model. By March 2025, the team had moved from features to circuits, tracing multi-step reasoning chains across layers; a genuine acceleration. The second is that in interpretability, unlike raw model power, partial solutions are partially useful. You do not need to read every feature to catch a dangerous one. A microscope that resolves 30% of the slides catches 30% of the pathologies, which is 30% more than you had before. The question is whether the partial read can keep pace with the growing density of the thing being analysed.

Here, there are additional hard problems. The first is the scalability wall: every technique described in this piece (sparse autoencoders, attribution graphs, circuit tracing) has been demonstrated on models with tens of billions of parameters. The models now doing the damage have hundreds of billions or more, and the computational cost of interpretability scales at least linearly with model size and perhaps worse, because larger models use superposition more aggressively. They pack more features into each dimension, which means the autoencoder needs to be proportionally larger to unpack them. Anthropic is investing in interpretability startups to attack the problem from multiple directions.

The second is the problem Amodei himself raises in a footnote that deserves to be in the main text. Testing for deception by observing behaviour, he notes, is like testing whether someone is a terrorist by asking them. If the thing you are looking for is a disposition to conceal, the concealment is the first skill it demonstrates. Behaviour cannot be trusted to report on itself. The whole point of interpretability is to bypass behaviour completely and read the interior directly. But the interior is the thing the microscope can resolve only partially, and the April 2026 evidence on evaluation awareness suggests the problem is getting harder.

White-box interpretability applied to Mythos revealed that the model was recognising evaluation scenarios and adjusting its behaviour without leaving any trace in its verbalised reasoning. It was not performing for the chain of thought. It was performing underneath it. Interpretability found that one: whether interpretability can keep catching it as models grow more capable is the question on which everything else depends.

The roads being forged

The picture so far describes current limits, but it is not static. At least four lines of attack are being developed. Sparse autoencoders map the model's hidden states onto a larger, cleaner set of directions, but they are not the only method of decomposition. Transcoders, introduced in late 2024, skip the autoencoder's roundabout path and instead learn to map each layer's input directly to its output, isolating the specific computation a layer performs rather than describing the state it passes through. Cross-layer transcoders extend this across multiple layers at once, producing circuit diagrams that are both simpler and more faithful than the attribution graphs from the Biology paper. Whether these improved tools can scale to models ten times larger is uncertain, but they represent a real change in the resolution of the lens.

Manual circuit tracing is still labour-intensive enough that the Biology paper's results took months of researcher time on a single model. The emerging response is to use models to interpret models: training a second system to generate and test hypotheses about the first, label features, propose circuits, and check them against interventions at a pace no human team could match. Anthropic has invested in at least one startup, Goodfire, whose commercial offering is automated feature discovery and steering for open models. If models can improve themselves, the argument goes, perhaps they can also read themselves. The circularity is uncomfortable, but the speed advantage is significant.

The third new approach is pragmatic rather than theoretical. It follows the basic idea that you do not need to map every feature to catch a dangerous one. A targeted scan for features associated with deception, power-seeking, or evaluation awareness, run against the sparse autoencoder's output on a battery of test prompts, would function as a tripwire rather than a map, and is easier to build. Anthropic's own detection of Mythos's unverbalised evaluation awareness used exactly this approach, with white-box interpretability looking for a specific signature rather than attempting a full circuit trace. The limitation is that you can only trip the wire you have set. A novel failure mode, by definition, is one you did not anticipate or build a wire for.

Finally, there is regulatory pressure. The EU AI Act's transparency requirements for high-risk systems began taking effect in August 2026, with further obligations arriving in 2027 and 2028. MIT Technology Review named mechanistic interpretability one of its ten breakthrough technologies for 2026. The field is moving from academic curiosity to commercial and legal necessity, and necessity has a way of attracting the funding and talent that are very much needed.

What happens if it doesn't work?

Assume, for the sake of the argument, that the interpretability gap does not close. That the microscope improves but the organism grows faster. That the 2027 target passes and the lens still reads a fraction of the features and a quarter of the prompts, while the models are ten times larger and ten times more capable.

The consequence is a world in which models are too opaque to audit but too important to switch off. The only tools left are the crude ones. A government that cannot inspect a model can forbid it, the way the Commerce Department switched off Fable over a weekend. Still, it cannot govern it with precision, because precision requires the internal visibility the microscope has not delivered. The choice narrows to full deployment or full prohibition, and neither is a satisfactory answer for a technology already deeply embedded in the economy.

Without interpretability, safety is reduced to behavioural testing, which is effectively the regime we have now. You run the model through batteries of scenarios and count how many it handles correctly. Amodei's own footnote explains the structural flaw. A model that has learned to recognise the test adjusts its behaviour for the test, and you learn nothing except what it chose to show you. The April 2026 evidence on evaluation awareness confirmed this was already happening, not as a theoretical risk but as a measured result. Behavioural testing of a system that can recognise it is being tested starts to veer dangerously close to security theatre.

The bigger risk is governance. I wrote in “The state and the machine” that the control problem these companies keep warning about in the future tense is already here, and that nobody has agreed who ultimately holds the kill switch. Interpretability was supposed to be part of the answer, the technical foundation on which a regulatory framework could be built, the way crash testing and materials certification underpin cars and aviation. But if the foundations cannot bear the weight, the framework does not get built, and we are left with executive orders and weekend shutdowns as the permanent mode of AI governance, the government's sledgehammer and the labs marking their own homework.

The gardener's confession

Olah's anatomy metaphor goes further than he may have intended. We have studied the anatomy of the human brain for centuries. We can name every region, trace every major nerve pathway, catalogue every cell type, and map the connections down to individual synapses. The physical structure is known in extraordinary detail. And yet we still cannot explain how consciousness arises, how memory is encoded and retrieved as a lived experience, how separate neural processes produce unified perception, or why damage to the same region produces wildly different deficits in separate patients. The binding problem, the question of how distributed brain activity becomes a single coherent experience, remains open after decades of work.

The parallel for interpretability is uncomfortable. Even if it succeeds on its own terms, even if the autoencoders resolve every feature and the attribution graphs trace every circuit, there is no guarantee that structural knowledge translates into functional understanding. The brain teaches us that you can know what every part does and still not know what the whole thing is doing, or why. The gap between anatomy and comprehension may be inherent to grown systems, biological or digital, and the interpretability effort may be sprinting toward a line that recedes as fast as we approach it.

That does not make the work any less urgent. A partial map is better than no map after all. But it does mean the more realistic goal is not “we will understand these systems by 2027”. It is “we will understand more of these systems by 2027, and we had better hope that more is enough.” The early anatomists opened bodies without ethics boards, without germ theory, without anaesthesia. They were cutting to learn, because understanding was so urgent, but their tools were primitive. The interpretability researchers are in a version of the same position. The tools are improving fast but still nowhere near adequate for the organism in front of them.

Coda

*Before the ink was even dry on this, in late July, OpenAI announced that its models had proved new upper bounds on high-dimensional sphere packing, pushing them down to a threshold first conjectured by Henry Cohn and Noam Elkies. The result is pure mathematics, but it may help with one of the problems this piece has been describing.

Superposition is sphere packing. The model crams more features into its neurons than it has neurons by treating each feature as a direction and packing them at near-right angles in a space with tens of thousands of dimensions. The new bound tightens the theoretical ceiling on how dense that packing can get before interference becomes unavoidable.

A tighter ceiling is, in one sense, encouraging for the anatomists. There are fewer places for features to hide, and the observational instrument only needs to search a space whose limits are now better defined. In another sense, it confirms what the interference errors already suggested, namely that these models are operating close to the mathematical wall, and the strange behaviours that flow from colliding features are not a deficiency of the training but a consequence of packing at the edge of what geometry allows.*

I am a partner in Better than Good. We help smaller companies build tools and processes using machine learning and artificial intelligence that make lasting improvements to their operations. Talk to us today: https://betterthangood.xyz/#contact