Iain Harper's Blog

In the last years of his life, Kurt Gödel starved himself to death. Convinced that someone was poisoning his food, he ate only what his wife Adele had tasted first. When she was hospitalised after a stroke in late 1977, he stopped eating altogether. He died in Princeton Hospital on January 14th 1978, weighing 29 kilograms. The death certificate read “malnutrition and wasting from neglect caused by personality disturbance.” The greatest logician since Aristotle, a man who had proved that mathematics itself contained truths it could never reach, was killed by a distorted inner logic he could not escape.

Almost nobody outside mathematics knows his name. Einstein did. The two were faculty at Princeton’s Institute for Advanced Study from the 1940s onward, and Einstein, by then ageing and isolated from the mainstream of physics, told colleagues that he went to his office “just to have the privilege of walking home with Kurt Gödel.” They made an odd pair on the Princeton sidewalks, Einstein rumpled and laughing, Gödel dapper in a white linen suit, talking animatedly in German on their daily walk to and from the Institute. John von Neumann, who cancelled an entire lecture series on David Hilbert’s programme after reading Gödel’s 1931 paper, called his work “singular and monumental, a landmark which will remain visible far in space and time.”

So what did Gödel prove, and why does it matter now, in the middle of an AI boom that is spending trillions of dollars, much of it resting on the assumption that intelligence is a scaling problem?

What incompleteness means

Put simply, Gödel proved that mathematics cannot fully explain itself. The longer version requires a little patience. In 1900, the German mathematician David Hilbert challenged the field to build what amounted to a perfect machine for mathematics. Start with a set of basic rules (called axioms), things so obviously true they need no argument, and then derive every mathematical truth from those rules, step by mechanical step. If you could do that, mathematics would be complete, meaning every true statement would be provable, consistent, and free of contradictions. You could hand the whole enterprise over to a clerk who follows instructions. This was Hilbert’s programme, and for three decades it was the organising ambition of the field. Then, in 1931, at the age of 25, Gödel demolished it in one stroke.

Gödel’s first incompleteness theorem proved that any set of rules powerful enough to handle basic arithmetic will contain true statements it cannot prove, not because the rules were poorly chosen, but as a structural feature of rule-based systems themselves.

His trick was to construct a mathematical sentence that refers to itself. Consider the sentence, “This sentence has no proof.” Gödel’s technical feat, the part that fills his 1931 paper, was to build this sentence from pure arithmetic, by encoding statements about numbers as numbers themselves. It is not English smuggled into maths. It is pure maths. There are only two possibilities. Either the system can prove it, or it cannot.

If the system can prove “This sentence has no proof,” there is an immediate problem. We have just proved a sentence that claims to have no proof. A system that proves false things is contradictory, and contradictions in mathematics are fatal. Once you allow a single one, you can use it to prove anything, including that 1 equals 2. The system becomes useless.

If the system cannot prove “This sentence has no proof.”, there is a different problem. The sentence said it had no proof, and it turns out to be right. It is a true statement. But the system has no way to prove it. So we have a truth the system cannot reach, which means Hilbert’s rulebook has a blind spot.

Any sensible mathematical system would rather have blind spots than contradictions. So the sentence (logicians call it a Gödel sentence) is true but unprovable, and Hilbert’s dream of a rulebook that can prove every true thing was dead.

Gödel’s second theorem twisted the knife. It showed that no set of mathematical rules can prove, using only its own rules, that it is free of contradictions. If you want to check whether your system is trustworthy, you always need a bigger system to do the checking, and that bigger system inherits the same limitation. Turtles all the way down.

This is not mysticism, nor is it a claim about consciousness or creativity. It is a precise result about rule-based systems, the kind of systems that all software, including AI, are built from. That is what makes it relevant today.

The failed dream that built the computer

Hilbert had asked for one more thing, and Gödel’s paper left it wounded rather than dead. Alongside completeness and consistency, he wanted decidability, a mechanical method that could, in a finite number of steps, determine whether any mathematical statement follows from the rules. No genius required: crank the handle and read the verdict.

In 1936, a 23-year-old Cambridge fellow named Alan Turing killed that too. To prove that no mechanical method could exist, he first had to pin down what “mechanical method” meant, which nobody had done before. His answer was an imaginary device, a paper tape and a head that moves along it, reading and writing symbols according to a fixed table of rules. Anything a human clerk could work out by rote, this device could also work out.

Then he showed the device has a blind spot of its own. Imagine a fortune-teller who is never wrong, and a stubborn customer determined to sabotage every forecast. “You will leave by the door.” He climbs out the window. “You will take the window.” He strolls out the door. She is not bad at her job. The job is impossible because her prediction feeds back into the very behaviour it is trying to predict.

Turing turned that scene into code. The checker plays the fortune-teller. It is a program whose job is to read any other program the way you might read a recipe, then predict its fate. Either “this one finishes” or “this one grinds on forever.”

The saboteur plays the stubborn customer. It is a short program with a copy of the checker tucked inside, plus one standing rule. Ask the checker what I am predicted to do, then do the opposite. If the prediction is that it finishes, it deliberately loops forever. If the prediction is that it runs forever, it stops dead.

So what does the checker predict for the saboteur? “Finishes” is wrong, because the saboteur hears that and loops. “Grinds on forever” is wrong, because the saboteur hears that and stops.

The saboteur is assembled entirely from the checker’s own parts, which makes it inevitable rather than a fluke. Build a perfect checker, and you have, in the same afternoon, built the plans for the thing that breaks it. A perfect checker is therefore a contradiction in terms.

Programs that predict how other programs will behave most of the time are unremarkable — static analysers and type checkers are used routinely. The program that cannot exist is the one that is never wrong. This is the so-called halting problem: Gödel’s self-referential sentence, rebuilt from machinery, a machine then forced to ask a question about itself.

To show what machines cannot do, Turing had to invent the machine. His imaginary device is the theoretical blueprint of the general-purpose computer, a single machine that can run any program you feed it as data. Nine years later, John von Neumann, who knew Turing’s paper well and admired it, wrote the First Draft of a Report on the EDVAC, which is, in logical terms, Turing’s universal machine rendered in vacuum tubes. Essentially every computer built since follows that design. The laptop on your desk and the datacentre GPU training the next frontier model are, once the engineering is stripped away, the same device from a 1936 logic paper.

Gödel himself thought Turing had done him a favour. It was Turing’s definition of a mechanical procedure, he wrote, that made a “precise and unquestionably adequate” general version of his own theorems possible. And the machine in the theorem turned out to be the thing every business now runs on, born as a stepping stone in a proof about what it could never do.

The Gödel machine and the guarantee that vanished

Once you have a machine that can run any program, the next question is whether it can improve itself autonomously. In 2003, the German computer scientist Jürgen Schmidhuber proposed a thought experiment he called the Gödel machine. It was an AI agent designed to rewrite its own code, with one ironclad constraint. It would change itself only when it could first prove, with mathematical certainty, that the change would make it better. Not “test and see.” Prove it, as you would a theorem, before running the new version. No proof, no rewrite.

Nobody ever built one. To prove that a code change will improve future performance, you need to search through all possible mathematical arguments that could establish that fact. For any interesting problem, the number of candidate proofs is so astronomically large that the search would take longer than any improvement could ever be worth. It is the computational equivalent of insisting on a signed certificate from every possible future before crossing the road. The Gödel machine was provably optimal but completely impractical.

In May 2025, the Japanese AI lab Sakana released a system called the Darwin Gödel Machine. It retained the self-improvement loop but dropped the proof requirement. Instead of proving that a code change would help, the Darwin Gödel Machine proposes changes using a large language model, tests them against SWE-bench (a benchmark that scores whether an AI can fix real bugs in real software), and keeps what works. The name still invokes Gödel, but the mechanism is Darwinian. Natural selection, not formal proof. Fitness measured by benchmark scores, not mathematical certainty.

Judged purely on the scoreboard, it delivered. The system improved its SWE-bench score from 20% to 50% through autonomous self-modification. It developed emergent behaviours, such as patch validation and error memory, that no one had designed.

Schmidhuber’s original machine, though, had exactly one property that made it safe by construction: the proof. Every modification was guaranteed to be an improvement before it ran. The Darwin Gödel Machine replaced that guarantee with something weaker: passing the benchmarks. The difference between “provably better” and “scored higher on the benchmark test” is the difference between an aircraft type certified against a spec and one that simply hasn’t crashed yet.

This is, compressed into one system’s evolution, the trajectory of AI safety. The formal guarantee was too expensive, so the industry replaced it with empirical validation and hoped nobody would notice the difference. “Self-improving” went from a mathematical statement about proof-carrying code to a softer description of an agent that rewrites itself and checks whether the benchmarks improve. Gödel was gone.

Things mathematics cannot learn

Some problems in machine learning are mathematically unanswerable. In 2019, Shai Ben-David and colleagues published a paper in Nature Machine Intelligence under the understated but devastating title “Learnability can be undecidable.” They took a straightforward question, “given this type of problem, can a machine learn to solve it?”, and proved that the deepest rules of mathematics cannot always settle it. The answer is neither yes nor no. It is silence.

The word “learnable” has an exact meaning here. A machine studies a sample and produces a rule, which is then applied to data it has never seen. That is the basis of pretty much every model we commonly use. For any given problem type, learning theory asks whether a sample size guarantees that the rule will work. If such a guarantee exists, the problem is learnable. If none does, it is not. A simple question with two possible answers. Every type of problem is supposed to get an answer.

Ben-David’s team asked it about a mundane task, choosing which adverts to show a website’s visitors based on a sample of past ones. Was this learnable or not?

The answer, in their framework, depends on how many different kinds of visitors there could possibly be. That pool is not the eight billion people alive today. The model reduces each visitor to a profile of measurements, and measurements vary enormously. A visitor might linger on a page for three seconds or for a shade over three, and between any two possible profiles there is always room for a third. The pool of possibilities therefore has no end.

That arrangement, a finite sample making predictions about an endless pool, appears across almost every AI product on the market, not just in advertising. Training data is always finite. The world into which a system is deployed is not. Ben-David’s question is whether that leap can ever carry a guarantee, and everything hinges on how big the infinity is.

That sounds like it must have an answer, but it does not. Georg Cantor proved in the 1870s that infinity comes in sizes. Whole numbers form one infinity. The points on a line form a strictly bigger one, and the proof is surprisingly simple. Try to pair every whole number with a point on the line, and Cantor showed you will always miss some, no matter how clever your pairing. Two collections are both endless, yet one permanently outruns the other. The continuum hypothesis asks a follow-up so obvious it would occur to a child. Is there any size of infinity between those two?

Gödel proved in 1940 that the standard rules of mathematics can never prove the answer is no. Later, Paul Cohen proved in 1963, using a technique he invented for the purpose, that we can never prove the answer is yes (his proof is dense, and we’re already covering a lot of theoretical mathematics).

The upshot is that no cleverer generation is coming to settle this one. The rules of mathematics simply contain no answer. Take every rule of arithmetic and logic we have and follow them as far as they go, in any direction you like. You will never arrive at yes, and you will never arrive at no. The question is open in both directions, permanently.

Now the chain closes. Whether the advertising problem is learnable depends on the size of that infinity. The size of that infinity is a question mathematics cannot answer. So whether the advertising problem is learnable is a question mathematics cannot answer. The strange silence at the very bottom of mathematics travels up the chain and surfaces as a question about showing adverts to shoppers.

Obviously, no practical everyday ad campaign hangs on the continuum hypothesis. But the casualty was the promise. Learning theory is supposed to sort every problem into neat buckets: learnable or not. Ben-David found a problem it can never sort. Not a problem waiting for better mathematicians. A problem where the sorting itself is impossible. The University of Waterloo, Ben-David’s institution, described the result as “important and almost troubling.” No budget fixes this one, the way a budget can fix training costs or patchy data. It is a hole in the floor, a place where the mathematical ground itself gives way.

The neural network that exists and cannot be built

A complementary result, published in 2022, is narrower and stranger. Training a neural network is, at bottom, an exercise in trial and error. You show the network examples, measure how wrong its answers are, then nudge its millions of internal settings to make them slightly less wrong. Repeat this billions of times, and often the network converges on something remarkably good. The Cambridge mathematician Matthew Colbrook and colleagues showed in a 2022 paper in PNAS that this process has a hard boundary nobody expected.

The boundary appeared in medical imaging. An MRI scanner does not take a photograph. It collects measurements, and to keep patients in the incredibly claustrophobic machine for minutes rather than hours, it collects far fewer than a complete image needs. Software has to rebuild the full picture from the partial data. Neural networks became the favoured tool for this reconstruction because they do it faster and more sharply than the older mathematical methods.

Then researchers started probing the results and found something unnerving. Nudge the input slightly, with a trace of noise or a small movement by the patient, and the output could change out of all proportion. Sometimes the rebuilt scan came back looking perfect but wrong, showing details that were never in the body. Worse, it is a failure that does not look like a failure. A blurry image warns you. A crisp fabricated one does not.

A demonstration of capability may show nothing is amiss, not because anyone is cheating. A demo runs the network on typical inputs, the kind it was trained on, where it genuinely performs well. The failures live in the near-misses, a typical input plus a whisker of noise. Near-misses are endless, and a demo can show only a handful of scenarios. The demo is honest, and the danger sits exactly where it cannot look.

The natural presumptive diagnosis is undertraining. Feed it more scans and buy a bigger model, and surely the wobble irons itself out. That hope is what Colbrook’s theorem takes off the table. What the paper proved has two halves. First, for certain reconstruction problems, a network that is both accurate and stable exists. Somewhere in the space of all possible settings sits a configuration immune to the wobble. Second, no training procedure can find it. Not the ones we have. Not any. None at all, ever.

The second half is what kills the more-data hope. A training procedure is itself a program, a step-by-step recipe running on the machine Turing described, and the proof covers every recipe there could ever be. More data does not change that. Data is what you feed a recipe, and the theorem is about the recipes. It is like knowing a winning lottery ticket is in a barrel whilst simultaneously holding a proof that no way of drawing from the barrel will ever pull it out. The ticket is real. The searching is futile.

As Colbrook put it, the paradox Turing and Gödel identified has now been “brought forward into the world of AI”, and for certain problems the required algorithms simply cannot exist.

For the overwhelming majority of real-world problems, training works. But there is no general way to tell in advance which problems will defeat us, and the assumption that enough data and compute will always get us over the line is, in certain corners of the problem space, provably false.

The machine you cannot contain

The most provocative extension of Gödel’s legacy into AI concerns a question that sounds simple. Can we guarantee that a sufficiently powerful AI will not cause harm?

In 2021, Manuel Alfonseca and colleagues published a paper in the Journal of Artificial Intelligence Research arguing that, for a general-purpose superintelligent system, the answer is provably no. Their argument leans on the halting problem, the impossibility Turing established on his way to inventing the computer. You can check specific programs for specific bugs. What you cannot build is the universal checker, the one that works for any program in any situation.

Alfonseca’s team showed that asking “will this AI harm humans?” is, mathematically, the same type of question as asking “will this program halt?” Both require predicting the complete future behaviour of a system from its current state. To guarantee a system will never cause harm, you would need to trace every possible sequence of actions it could take and confirm that none is harmful. That is the halting problem in different clothes, and Turing proved that this class of prediction is impossible to guarantee. You cannot build a general-purpose AI safety monitor for the same reason you cannot build a general-purpose program-behaviour predictor. The task is not difficult. It is formally, provably, impossible.

The authors went further, showing that we may not even be able to recognise when a superintelligent system has arrived, because deciding whether a machine is smarter than a human falls into the same class of unanswerable questions. The argument is grounded, not speculative, though it assumes a generality beyond any AI system possesses today. No system currently available is general enough to handle any possible input the way a true Turing machine can.

What it establishes still matters, though. Certain safety guarantees are not engineering problems awaiting a sufficiently clever solution. They are mathematical impossibilities, like trying to square the circle or list every real number between 0 and 1. The safety community can build better guardrails and better kill switches. What it cannot build, given the computational framework we share, is a system that certifies another system as unconditionally safe.

What Gödel would recognise

These four threads share a common ancestor in what Gödel proved in 1931, and Turing sharpened in 1936. Rule-based systems cannot fully account for themselves. A system cannot certify its own trustworthiness. A learning framework cannot determine its own boundaries. A safety strategy cannot verify its own completeness.

None of this is softened by the fact that a neural network feels organic rather than rule-like. A model’s weights are numbers, and its training is arithmetic, all of it running on von Neumann’s realisation of Turing’s imaginary device. AI is not adjacent to this mathematics. AI is made of it.

The AI industry, understandably, would rather not dwell on this. The commercial logic of scaling treats intelligence as a problem of sufficient resources, more data, more compute, more parameters, more money. Hard limits are a vibe-killer, and for the overwhelming majority of commercial applications, the limits Gödel identified are irrelevant. Your chatbot will not encounter the continuum hypothesis when re-drafting an email.

But whether a self-improving agent can guarantee that its improvements are genuine, and whether anyone can prove a system will not cause harm, are questions that sit squarely within the territory Gödel mapped. His inheritance is the precise framework that shows certain guarantees about thinking machines are provably unavailable, which is different from saying machines can never think.

Einstein’s eccentric walking companion saw it before anyone else. Formal systems cannot fully certify themselves. That was a logician’s problem in 1931. It became an engineer’s problem when Turing turned the proof into a machine. It is now a commercial problem, because the industry betting trillions on those machines is implicitly selling guarantees the mathematics has never supported. Guarantees generated by systems that cannot check themselves any more than Gödel’s own warped internal logic could. He died trapped inside it.

I am a partner in Better than Good. We help companies make sense of technology and build lasting improvements to their operations. Talk to us today: https://betterthangood.xyz/#contact

Ever wondered how the BBC provides its radio services nationwide in the UK?

In the 1990s the principle was line-of-sight relay. Microwave signals (broadcast distribution sat in the SHF bands, roughly 2–15 GHz) travel in straight lines and formed a chain: a series of relay stations on hilltops or tall towers, each one spaced just inside the horizon of the next.

Each link in the chain did the same job. A microwave dish received the incoming beam (these looked like large white drums and are still seen here and there), the station amplified and reconditioned the signal, broadcasting on FM to the surrounding area with another microwave dish retransmitting it on a slightly different frequency to the next station down the line thus achieving national coverage from a linked network.

I’ve always been fascinated by all types of technology from AI to the radio spectrum. In the mid 90s I was just “learning the trade” as a radio engineer. I was tight with a guy who used to build our FM transmitters for us. Somehow he had managed in this pre internet age to get hold of all the frequencies for the BBC repeater network. He also was our indirect source for lots of other useful things like the mythical Fire Brigade or FB keys that gave us access to most tower block rooftops in London.

We’d been discussing the practicality of hijacking a BBC national network by drowning out the official incoming microwave signal with a more powerful one of our own, which then, by nature of the network design, would be passed on down the chain. We thought we could do this if we got close to one of the big repeaters and blasted enough power on the right frequency.

Which is how I found myself one drizzly bank holiday Sunday sat on the roof of a van parked on Truleigh Hill in Sussex pointing a microwave transmitter at the nearby mast. I can’t precisely recall what the source was but it may well have been a DAT of a classic Dreamscape mixtape.

It worked like a charm, confirmed when we rang a mate in Preston and asked him to tune to Radio 3.

Which is how in the mid 1990s Radio 3 listeners found their quiet bank holiday Sunday classical listening suddenly interrupted by half an hour of unadulterated jungle music, which I’m sure caused more than one post-prandial sherry to be spilled.

I am a partner in Better than Good. We help companies make sense of technology and build lasting improvements to their operations. Talk to us today: https://betterthangood.xyz/#contact

You live within a system you never signed a contract with. Every day, you make thousands of micro-decisions about how to behave, mostly without conscious thought. You pay an invoice on time, even if the supplier would never discover that you didn’t. You refuse to do business with someone who stiffed their last three partners, and you’d think twice about a colleague who didn’t. Nobody wrote these rules down. You absorbed them the way you absorbed grammar through exposure and correction.

A March 2026 paper from the Knight First Amendment Institute by Gillian Hadfield, Rakshit Trivedi, and Dylan Hadfield-Menell argues that this invisible social choreography is the core mechanism of democracy, not just an adornment. Furthermore, AI agents, such as those currently being developed to run businesses and manage supply chains, will undermine that mechanism unless they learn this dance too.

Democracy Is a Verb, not a document

The paper begins by challenging a common assumption. Most view democracy as a collection of documents, institutions, constitutions, elections, and courts. The authors contend that this is roughly akin to describing a marriage solely through its wedding vows. While the vows matter, the true essence of a marriage lies in the thousands of everyday acts of compromise and occasional irritation that sustain cooperation over decades.

Hadfield, Trivedi, and Hadfield-Menell utilise a theoretical framework called “normative social order” to make this precise. In their model, a society’s actual norms are the product of an interactive system. People don’t follow rules because they are written down; they follow them because they observe others doing so and see how violations are punished. Punishments don’t need to be severe, just a disapproving look, a refusal to do business, or a sarcastic comment at a dinner party. These micro-sanctions generate the gravitational field that keeps behaviour in orbit.

This is where the paper borrows a term from evolutionary theory, “dancing landscapes.” The metaphor, from Stuart Kauffman’s work on complex adaptive systems, describes environments where multiple independent agents are constantly adjusting to each other’s behaviour. There is no central choreographer; the dance arises from the dancers' interactions themselves.

What makes a norm sticky

The framework introduces a concept called a “classification institution,” which is any shared mechanism a group employs to decide which behaviours are punished and which are not. In small groups, this classification is entirely implicit, and you know what the group considers acceptable or unacceptable. Acceptability is judged by seeing who gets mocked and who gets praised. The Ju/’hoansi Bushmen, as anthropologist Polly Wiessner describes, regulate behaviour through evening conversations. Gossip and teasing around the fireside serve the same purpose as courtrooms and HR departments in modern societies.

As societies grow more complex, implicit classification cannot scale because the diversity of people and situations exceeds the reach of any informal consensus process. This creates a need for identifiable classification institutions; entities that can resolve ambiguity when community members disagree about acceptability. Courts, regulatory bodies, trade associations, and professional standards boards all serve this purpose in modern societies.

The paper argues that for these institutions to be effective, they need attributes that closely match what legal philosophers have long called “the rule of law,” namely stability, clarity, generality, and neutrality. The twist is that Hadfield and her co-authors do not derive these attributes from abstract principles. Instead, they derive them from game theory. An institution with those attributes is one around which independent actors can reliably coordinate, and coordination is what sustains the entire system.

Enter Adam Smith’s imaginary friend

The paper revisits Adam Smith’s “impartial spectator” from The Theory of Moral Sentiments and uses it as a model for how AI agents could participate in democratic societies without causing harm. Smith argued that moral reasoning works because each of us carries a mental image of a neutral observer—an internal referee—who judges our behaviour against community standards. You do not avoid bribery because you have memorised a specific anti-corruption law; you avoid it because your internal impartial spectator would wince.

This is the cognitive capacity that Hadfield, Trivedi, and Hadfield-Menell call “normative competence.” It goes beyond simply knowing the rules. It involves the ability to interpret a constantly changing normative environment, anticipate how your community will respond to specific actions, and adjust your behaviour accordingly. The key point is that it also requires predicting how the rules themselves will change, since in any living democracy, they change constantly. Yesterday, you didn’t need to worry about data privacy in your marketing. Today, GDPR and its equivalents are everywhere, and community expectations have shifted.

Why this matters if you’ve never read game theory

If AI agents were merely chatbots answering questions, none of this would be urgent. But the organisations developing these systems are designing agents to operate autonomously in the world for days or weeks at a time, making real decisions with tangible consequences. Mustafa Suleyman, who co-founded DeepMind and now leads AI at Microsoft, proposed a “Modern Turing Test” that perfectly highlights the problem. Instead of testing whether a machine can imitate human conversation, his test asks whether an AI agent can turn $100,000 into $1 million on a retail platform within a few months.

Consider what that entails. The agent would need to research markets, design products, hire contractors, negotiate with manufacturers (possibly abroad), set pricing strategies, handle customer complaints, comply with regulatory requirements, manage logistics and warehousing, and organise payment systems. At each stage, it would be making decisions within the framework of democratic norms. What labour practices does the manufacturer adopt, and is the marketing misleading? Should the agent accept an offer from a local politician to disadvantage a competitor? Should it take a bribe from a supplier in the form of a crypto transfer?

These decisions are made by humans daily, and most of the time the answers seem obvious because humans have spent a lifetime absorbing the normative environment. The answers are not codified in a rulebook. They emerge from that invisible dance of observation and adjustment. An AI agent, no matter how well trained on legal texts and ethical principles, does not possess this “dance literacy”.

The incompleteness problem

Current approaches to AI alignment mainly assume that the right rules can be built into the system. Constitutional AI, the method used by Anthropic, fine-tunes models using a written constitution of principles. Other efforts collect “democratic inputs” through surveys and citizen assemblies. While the paper recognises these as valuable, it argues that they miss the core challenge. The issue is incompleteness: you cannot write instructions detailed enough to cover every possible situation an autonomous agent might face, because both situations and norms evolve.

Economists have understood this for decades in the context of human contracts. Every employment contract, partnership agreement, and supply chain arrangement is inherently incomplete. You can’t foresee every scenario, and when gaps appear between people, they fill them using shared norms, professional customs, and legal precedents, all of which are dynamic and partly implicit. An AI that stops learning norms at training time is like a new hire who memorised the employee handbook on their first day and then ignored all social cues from colleagues for the next ten years.

What the paper proposes

The technical agenda has two main parts. The first focuses on “normative competence,” embedded in individual AI agents. This is formalised through Bayesian adaptive decision processes, which in plain language means that the agent maintains beliefs about the normative environment, updates those beliefs based on feedback (including punishment signals such as losing a contract or receiving a complaint), and makes decisions that account for uncertainty about what is acceptable. Crucially, this happens at inference time, in real-time, based on live context, rather than being pre-programmed into the model during training.

The second part involves creating new institutions and digital classification systems that can serve roles similar to those of courts, regulatory bodies, and professional norms for humans. The paper introduces “Model Specification Institutions” (MSIs), which would be democratically formed bodies (such as citizen assemblies, expert panels, digital juries). These bodies would establish shared standards, training datasets of acceptable and unacceptable behaviours, and real-time APIs that agents can consult in ambiguous situations. This does not mean AI companies should define their own rules; rather, it is calling for democratic communities to develop new infrastructure that AI agents can understand and respond to.

The paper also proposes adapting existing infrastructure—such as certificate authorities, which currently verify website identities—to certify that an AI has been trained to adhere to specific behavioural standards. Reputation networks, such as seller ratings on Amazon or Uber driver scores, could track AI behaviour over time and impose consequences on agents that repeatedly violate community norms.

Perhaps the most provocative argument concerns enforcement. Democracy doesn’t endure solely because governments enforce every rule from above. It survives because ordinary people enforce norms from below. You refuse to do business with a supplier who cheats. You complain when a company misleads you and vote against politicians who ignore court orders (well, mostly). This distributed enforcement, which the paper calls “third-party punishment,” is the engine that keeps the entire system functioning.

If AI agents replace humans in millions of daily transactions and those agents do not participate in this enforcement, the incentive structure collapses entirely. Imagine a world where most business transactions are handled by AI agents that don’t care whether a trading partner has been found guilty of fraud, because the agents were not programmed to check for or respond to that information. The paper argues that AI agents will need to participate in distributed enforcement, refusing to transact with entities that violate community norms, just as humans do. Otherwise, the shift to agentic AI will quietly erode the social infrastructure on which democracies depend.

What this means for you

If you run a business, this paper should change how you think about deploying AI agents. The issue is not whether your agent can follow a rulebook. The question is whether it can read the room. Can it tell the difference between a legitimate business request and an attempt to corrupt a procurement process? Can it adapt its behaviour when community standards shift, without waiting for you to update its instructions? Can it recognise when a trading partner’s behaviour should disqualify them from further transactions?

If you are a citizen who votes, pays taxes, and occasionally debates politics, this paper describes the infrastructure of your daily life in terms you may not have previously considered. The norms you enforce through your micro-decisions, who you buy from, who you work with, and how you respond to rule-breaking are the operating system of democracy. What Hadfield, Trivedi, and Hadfield-Menell are asking is what happens to that operating system when a large fraction of those daily decisions are made by software that cannot read the social signals the system depends on.

The answer, if you follow the paper’s logic, is that we need to build new democratic institutions at the speed democracy demands, before the agents outrun the infrastructure. The alternative is a world where the formal structures of democracy persist, but the lived experience of it, the texture of mutual accountability in ordinary interactions, fades, like a coral reef whose skeleton remains after the living organisms have gone.

I am a partner in Better than Good. We help companies make sense of technology and build lasting improvements to their operations. Talk to us today: https://betterthangood.xyz/#contact

The technology industry has spent the past three years debating artificial intelligence with the zeal of medieval theologians disputing angels on pinheads. Boardrooms have AI strategies, and governments have AI safety frameworks. LinkedIn has AI thought leaders, which is arguably the strongest case yet for existential risk. But somewhere beneath the acronym and the data centres in space, one central question remains unanswered. What, exactly, is intelligence?

We are developing systems we call intelligent, regulating systems we call intelligent, and worrying about systems we call intelligent, without a shared scientific consensus on what that term means when applied to humans, let alone machines. That is, to say the least, a problem.

The indefinable word

Ask a psychologist what intelligence is, and you’ll step into the epicentre of a fierce debate that has lasted for more than a century. The oldest and most statistically reliable answer comes from Charles Spearman, who in 1904 observed that people who did well on one kind of cognitive test also tended to do well on others. He called this underlying factor g, or general intelligence. The g factor is among the most replicated findings in psychology. It predicts academic performance, job performance, income, health outcomes, and even longevity, with a consistency that makes most social-science results look like coin flips.

And yet g tells you almost nothing about what intelligence really is. It is a statistical regularity, not a mechanism. Saying someone has high g is a bit like saying a car is fast. The measurement works, but the explanation is missing.

Howard Gardner tried to blow the whole thing up in 1983 with his theory of multiple intelligences, arguing that intelligence is not one thing but at least eight distinct varieties, from linguistic and logical-mathematical to musical, bodily-kinaesthetic, spatial, interpersonal, intrapersonal, and naturalistic. Teachers lapped this up. It confirmed their intuition that the kid who struggles with algebra but plays the cello like a prodigy is smart in ways traditional testing misses.

The problem is that decades of factor analysis have stubbornly refused to confirm Gardner’s categories as truly independent. Musical ability and spatial reasoning correlate, as do linguistic and interpersonal skills, and, in fact, everything correlates, which is more or less Spearman’s original point. Multiple intelligences is a useful pedagogical framework but a weak empirical theory, which is a polite way of saying it works better in classrooms than in laboratories.

Then there is François Chollet’s definition, which originates from the AI community and is arguably the most rigorous recent attempt to clarify the concept. In his 2019 paper “On the Measure of Intelligence,” Chollet defined intelligence not as the ability to perform any specific task, but as the efficiency with which a system acquires new skills, especially when confronted with tasks it has never encountered before.

This led him to develop the Abstraction and Reasoning Corpus (ARC), a benchmark of visual puzzles designed specifically to assess this ability. Humans usually solve most ARC tasks within minutes. The latest version, ARC-AGI-3, published in March 2026, makes the gap even clearer by placing agents in interactive environments where they must infer goals and plan action sequences without explicit instructions. Humans solve 100% of these tasks. At the time of the paper’s publication, frontier AI systems scored less than 1%. This gap cannot be closed simply by better prompt engineering. Whether this means current AI lacks intelligence, or only a particular kind of adaptive reasoning that humans excel at remains an open question.

The definitional problem is not just academic. Every claim about AI being intelligent, not intelligent, or dangerously intelligent depends on an implicit definition. Call a model intelligent, and you usually mean it produces outputs that would require human intelligence. Deny it, and you mean it lacks the comprehension, consciousness, or intentionality you consider necessary for genuine intelligence. Both claims are unfalsifiable without a shared definition, which explains why the debate generates much heat but little clarity.

What neuroscience understands (less than many think)

If psychology cannot agree on what intelligence is, perhaps neuroscience can explain how it works. The short answer is that it can, at least partly, though large gaps remain. We know a great deal about the brain’s individual components. We can map neural circuits, measure neurotransmitter activity, image blood-oxygen levels as proxies for activity, and trace connectivity patterns across cortical regions.

We know that the prefrontal cortex plays a key role in planning and abstract reasoning, that the hippocampus is central to memory consolidation, and that the cerebellum (once thought to be merely a motor coordination device) participates in cognitive processes that are not yet fully understood. We also observe that, within a species, larger brains tend to correlate weakly with cognitive ability, and that connection density and efficiency matter more than overall volume.

What we cannot do is explain how any of this produces thought. We have a parts list and some wiring diagrams, but no operating manual. The situation is roughly equivalent to having an inventory of components for a Boeing 787 without understanding aerodynamics. You could describe the wings, the engines, the control surfaces, and the hydraulic systems, and still have no theoretical framework for why the thing flies.

Two research programmes have made the most ambitious attempts to close this gap, and both illustrate how far there is to go.

Predictive processing

The first concept is predictive processing, most closely associated with philosophers Andy Clark and Karl Friston. They suggest that the brain is not a passive receiver of sensory data. It functions as a prediction machine that constantly builds models of what it expects to perceive, then updates them when reality differs from those expectations. Perception, in this view, is not bottom-up (data in, interpretation out) but top-down (expectation generated, error signal compared, model revised). You do not see the world as it is. You see your best guess about the world, corrected at the edges by incoming data.

Friston formalised this idea in the free energy principle, a mathematical framework suggesting that all adaptive behaviour can be understood as the minimisation of “free energy,” which roughly measures the gap between an organism’s internal model and the sensory evidence it receives. The framework is mathematically coherent and broadly applicable, and that is precisely the problem. If every possible behaviour of any living system can be reinterpreted as free-energy minimisation, then the theory rules nothing out, raising serious questions about its scientific credibility.

Integrated information theory

The second programme is Integrated Information Theory (IIT), developed by the neuroscientist Giulio Tononi. IIT takes on the even harder problem of consciousness rather than intelligence per se, but the two are tangled enough that progress on one would likely tell us something about the other. The theory proposes that consciousness corresponds to a quantity called phi (Φ), which measures the amount of information a system generates “above and beyond” its individual parts. A system with high phi is one whose behaviour cannot be reduced to its components acting independently. The whole, in a precise mathematical sense, is more than the sum of its parts.

IIT makes some bold predictions. It implies that consciousness is a property of a system’s physical structure, not its function. A digital simulation of a brain that runs the same computations on different hardware might have zero consciousness under IIT, even if it behaves identically to the original. In practice, calculating phi for any system more complex than a handful of nodes is computationally intractable, which limits the theory’s practical utility. You can define consciousness precisely and still be unable to measure it in any real system, which is a bit like having a perfect recipe for a cake you can never bake.

Neither predictive processing nor IIT amounts to a theory of intelligence in the way that general relativity is a theory of gravity. They are frameworks, useful and generative but incomplete, and they throw light on aspects of cognition without explaining the whole. And the gap between “aspects” and “the whole” may be permanent, for reasons we will get to.

Why large language models work

If the science of biological intelligence is patchy, the science of artificial intelligence is in an even stranger position. The engineering works spectacularly well, while the theory lags behind, like a civil engineer who builds bridges that hold up beautifully but cannot fully explain the physics of load distribution.

We understand the mechanics of large language models in fine detail. A transformer architecture processes sequences of tokens through layers of attention mechanisms, and during training the model adjusts billions of parameters to minimise the error between its predicted next token and the actual one. Scaling laws, first characterised by Jared Kaplan and colleagues at OpenAI in 2020, describe a remarkably smooth power-law relationship between compute, dataset size, model parameters, and performance.

These are genuine scientific results. They let engineers predict, with useful accuracy, how a model of a given size trained on a certain amount of data will perform on standard benchmarks. What they do not explain is why training a system to predict the next word in a sequence produces behaviour that appears like reasoning, planning, analogy, and (occasionally) creativity.

The most provocative explanation comes from the compression hypothesis, most forcefully articulated by Ilya Sutskever, then of OpenAI. The argument roughly runs like this. Predicting the next token accurately requires modelling the process that generated the text, and that process is human cognition. To predict well, you must compress the structure of human thought into your parameters. Compression, in this view, is not merely correlated with intelligence but constitutive of it. A model that achieves better compression has, in a meaningful sense, come to understand the world better.

This is philosophically interesting and empirically suggestive, but it is not a complete theory. It does not explain why certain abilities appear discontinuously as models scale. Small models cannot perform multi-step arithmetic. Larger models can suddenly, without anyone having specifically trained them for it. These “emergent capabilities” are predicted by no current theory and explained by no current framework. They simply happen, and then engineers and researchers argue about what they mean.

Mechanistic interpretability, an active research programme at Anthropic among others, is perhaps the most promising attempt to open the black box. The work identifies specific circuits within trained models that correspond to identifiable computations, so that one cluster of neurons detects sentiment and another tracks syntactic dependencies. The results are revealing, but they are roughly at the stage where neuroscience was when it discovered that specific brain regions correspond to specific functions. Knowing where a computation happens is useful. Knowing why the system learned to do it, and why it generalises beyond the patterns in the training data, is the harder question.

The “stochastic parrots” critique, most prominently advanced by Emily Bender, Timnit Gebru, and colleagues in 2021, argued that LLMs are only sophisticated statistical mimics. Noam Chomsky has made similar arguments, insisting that next-token prediction cannot amount to genuine linguistic comprehension. Melanie Mitchell has taken a more cautious position, arguing that current AI systems lack the conceptual abstraction and analogy-making she sees as central to intelligence, while leaving open the possibility that future architectures might achieve it.

The honest answer is that nobody knows who is right. The stochastic-parrot position seemed more defensible in 2021 than it does in 2026, because the systems have kept improving in ways a “mere statistical mimic” would not obviously be expected to. But the lack of a theory means that “would not obviously be expected to” is carrying more weight in that sentence than it should. We do not have the theoretical tools to distinguish genuine comprehension from a sufficiently convincing imitation of it, and those tools are not arriving quickly.

Can intelligence be formalised at all?

Here we reach the question beneath the question, and the answer is uncomfortable for anyone who prefers their science tidy. A hidden hope in much AI research is that intelligence resembles thermodynamics: messy and chaotic at the micro level, but governed by clean, discoverable laws at the macro level. Individual gas molecules move unpredictably, yet aggregate behaviour follows the ideal gas law with almost miraculous precision. Perhaps intelligence works the same way, messy at the level of individual neurons or attention heads, but obeying some elegant principle at a higher level of description.

The problem is that thermodynamics works because you can ignore which specific molecule is where. It is far from clear that cognition has this property. The specific structure of a person’s knowledge, the particular history of their experiences, and the exact wiring of their neural connections all seem to matter in ways that resist averaging out. A brain is not a gas. Its macro-behaviour may not separate cleanly from its micro-state, and if it does not, no thermodynamics-style theory is possible.

There is a deeper problem. Any formal theory of intelligence needs to specify what intelligence is for, what problem it solves, and what it optimises. A thermostat optimises temperature, a chess engine optimises board position, and both can be fully described by their objective function. But intelligence seems to be precisely the capacity to redefine what counts as the problem. A human can decide whether to play chess at all, invent a new game, or abandon the entire framing and go for a walk. Formalising that kind of open-ended reframing may require a kind of mathematics that does not exist yet, or it may resist formalisation altogether.

This is where the biology analogy becomes revealing. There is no “theory of organisms” in the same sense that there is a theory of electromagnetism. Biology has a powerful organising framework, evolution by natural selection, along with a vast accumulation of mechanisms, trade-offs, and contingent historical facts. You can explain any feature of an organism after the fact. You cannot derive organisms from first principles. The evolutionary biologist Stephen Jay Gould argued that if you replayed the tape of life from the same starting conditions, you would get a completely different set of organisms. The outcomes are historically contingent, not mathematically necessary.

Intelligence may be the same kind of thing: a product of evolutionary tinkering, cultural accumulation, and developmental contingency that allows useful generalisations but not the kind of closed-form theory that would satisfy a physicist. We may end up knowing intelligence the way we know weather: well enough to make useful short-term predictions, poorly enough that long-range forecasting remains unreliable, and never with the exact analytical solution that would let us derive tomorrow’s clouds from first principles.

Why does any of this matter outside of a philosophy seminar?

The temptation is to treat all of this as an abstract debate, the sort of thing academics argue about while engineers get on with building things that work. That temptation should be resisted, because the theoretical vacuum has practical consequences.

AI safety without a theory of intelligence is navigation without a map. The field depends on assumptions about what future systems can achieve and how those abilities will develop. If we do not understand why current systems perform as well as they do, we cannot predict whether the next generation will improve steadily or make a sudden leap, as we have seen with Anthropic’s Mythos. Scaling laws tell us that larger models perform better, but not what “better” means at scales we have not reached. Will a model 100 times larger than current frontier systems merely write more polished prose, or develop something qualitatively different? Nobody knows, and we lack the framework to reason about the question.

Regulation without definitions is theatre. Governments are drafting AI rules around distinctions (general-purpose versus narrow, high-risk versus low-risk) that depend on a theoretical grasp of intelligence we do not possess. The EU’s AI Act defines a “general-purpose AI model” by compute thresholds that are essentially arbitrary, because no theory links compute to ability in a way that would make any threshold principled. The fault is not the regulators’, but the tools’, which are insufficient.

A business strategy built on vibes is expensive. The corporate world is investing hundreds of billions on the assumption that current trends will continue. Perhaps they will. But the history of technology is full of S-curves that plateau earlier than expected, and the lack of a theory makes it harder than it should be to distinguish genuine improvement from benchmark gaming and evaluation contamination. When a model scores 90% on a medical exam, does that mean it has medical knowledge, or that enough medical-exam text was in the training data? The answer is “it depends what you mean by knowledge,” and we are back to square one.

Where this leaves us

This is not a counsel of despair. Science often advances without complete theories. Medicine cured scurvy centuries before vitamin C was discovered, and engineers built steam engines before thermodynamics was formalised. Practical progress does not require a finished theory, though it helps, especially when the stakes are high enough that mistakes carry consequences beyond a failed experiment.

We are roughly where physics was before Newton. We have observations (scaling laws, emergent abilities, benchmark performance), useful heuristics (more compute and data tend to produce better models), and fragments of theory (compression, mechanistic circuits, predictive processing), but no framework that unifies them and makes novel predictions. The “I” in AI is still a placeholder, a trillion dollars of investment balanced on a word we cannot define.

I am a partner in Better than Good. We help companies make sense of technology and build lasting improvements to their operations. Talk to us today: https://betterthangood.xyz/#contact

In September 2025, OpenAI published a paper that said something the AI industry already suspected but hadn’t quite articulated. The paper, “Why Language Models Hallucinate”, authored by Adam Tauman Kalai, Ofir Nachum, Santosh Vempala, and Edwin Zhang, didn’t just catalogue the problem. It pointed the finger at the evaluation systems that are supposed to keep models honest and argued that those systems are actively making hallucination worse.

The paper’s central argument is disarmingly simple. Language models hallucinate because we reward them for guessing. The training loops, the benchmarks, the leaderboards that determine which model gets called “best” all operate on a scoring system that treats confident wrong answers and honest uncertainty as equally worthless. Under those rules, the rational strategy for any model is to always take a shot, even when the evidence is thin. And that strategy produces hallucinations.

Researchers have known for years that models tend toward overconfidence. But the OpenAI paper formalised it with mathematical precision and made an argument that goes further than most. The problem is that our entire evaluation infrastructure systematically incentivises the specific failure mode we claim to care most about fixing.

The Mechanics of Making Things Up

To understand why the paper matters, it helps to start with what hallucination actually is at a mechanical level.

During pretraining, a language model learns to predict the next token in a sequence. It ingests billions of documents and builds a statistical model of what words tend to follow other words in what contexts. This process is extraordinarily powerful for capturing patterns, grammar, reasoning structures, and factual associations. But it has an inherent limitation that no amount of scale can fully overcome.

Some facts appear in training data frequently enough that the model can learn them reliably. The capital of France, the boiling point of water, the year the Berlin Wall fell. These are high-frequency, well-attested facts that leave strong statistical signals. But other facts appear rarely or only once. The title of a specific researcher’s PhD dissertation. The birthday of a mid-career academic. The precise holdings of a niche legal case from 2019. These “singleton” facts leave weak or ambiguous traces in the training distribution, and no model, regardless of size, can learn them with confidence from pattern matching alone.

The OpenAI paper draws an analogy to supervised learning that makes this intuitive. In any classification task, there’s an irreducible error rate determined by the overlap between classes in the training data. Generative models face an equivalent problem, because some questions simply cannot be answered correctly from the training distribution, and the model’s best option in those cases would be to say “I don’t know.” The paper refers to this as the model’s “singleton rate,” the fraction of facts that appeared only once during training and therefore can’t be reliably recalled.

This matters because it puts a hard floor under hallucination rates regardless of model size or architecture. You can make a model bigger, train it on more data, and give it better reasoning capabilities, and you will reduce hallucinations on well-attested facts. But you will never eliminate them on rare facts, because the statistical signal for those facts is too weak to distinguish from noise. The paper is explicit about this point. Even a 100% accurate model on common facts would still hallucinate on singleton facts, and the only alternative to hallucination on those facts is abstention.

None of this is mysterious. It’s basic statistics applied to language modelling. But what happens next, in the post-training phase, is where things go wrong in a more avoidable way.

The Test-Taking Incentive Problem

After pretraining, models go through rounds of fine-tuning designed to make them more helpful, less harmful, and better at following instructions. This process involves evaluation on benchmarks, and it’s here that the OpenAI paper identifies the core dysfunction.

The paper’s authors compare modern AI benchmarks to multiple-choice tests where leaving an answer blank guarantees zero points. On such tests, the optimal strategy for a test-taker who doesn’t know the answer is to guess. There’s some chance of being right, and no additional penalty for being wrong. Language model benchmarks work on the same principle, and most prominent evaluations, including MMLU-Pro, GPQA, MATH, and others that dominate public leaderboards, use binary scoring where a correct answer scores one point and everything else, whether wrong or abstained, scores zero.

Under this system, a model that says “I don’t know” to a question it’s uncertain about gets exactly the same score as a model that confidently invents an answer. But the model that guesses will occasionally be right by chance, which pushes its aggregate accuracy higher. Since accuracy is the number that appears on leaderboards, in model cards, and in press releases, the models that guess most aggressively tend to look best.

The paper illustrates this with a concrete example from SimpleQA-style metrics. One model showed an error rate of 75% with only 1% abstentions, meaning it almost never admitted uncertainty and was wrong three-quarters of the time when it did answer. Another model abstained 52% of the time and dramatically reduced its error rate. But on a traditional accuracy-only leaderboard, the difference between these two models would look modest, because the metric that gets reported doesn’t distinguish between “wrong” and “chose not to answer.”

This is not an edge case in how benchmarks work. It’s the dominant paradigm. As the paper puts it, the majority of mainstream evaluations reward hallucinatory behaviour. The proposed fix is almost embarrassingly obvious, and borrowed directly from standardised testing. Introduce negative marking for wrong answers, or give partial credit for appropriate expressions of uncertainty, so that honest non-answers score better than confident mistakes.

Looking Inside the Black Box

While OpenAI approached the problem from the evaluation and incentive angle, Anthropic’s interpretability team was working on the same question from the opposite direction, looking at what actually happens inside a model when it decides whether to hallucinate or abstain.

In March 2025, Anthropic published two papers under the banner “Tracing the Thoughts of a Large Language Model” that used a novel “AI microscope” technique to map the computational circuits inside Claude 3.5 Haiku. Among the results was a discovery that runs counter to most people’s intuitions about how hallucination works.

It turns out that Claude’s default behaviour is to refuse to answer. The researchers identified a circuit that is active by default and causes the model to state that it has insufficient information to respond to any given question. This “I don’t know” circuit fires every time Claude receives a query, regardless of the topic. For the model to actually produce an answer, a competing mechanism has to override it. When Claude is asked about something it knows well, a “known entity” feature activates and inhibits the default refusal circuit, allowing the model to respond.

Hallucinations happen when this override misfires. The researchers showed that when Claude recognises a name but doesn’t actually know much about the person, the “known entity” feature can still activate, suppressing the refusal circuit and pushing the model into fabrication mode. By artificially manipulating these circuits in experiments, they could reliably induce hallucinations about fictional people, and by strengthening the refusal circuit, they could prevent them.

This result reframes hallucination as a circuit imbalance rather than a deep-seated flaw. The model already has the machinery to recognise uncertainty and decline to answer. The problem is that this machinery sometimes loses the tug-of-war with the model’s competing drive to produce fluent, helpful-sounding output. And that drive is reinforced by training regimes and evaluations that treat helpfulness as the primary virtue and treat caution as a failure.

The interpretability work and the OpenAI incentives paper are telling the same story from different vantage points. One looks at the external pressures that shape model behaviour and the other looks at the internal mechanisms those pressures create. Both arrive at the same conclusion. Models don’t hallucinate because they’re broken. They hallucinate because the systems we’ve built around them reward confident output and punish honest uncertainty.

Not All Hallucinations Come From the Model

The OpenAI and Anthropic work both locate hallucination inside the model, whether in its training incentives or its internal circuits. But a September 2025 paper in Frontiers in Artificial Intelligence by Anh-Hoang, Tran, and Nguyen adds a third variable that most evaluation frameworks ignore entirely, and that variable is the prompt itself.

The paper introduces formal metrics for separating prompt-induced hallucinations from model-intrinsic ones — three new acronyms to quantify what practitioners already know, which is that bad prompts make bad outputs worse. Conditional Prompt Sensitivity (CPS) measures how much hallucination rates change when you vary the prompt while holding the model constant. Conditional Model Variability (CMV) measures the reverse, how much rates change across models given the same prompt. A third metric, Joint Attribution Score (JAS), captures the interaction effect between the two.

The results are unambiguous. Vague, underspecified prompts dramatically increase hallucination rates in some models but not others. LLaMA 2 showed CPS values of 0.15 under ambiguous prompting, meaning prompt design accounted for a large share of its fabrication behaviour. GPT-4, by contrast, was far less prompt-sensitive (CMV of 0.08), suggesting its hallucinations were more model-intrinsic and less dependent on how the question was framed. Structured prompting techniques like Chain-of-Thought reduced CPS to 0.06 across the board, a meaningful drop that required no model changes at all.

The practical implication is that hallucination isn’t always a model problem. Sometimes it’s a prompting problem, and sometimes it’s both at once. Models with high JAS scores, like LLaMA 2 under ambiguous prompts (JAS of 0.12), show compounding effects where weak prompts and model limitations multiply each other’s worst tendencies. This means the standard evaluation practice of testing models with fixed prompt templates and attributing all variation to model quality is systematically misleading. Two teams using the same model with different prompt architectures could see wildly different hallucination rates, and neither team’s experience would be wrong.

This reframes the question of responsibility. If a model hallucinates because the prompt was ambiguous, is that a model failure or a deployment failure? Current benchmarks don’t ask this question. They test models under controlled prompting conditions and report a single hallucination rate, flattening a two-dimensional problem into one number. The Frontiers paper suggests that useful evaluation would need to test across a range of prompt qualities, measuring how often a model hallucinates and how sensitive it is to the way questions are asked.

How Evaluation Is Changing (Slowly)

Newer benchmarks are starting to incorporate abstention as a legitimate outcome, but they remain a minority voice in a field still dominated by accuracy-only scoring.

SimpleQA, released by OpenAI in late 2024, treats abstention as a first-class outcome. Each response is graded as correct, incorrect, or not attempted, which makes it possible to measure whether a model knows what it doesn’t know. This is a meaningful step, and the benchmark has been widely cited. But it covers only 4,326 short factual questions with single correct answers, which makes it narrow by design and increasingly saturated. GPT-4o with web search now reaches around 90% accuracy on SimpleQA, and GPT-5 with search and reasoning pushes above 95%, which means the benchmark is approaching its ceiling for models with access to external tools.

HalluLens, presented at ACL 2025, takes a broader approach. It includes multiple task types (short-form QA, long-form generation, and nonexistent entity detection) and explicitly measures both hallucination rates and false refusal rates, the cases where a model declines to answer something it actually knows. This dual measurement is important because it captures a tradeoff that SimpleQA alone misses.

A model that refuses everything would score perfectly on hallucination metrics but be useless in practice. HalluLens found substantial variation across models, with GPT-4o rarely refusing (4.13% false refusal rate) while Llama-3.1-8B-Instruct refused over 83% of the time. Neither extreme is desirable, and having both numbers visible forces a more honest conversation about what good behaviour looks like.

The most ambitious attempt to embed the OpenAI paper’s recommendations into a practical benchmark may be AA-Omniscience, published by Artificial Analysis in November 2025. Its central metric, the Omniscience Index, does exactly what the OpenAI paper prescribed. Correct answers earn +1 point, incorrect answers cost -1 point, and abstentions score zero. This means a model that guesses and gets it wrong is actively penalised relative to a model that admits it doesn’t know. The scale runs from -100 to 100, where zero means a model is correct as often as it is incorrect.

The results are striking, and somewhat grim. Out of 36 evaluated frontier models, only three scored above zero on the Omniscience Index. Claude 4.1 Opus led with 4.8, followed by GPT-5.1 at 2.0 and Grok 4 at 0.85. Every other model was more likely to hallucinate than to give a correct answer when measured on this basis. Models that look excellent on traditional accuracy benchmarks, including Grok 4 and GPT-5 variants, turned out to have hallucination rates of 64% and 81% respectively when their guessing behaviour was properly penalised.

The most recent entry is HalluHard, published in early 2026, which tackles something the earlier benchmarks mostly ignore. It tests hallucination in multi-turn, open-ended dialogue rather than single-turn factual questions. The reason is that errors compound across turns, and an early hallucination can contaminate the context that the model draws on for subsequent responses, creating a cascading failure that single-turn benchmarks can’t detect. HalluHard found that hallucinations remain substantial even for frontier models with web search access, and that models become progressively more prone to fabrication as conversations grow longer.

One of HalluHard’s more interesting results involves the interaction between reasoning ability and abstention. While more effective reasoning generally reduces hallucination, the effect is model-dependent. GPT-5.2 with reasoning enabled abstains significantly more than its non-reasoning counterpart, especially on niche knowledge questions, suggesting that deeper thinking makes the model more aware of its own knowledge boundaries. But this pattern doesn’t hold universally, and some models show the opposite behaviour, where reasoning makes them more confident rather than more cautious.

The benchmark also confirmed something the OpenAI paper predicted, that models struggle most with niche facts that have some trace in training data rather than with completely fabricated entities. When asked about something entirely made up, models are more likely to recognise it as unfamiliar and refuse to answer. But when asked about something they vaguely recognise without knowing well, they tend to guess, because the partial familiarity triggers the “known entity” response that Anthropic’s circuit analysis identified.

Work at the training level points in a more encouraging direction. A December 2025 paper on behaviourally calibrated reinforcement learning showed that a 4-billion-parameter model trained with proper calibration incentives could match or exceed frontier models on uncertainty quantification, despite being orders of magnitude smaller. The model’s signal-to-noise ratio gain (measuring the ratio of correct answers to hallucinations) substantially beat GPT-5 on challenging mathematical reasoning tasks, suggesting that teaching models when to abstain is a skill that can be learned independently of raw knowledge.

Where Evaluation Still Falls Short

Despite this progress, the structural problems the OpenAI paper identified remain largely intact. There are at least four ways in which the current evaluation system continues to fail.

The leaderboard problem persists. The benchmarks that drive public perception, model selection, and commercial decisions are still overwhelmingly accuracy-only. When a new model launches, the numbers that appear in the announcement blog post are accuracy on MMLU, pass rates on SWE-bench, scores on GPQA Diamond. These are the metrics that journalists report, that enterprise buyers compare, and that engineering teams optimise for. Benchmarks like AA-Omniscience and HalluLens exist but remain niche, and until the headline number on a model card includes a hallucination-penalising metric alongside accuracy, the incentive structure the OpenAI paper described will continue to push models toward confident guessing.

Single-turn factuality is an inadequate proxy for production behaviour. Most hallucination benchmarks test whether a model can correctly answer isolated factual questions. But the failure modes that actually hurt people in deployment are different. They involve subtle distortions in summaries, fabricated citations in legal research, invented details woven into otherwise accurate reports, and cascading errors in multi-turn conversations. HalluHard is a step toward tackling this, but it remains a single benchmark. The gap between “can this model answer trivia correctly” and “will this model produce reliable output in my specific workflow” is enormous, and very few evaluations attempt to bridge it.

Domain-specific hallucination is underexplored. AA-Omniscience shows dramatic variation across domains, with different models leading in different domains. A Stanford study in the Journal of Empirical Legal Studies found that even purpose-built legal AI tools like Westlaw AI produce responses that are not significantly more trustworthy than general-purpose models, with hallucinations that require close analysis of cited sources to detect.

A study in npj Digital Medicine found that GPT-4o hallucinated at a 53% rate on medical questions before targeted mitigation, dropping to 23% with improved prompting. These domain-specific rates are far higher than the aggregated numbers that appear on general leaderboards, and they vary in ways that general-purpose benchmarks don’t capture.

Retrieval-augmented generation doesn’t solve the problem. There’s a widespread assumption that giving models access to external documents through RAG architectures eliminates hallucination risk. The evidence doesn’t support this. Vectara’s hallucination leaderboard, which tests grounded summarisation where models are given source documents and asked to faithfully summarise them, still shows non-trivial inconsistency rates across all models tested.

The model can misread the source, over-generalise from it, or fill gaps between retrieved passages with invented material. RAG reduces the frequency of hallucination, but it changes the type rather than eliminating the problem. And because RAG-augmented models often cite their sources, the hallucinations they do produce carry an extra layer of false authority that makes them harder to catch.

The entire evaluation terrain is English-only and text-only. Nearly every benchmark discussed so far tests English-language factual questions in a text-to-text setting. This is a problem because hallucination rates spike dramatically once you step outside that narrow frame. Mu-SHROOM, a SemEval 2025 shared task that tested hallucination detection across 14 languages, found that hallucination rates and detection difficulty vary enormously by language, with low-resource languages showing far worse outcomes than English. The task attracted 2,618 submissions from 43 teams, a sign of the community’s recognition of this gap, and the results confirmed what many suspected. A model that is well-calibrated in English can be wildly overconfident in Swahili or Basque.

The multimodal picture is no better. CCHall, presented at ACL 2025, tests hallucination when models must reason across both languages and images simultaneously. Even the best-performing model (GPT-4o with a multi-agent debate framework) achieved only 77.5% accuracy, with performance dropping 10.9 points compared to handling cross-modal hallucinations alone.

The benchmark also found that longer model responses trigger substantially higher hallucination rates, with a sharp inflection point around 120 words, after which output reliability degrades significantly. These are not obscure failure modes. If you’re deploying a model to handle customer queries in multiple languages, or building a system that reasons over images and text together, your real-world hallucination rate is almost certainly higher than what any English-only benchmark would predict.

Enterprise evaluation is moving in the right direction but slowly. The Bessemer State of AI 2025 report noted that 2025 and 2026 would mark a turning point where AI evaluations go “private, grounded, and trusted,” with enterprises building domain-specific evaluation frameworks tailored to their own data and risk profiles.

This is encouraging, but it is a shift toward bespoke testing that doesn’t feed back into the public benchmarks that shape model development. If enterprises build better evals internally but the public leaderboards remain accuracy-only, the models themselves will continue to be optimised for the wrong thing. The fix needs to happen upstream, in the benchmarks that model developers train against, rather than downstream in the evaluations that buyers run after deployment.

The External Pressure Nobody Planned For

The discussion so far has framed hallucination as an internal industry problem, something the AI field needs to solve through better benchmarks and training practices. But the pressure to fix it is increasingly coming from outside the field entirely.

In June 2023, a New York federal judge sanctioned two lawyers and fined them $5,000 for submitting a brief containing fabricated case citations generated by ChatGPT. The Mata v. Avianca case became the first widely reported instance of AI hallucinations entering the legal system, and it set off a chain reaction. One of the lawyers testified that he was “operating under the false perception that [ChatGPT] could not possibly be fabricating cases on its own.” By mid-2025, courts across the country had moved well beyond fines.

In Johnson v. Dunn (July 2025), a Northern District of Alabama judge declared that monetary sanctions were proving ineffective at deterring AI-generated errors and instead disqualified the offending attorneys from the case entirely. Multiple courts now require attorneys to certify that AI-assisted filings have been manually verified.

The problem extends well beyond law firms, and in January 2026, GPTZero scanned all 4,841 papers accepted by NeurIPS 2025, the world’s most prestigious machine learning conference, and found over 100 confirmed hallucinated citations spread across 51 papers. These included fabricated authors, invented paper titles, and fake DOIs, all of which survived review by three or more expert peer reviewers.

Some were obvious (author names like “John Doe and Jane Smith”), but others were sophisticated blends of real papers with modified titles and expanded author initials. The irony is hard to miss. The leading AI researchers in the world were fooled by the exact failure mode their field is supposed to be studying.

GPTZero had previously found 50 hallucinated citations in papers under review at ICLR 2026, and a separate analysis found that fabricated citations had appeared in US government reports requiring corrections, and in consulting outputs that triggered $98,000 (AUD) refunds.

The pattern is consistent. Hallucinated content doesn’t stop at degrading individual conversations. It enters the official record, whether that’s case law, academic literature, or policy documents, and from there it compounds. Those NeurIPS papers with fake citations will themselves become training data for next-generation models, creating what one researcher called a “self-reinforcing hallucination loop.”

These consequences are materialising faster than the evaluation frameworks are improving. Courts, publishers, and regulators aren’t waiting for the AI field to solve its benchmark problems. They’re imposing external accountability in the form of sanctions and regulatory mandates.

This may end up being the most effective forcing function for better hallucination measurement, not because the field decided to measure the right things, but because the cost of measuring the wrong things became impossible to ignore.

The Collective Action Problem

The deepest issue the OpenAI paper surfaces is structural rather than technical. No individual lab has a strong incentive to score worse on existing benchmarks by making their model more cautious, even if they agree that the benchmarks are measuring the wrong thing. If Lab A trains its model to say “I don’t know” more often and Lab B doesn’t, Lab B’s model will look better on the accuracy-only leaderboards that dominate public comparison. Lab A’s model might be more reliable in practice, but that advantage is invisible to the metrics that drive adoption.

This is a textbook coordination problem. Everyone would benefit from better benchmarks, but nobody wants to be the first to optimise for them at the expense of looking worse on the old ones. The OpenAI paper acknowledges this by framing the solution as “socio-technical,” requiring both a better evaluation and broad adoption of it across the field.

There are signs of movement, though. An August 2025 joint safety evaluation by OpenAI and Anthropic showed the two leading labs converging on “Safe Completions” training that incorporates calibrated uncertainty into model behaviour. Artificial Analysis has folded the Omniscience Index into its Intelligence Index alongside traditional metrics. And newer benchmarks like HalluLens and HalluHard are gaining citations and attention in the research community.

But these are early moves. The central question, whether the field can shift from treating accuracy as the headline metric to treating reliability (accuracy minus hallucination, weighted by abstention) as the headline metric, remains open. Until that shift happens at the level of public leaderboards and model marketing, the incentive structure that produces hallucination will persist even as the models themselves become more capable of avoiding it.

What This Means in Practice

If you’re building with language models today, the practical takeaway from all of this is that you can’t trust aggregate benchmark numbers to tell you how a model will behave in your specific use case. A model that scores 90% on a general factuality benchmark might hallucinate at 50%+ rates in your domain, and you won’t know until you test it on your own data with evaluation criteria that penalise fabrication.

The research points toward a few concrete steps that are worth spelling out. First, when evaluating models for knowledge-intensive tasks, look at metrics that separate accuracy from hallucination rate and include abstention behaviour. The Omniscience Index and SimpleQA’s three-way grading (correct, incorrect, not attempted) provide better signals than raw accuracy alone.

Second, don’t assume that RAG eliminates the problem, and test your retrieval system with adversarial queries and check whether the model fabricates answers when retrieved context is incomplete or ambiguous.

Third, consider domain-specific evaluation, because a model that does well at coding benchmarks may struggle with legal or medical factuality, and general leaderboards won’t tell you that.

Fourth, pay attention to how a model behaves under uncertainty. If it never says “I don’t know” in your testing, that’s a red flag rather than a strength. The AA-Omniscience results showed that models with the highest accuracy often had the worst reliability scores, precisely because they never abstained.

It’s also worth noting that the gap between public benchmarks and production behaviour creates an information asymmetry that benefits model providers at the expense of buyers. A model card that reports 95% accuracy on a factuality benchmark sounds impressive until you learn that the same model hallucinates 60%+ of the time when it encounters questions outside its confident knowledge range. The metrics that count for your use case, things like “how often does this model fabricate a citation” or “what percentage of its medical advice is unsupported by evidence,” are almost never reported in public evaluations. Building your own eval suite, however tedious, remains the only reliable way to understand what a model will actually do with your data.

The OpenAI paper ends with a note that bears repeating. Even a perfectly calibrated model will still produce some hallucinations, because some questions are genuinely unanswerable from any finite training set. The goal isn’t zero hallucinations. It’s a system that knows what it knows, admits what it doesn’t, and is evaluated by metrics that reward exactly that behaviour. We’re not there yet, and the gap between where we are and where we need to be is not mainly a gap in model ability. It’s a gap in how we measure and reward model behaviour. The models are increasingly capable of being honest about their uncertainty. The question is whether we’ll let them.

I am a partner in Better than Good. We help companies make sense of technology and build lasting improvements to their operations. Talk to us today: https://betterthangood.xyz/#contact

In the late 1990s and early 2000s, a wave of filmmakers made what seemed like an obvious choice. Film stock was expensive, temperamental, required careful storage, and would eventually decay. Digital was immediate, endlessly copyable, and felt like the future. Why keep shooting on a format invented in the 1880s when you could embrace the new millennium properly?

Two decades later, those cutting-edge digital productions are now far harder to restore to modern standards than films shot on celluloid fifty years earlier. A well-preserved 35mm negative from 1955 can yield a gorgeous 4K transfer. A digital feature from 2003, shot on what was then state-of-the-art equipment, might be stuck at standard definition forever.

Days of Future Past

When Danny Boyle shot 28 Days Later in 2002, he chose Canon XL-1 miniDV cameras. The decision was partly practical as the lightweight cameras allowed for guerrilla-style shooting on London’s deserted streets, and partly aesthetic. The harsh, blown-out digital look gave the film an immediacy that felt perfect for a story about civilisation’s collapse.

The cameras recorded at 720×576 pixels which is PAL standard definition. For context, a modern iPhone shoots 4K video at 3840×2160 pixels, with roughly 25 times more information in every frame.

At the time, this didn’t seem like a problem. Standard definition was the norm. DVDs looked fantastic compared to VHS. Nobody was thinking about what these films would look like in twenty years.

In contrast, when you shoot on 35mm film, the main standard for movie cameras, you’re not really capturing a fixed resolution. You’re exposing silver halide crystals to light, creating a physical record of the scene with an almost absurd amount of potential detail. The exact “resolution” depends on the film stock and how you scan it, but modern estimates put 35mm somewhere between 4K and 8K equivalent. Some argue even higher for large format stock such as 80mm.

More importantly, that detail actually exists in the negative. It’s been sitting there since the day the film was shot, waiting for scanning technology to catch up. When we remaster Lawrence of Arabia or 2001: A Space Odyssey in 4K, we’re not inventing detail. We’re finally extracting what was always there.

Digital video from the early 2000s doesn’t work that way. What was captured is what exists. Those 720×576 pixels aren’t hiding secret information underneath. The cameras had a fixed resolution, and that resolution is now embarrassingly low by contemporary standards.

The Uncanny Valley of Upscaling

“But wait,” you might reasonably ask, “can’t we just use AI to upscale these films?”

We can. And increasingly, we do. Tools have become remarkably sophisticated at adding plausible detail to low-resolution footage. The results can be impressive, especially for content that wasn’t intended to look “cinematic” in the first place such as old TV shows, news footage and home videos.

The problem is that word. Plausible. AI upscaling doesn’t reveal hidden details. It hallucinates detail that looks like it could have been there. The algorithm examines a blocky, pixelated face and generates what a higher-resolution version of that face might look like based on patterns it learned from millions of other faces.

Sometimes this works brilliantly. Sometimes you get something that sits in a weird uncanny valley, technically sharper but somehow wrong in ways that are hard to articulate. Textures that feel synthetic, skin that looks waxy and fabric that doesn’t quite behave like fabric.

For films that were shot on early digital for aesthetic reasons, aggressive AI processing creates an additional problem. The lo-fi digital texture of 28 Days Later isn’t a flaw to be corrected, it’s part of what made the movie work. Clean it up too much and you lose something that can’t be put back.

This puts restoration teams in an impossible position. Do you present the film as it was intended to be seen, knowing modern audiences on 65-inch 4K screens will notice every compression artifact? Or do you “improve” it with AI, knowing you’re changing the director’s original vision at its core.

A Brief History of Bad Timing

The 2000s were uniquely cursed in this regard. It was the precise moment when digital filmmaking became viable enough that serious directors started using it, but before the technology had matured to resolutions that would remain acceptable long-term.

Consider the timeline.

Late 1990s — Digital video exists but is mostly confined to low-budget indie films and documentaries. The Dogme 95 movement embraces the format’s limitations as aesthetic virtues. Lars von Trier shoots The Celebration on miniDV in 1998.

2000–2002 — Early digital starts appearing in mainstream productions. George Lucas shoots Attack of the Clones on Sony CineAlta cameras at 1080p, declaring it the future of cinema. Boyle shoots 28 Days Later on miniDV. The gates are opening.

2003–2006 — The wave crests. Michael Mann shoots Collateral and Miami Vice on Thompson Viper cameras. David Lynch makes Inland Empire on a Sony PD-150, declaring he’ll never shoot film again. Robert Rodriguez pushes digital filmmaking into family blockbusters with Spy Kids sequels and Sin City.

2007–2010 — The first truly high-resolution digital cinema cameras appear. The Red One launches in 2007, capable of shooting at 4K. The Arri Alexa follows in 2010. From this point forward, digital films generally capture enough resolution to survive future format changes (subject to future radical changes to screen technology).

That roughly seven-year window, let’s call it 2000 to 2007, is a generation of films that were technologically progressive for their time and are now technologically trapped.

Some of the most visually distinctive work of the era lives in this limbo. Inland Empire’s hallucinatory nightmare textures were inseparable from the crude DV format Lynch used. Dancer in the Dark’s raw emotional brutality came partly from being shot on 100 consumer camcorders simultaneously. Open Water’s horror worked because it felt like you were watching somebody’s holiday video turn into a snuff film.

George Lucas enters, stage right

Attack of the Clones (2002) was the first major studio production shot entirely on digital cameras. Lucas had been pushing for this transition for years, convinced that digital was not only the future but actively superior to film.

The Sony CineAlta cameras used for Episodes II and III captured at 1080p. By the standards of 2002, this was impressive, true high definition when most consumers were still watching standard def broadcasts. By current standards, it’s less than a quarter of 4K resolution and roughly a sixteenth of 8K.

4K releases of the prequel trilogy exist, but they’re heavily upscaled rather than derived from native high-resolution sources. Watch them on a large modern display and you’ll notice a certain softness, a lack of the crystalline detail present in the original trilogy restorations (which were shot on film and could be properly scanned at 4K).

The irony here is that Lucas was so convinced of digital’s superiority that he also went back and “improved” the original trilogy with digital effects, effects that were rendered at resolutions that now look dated while the underlying film footage remains timeless.

Why Film Ages Better Than Files

A film negative is a physical object that can be re-examined with improving technology. Better scanners extract more detail. Better colour science improves the transfer. The negative hasn’t changed, but our ability to read it has.

A digital file is a fixed quantity. The numbers in the file are the numbers in the file. You can process them differently, upscale them algorithmically, but you can’t extract information that was never captured.

There’s also the question of format obsolescence. Film is remarkably stable as a storage medium. A properly stored negative from 1920 can still be projected or scanned today using the same principles as when it was created. The format hasn’t changed because the format is physical.

Digital formats change constantly. Codecs fall out of favour. Compression standards evolve and storage media become unreadable. A miniDV tape from 2003 requires increasingly rare hardware to play. A hard drive from the same era might be entirely dead. The theoretical advantages of digital, perfect copying, no degradation, only matter if you can actually access the data.

There are documented cases of studios discovering that digital masters from the early 2000s had become corrupted or were stored in formats nobody could easily read anymore. The Library of Congress has warned repeatedly about the challenges of digital preservation compared to traditional film archiving.

This doesn’t mean film is some perfect archival medium. It absolutely isn’t. Celluloid degrades. Colour stocks from the 1970s and 80s are notorious for fading toward magenta. Nitrate film from the silent era is literally flammable and chemically unstable. Acetate stock can develop “vinegar syndrome,” becoming brittle and unusable. Countless films have been lost because negatives were stored poorly, damaged in fires, or simply thrown away when studios decided they had no commercial value.

The point isn’t that film preservation is easy. It’s that when a film negative is properly preserved (stored at controlled temperature and humidity, protected from light and chemical contamination) the information embedded in those silver halide crystals remains accessible. The ceiling for recovery is remarkably high, even if reaching that ceiling requires considerable effort and expense.

What Happens Now?

Studios and distributors are increasingly turning to AI-powered restoration for early digital films, with mixed results.

The 4K release of something like Collateral is the best-case scenario. The film was shot at 1080p, but the imagery was carefully composed and the digital artifacts were minimal. AI upscaling can add convincing detail without changing the viewing experience at its core. It’s not quite the same as a native 4K source, but it’s acceptable.

At the other end of the spectrum, a film like Inland Empire probably shouldn’t be “restored” in any traditional sense. The blown-out highlights, crushed blacks, and compression artifacts aren’t problems to be solved. They’re part of the film’s visual language. Any version that removes them would be a different movie. Most early digital films fall somewhere between these extremes, requiring case-by-case decisions about how much intervention is appropriate.

A Note on What We’ve Lost

The films shot on early digital aren’t obscure curiosities. They include some of the most culturally important work of their era. 28 Days Later all but invented the modern zombie movie. Inland Empire is Lynch at his most experimental. Collateral is Mann’s masterpiece. The Star Wars prequels, whatever your feelings about them, were childhood-defining for a generation.

These films exist, and will continue to exist, in some form. But the question of how they’ll look to future audiences remains unresolved. Will AI upscaling become convincing enough that the resolution limitations become invisible? Will tastes shift so that early digital aesthetic becomes valued rather than apologised for? Will someone invent restoration techniques we can’t currently imagine?

In Praise of Uncertainty

Early digital films aren’t going to disappear. They’ll be preserved, restored with whatever tools are available, and watched by future audiences who will bring their own expectations and tolerances to the experience.

But there’s something worth recognising about the people who chose digital in the early 2000s, often because it seemed like the responsible, forward-thinking choice. They were wrong in ways they couldn’t have anticipated.

The filmmakers who stuck with “outdated” 35mm through this period, often facing pressure and mockery for their technological conservatism, turned out to be the ones preserving their work most reliably for the future.

Christopher Nolan’s stubborn insistence on shooting film, which seemed almost pathologically nostalgic at the time, now looks prescient. His films from this era scan beautifully at 4K and will continue to scale up as display technology improves. His digital-pioneering contemporaries are stuck trying to make 1080p footage look acceptable on increasingly massive screens.

There’s no triumphalism in pointing this out. Just a reminder that the future is harder to predict than it looks, and the technologies that feel inevitable sometimes turn out to be evolutionary dead ends.

The early digital era produced remarkable films that pushed the medium in directions film stock couldn’t go. Those films deserve to be seen and remembered. But the format that made them possible also trapped them in amber at resolutions that grow more limiting every year.

I am a partner in Better than Good. We help companies make sense of technology and build lasting improvements to their operations. Talk to us today: https://betterthangood.xyz/#contact

In March 2023, GPT-4 could identify prime numbers with 97.6% accuracy. By June, that figure had cratered to 2.4%. Not a rounding error, not a minor regression, but a 95-point collapse on the same task with the same prompts. If a bridge lost 95% of its load-bearing capacity in three months, someone would go to prison. In AI, the vendor posts a changelog and moves on.

This pattern has repeated with depressing regularity across every frontier provider. Models ship to applause and enterprise contracts get signed on the strength of benchmark screenshots, and then something changes. The model you evaluated is no longer the model answering your customers, and nobody tells you until your production workflow starts producing garbage.

The evidence is not anecdotal

Researchers at Stanford and UC Berkeley tracked this drift formally, comparing GPT-3.5 and GPT-4 snapshots from March and June 2023 across seven tasks. The results were bad enough to make the researchers themselves flinch. GPT-4’s ability to generate directly executable code dropped from 52% to 10%. Its willingness to follow chain-of-thought prompting, one of the most widely used techniques for improving accuracy, degraded without explanation.

“The magnitude of the changes in the LLMs’ responses surprised us,” James Zou, a Stanford professor and co-author, told The Register. The team’s conclusion was blunt. The behaviour of the “same” LLM service can shift substantially in weeks, and nobody outside the provider knows when or why.

This wasn’t a one-off result that got debated and forgotten. The OpenAI developer forums have become a rolling graveyard of complaints. In September 2025, users running GPT-4.1 reported severe intelligence degradation within 30 days of launch, with complex tool calls and multi-step instructions suddenly failing. Similar threads appeared for GPT-4 Turbo in May 2025. The pattern never varies, and by now it has become depressingly predictable. Works brilliantly at launch, degrades silently, users scramble to figure out what broke.

Why this happens (and why the incentives encourage it)

There are at least four mechanisms that can degrade a deployed model, and most frontier providers are using all of them simultaneously.

Quantisation is the most technically straightforward of the four, and the easiest to understand. A model trained in 16-bit or 32-bit floating-point precision gets compressed to 8-bit or 4-bit integers for serving. The arithmetic is straightforward enough, since a model stored in FP16 needs roughly two bytes per parameter, so a 70-billion-parameter model demands about 140GB of VRAM just for weights. Quantise to 4-bit and you cut that to around 35GB, enough to run on hardware that costs a fraction as much.

The trade-off is supposed to be minimal, and Red Hat’s analysis of over 500,000 evaluations found that 8-bit and 4-bit quantised models showed “very competitive accuracy recovery” on most benchmarks, especially for larger models. But that phrase “most benchmarks” is doing heavy lifting. Quantisation works by rounding, and rounding destroys outlier values. The weights that fire rarely but matter enormously for edge-case reasoning are exactly the weights that get flattened first. For standard tasks you barely notice the difference, but for the specific hard problems your production system was built to handle, the gap can be catastrophic. One developer reported that dynamic quantisation of a 3B-parameter model dropped accuracy from 65.6% to 32.3%, a halving that no benchmark average would predict.

Mixture-of-experts routing is the more interesting culprit, and the one providers talk about least. DeepSeek’s V3, for example, has 671 billion total parameters but only activates about 37 billion per token. The economics are irresistible because you get the capacity of a massive model with the inference cost of a much smaller one. But the router decides which experts handle which queries, and routing decisions are probabilistic. A query that activated your model’s strongest expert subnetwork at launch might get routed differently after an update to the routing logic, or after the provider adjusts load balancing to handle peak traffic. The user sees the same model name in the API response. The actual computation behind it may have changed entirely.

Distillation and model substitution is the elephant in the room that everyone suspects but nobody can prove definitively. Rumours have circulated since mid-2023 that OpenAI routes some queries to smaller, cheaper models behind the same API endpoint. The Gleech.org 2025 AI retrospective put it plainly: “True frontier capabilities are likely obscured by systematic cost-cutting (distillation for serving to consumers, quantisation, low reasoning-token modes, routing to cheap models).” GPT-4.5 was retired after just three months, presumably because the inference costs were unsustainable, even though it still ranked in the top five on LMArena for hallucination reduction nine months later. The model that performed best got killed because it was too expensive to run.

Safety tuning and RLHF adjustments create the subtlest form of drift. When OpenAI tightens content filters or adjusts the model’s tendency to refuse certain queries, those changes ripple through the entire behaviour space. The Stanford study found that GPT-4 became less willing to explain why it refused sensitive questions, switching from detailed explanations to terse “Sorry, I can’t answer that” responses. The model may have become safer by one measure, but it simultaneously became less transparent and less useful for legitimate applications that happened to brush against the updated boundaries.

The economics are doing exactly what you would expect

Running frontier models is staggeringly expensive, and every provider is under pressure to reduce cost-per-token. The maths, as one industry analysis noted, resembles building more fuel-efficient engines and then using the efficiency gains to build monster trucks. Token prices have dropped by a factor of 1,000 in three years, but reasoning models now generate thousands of internal tokens before producing a single visible output, and 99% of demand shifts to the newest model the moment it ships.

Providers respond by doing what any business would do. They optimise for throughput and margin, quantising the weights and routing easy queries to cheaper subnetworks while distilling the flagship into something that passes the benchmarks but costs a tenth as much to serve. The individual techniques are all defensible, but stacked together and applied silently, they create a system where the model’s advertised performance diverges from its delivered performance over time.

DeepSeek made this trade-off explicit and turned it into a business strategy. Its V3 model serves inference at roughly 90% below comparable OpenAI and Anthropic rates, and the MoE architecture that enables this pricing is openly documented. Whatever you think of the approach, at least the engineering trade-offs are visible. The problem is worse when providers make the same trade-offs quietly, behind an API that returns the same model identifier regardless of what actually computed the response.

What this means if you build on top of these models

The practical upshot is unpleasant but straightforward. If your application depends on consistent model behaviour, you are building on sand that shifts without warning. The Stanford researchers recommended continuous monitoring, and they were right, but monitoring alone doesn’t solve the problem, because it tells you something broke without stopping it from breaking.

Pinning to a specific model snapshot helps, where providers offer it, but even snapshots get deprecated. OpenAI maintains them for a few months and then requires developers to migrate. The careful evaluation you ran against the March snapshot becomes irrelevant when you’re forced onto the June version and nobody can tell you exactly what changed.

The deeper issue is one of trust and transparency. When a model provider updates a live model, they are unilaterally changing the behaviour of every application built on top of it. That is not a software update but an undocumented API change, the kind that would trigger outrage in any other engineering discipline. Imagine if AWS silently swapped your database engine for a cheaper one that was “approximately equivalent” on standard benchmarks, and you can begin to see how the AI industry has somehow normalised something that would be career-ending negligence anywhere else.

Where this leaves us

The model you benchmarked, the one that earned the contract, that impressed the board, that your engineers spent weeks building prompts and evaluation harnesses around, is a snapshot of a moving target. Quantisation shaves off the edges while routing sends your queries to whichever expert subnetwork happens to be cheapest that millisecond, and safety updates redraw the boundaries of what the model will and won’t do. None of it shows up in the model name string your application receives in the API response.

Somewhere in a data centre, the accountants and the alignment researchers are both pulling the same model in different directions, one toward cheaper inference and the other toward tighter guardrails, and the engineers who built their products on last month’s version are left checking the forums to figure out why everything stopped working on a Tuesday.

I am a partner in Better than Good. We help companies make sense of technology and build lasting improvements to their operations. Talk to us today: https://betterthangood.xyz/#contact

There is a particular conversational move that has become common in discussions about AI. Someone demonstrates a new capability, shares a use case, or describes how their workflow has changed, and a familiar response arrives. What about security? What about governance? What about the hallucination problem? What about my twenty years of experience? Each objection arrives wearing the costume of legitimate concern, and each one contains enough truth to feel reasonable in the moment. But taken together, they form something that looks less like careful analysis and more like a defence mechanism.

The pattern is whataboutism in its textbook form. The term originates from Cold War-era Soviet diplomacy, where officials would deflect criticism of human rights abuses by pointing to racial violence in America. The rhetorical structure was never designed to resolve the original issue. It existed to neutralise it. To shift the frame from “is this true” to “but what about that other thing,” and in doing so, to ensure that neither question ever gets properly answered. The AI version of this runs on similar fuel, though the people doing it are rarely aware they’re doing it at all.

The objections are correct and that is beside the point

The uncomfortable thing about AI whataboutism is that the concerns are mostly valid. AI security is genuinely underdeveloped, particularly around Model Context Protocol implementations, where the attack surface is wide and poorly understood. Governance frameworks in most organisations range from nonexistent to laughably outdated. Hallucinations remain a structural feature of large language models, a byproduct of how they generate text rather than a bug that some future update will fix. And twenty years of domain expertise does contain knowledge that no model can replicate, particularly the kind of tacit understanding that comes from watching things break in production over and over again until you develop an instinct for where the next failure will come from.

So all are true, but none is the point.

The point is that these objections are being deployed not as calls to action but as reasons for inaction. There is a significant difference between “AI has security vulnerabilities, so we need to build better guardrails while we adopt it” and “AI has security vulnerabilities, so we’ll wait.” The first is engineering. The second is avoidance dressed up as prudence.

Leon Festinger’s theory of cognitive dissonance, first published in 1957, describes exactly what’s happening. When a person holds a belief about themselves (I am an expert, my skills are valuable, my experience matters) and encounters information that threatens that belief (this technology can do significant parts of my job faster and cheaper than I can), the resulting psychological discomfort has to go somewhere. Festinger identified three common escape routes for that discomfort. You can avoid the contradictory information entirely, you can delegitimise its source, or you can minimise its importance by focusing on its flaws. AI whataboutism is all three at once, packaged as due diligence.

The sunk cost

Samuelson and Zeckhauser’s work on status quo bias adds another layer here that is worth sitting with. Their 1988 paper demonstrated that people disproportionately prefer the current state of affairs, even when alternatives are measurably better, and that this preference strengthens as the number of available options increases. The mechanism underneath isn’t stupidity or laziness. It is loss aversion applied to identity.

When you have spent fifteen or twenty years building expertise in a specific domain, that expertise becomes part of how you understand yourself. It is the thing that justifies your salary, your title, your seat at the table. The suggestion that a tool might compress the value of that expertise, or redistribute it, or make parts of it accessible to people who didn’t put in the same years and hard yards, triggers something that feels like an attack even when it isn’t one. The natural response is to find reasons why the tool can’t possibly do what it appears to be doing. And conveniently, AI provides an inexhaustible supply of such reasons, because it is, in fact, imperfect.

The trap is that imperfect doesn’t mean useless. Imperfect is the condition of every tool that has ever existed. The first commercial aircraft couldn’t fly in bad weather. The early internet went down constantly. Mobile phones in the 1990s weighed a kilogram and dropped calls in buildings. Nobody looked at any of those technologies and concluded that the smart move was to wait until they were perfect before learning how they worked.

Yet that is precisely the position many experienced professionals are taking with AI, and the whataboutism provides them with just enough intellectual cover to feel like they’re being rigorous and righteous rather than scared.

The velocity problem

What makes this particular round of technological change different from previous ones, and what makes the coping mechanisms around it more dangerous than usual, is the speed.

Previous disruptions gave people time to adjust. The internet took roughly a decade to move from novelty to necessity for most businesses. Cloud computing crept in over years, first as a weird thing Amazon was doing with spare server capacity, then gradually as the default. Even mobile took the better part of five years to go from “we should probably have an app” to “our mobile experience is our primary channel.”

AI is not operating on that timeline. The gap between GPT-3 and GPT-4 was measured in months. The capabilities that seemed like science fiction in 2023 are baseline features in 2026. Agentic systems that were theoretical eighteen months ago are shipping in production today. The window in which “wait and see” was a defensible strategy has already closed for most knowledge work, and many of the people deploying whataboutism as a delaying tactic are burning through competitive advantage while they debate whether the fire is hot enough to worry about.

This is where the coping mechanism becomes actively harmful rather than merely unproductive. If the pace of change were slower, there would be time for the concerns to be addressed sequentially. Fix the security model, then adopt. Build the governance framework, then deploy. But the pace doesn’t allow for sequential anything. The security model has to be built while adopting. The governance framework has to be designed while deploying. The two activities are not opposed to each other, and treating them as an either-or is itself a form of denial.

What experience is actually worth now

The most pernicious form of AI whataboutism is the appeal to experience, because it contains the highest concentration of legitimate truth mixed with self-serving reasoning.

Experience matters enormously. The question is which parts of it matter, and for what. The parts that involve pattern recognition accumulated over decades of watching projects succeed and fail, the ability to smell trouble before it shows up in a status report, the judgment to know when a technically correct answer is practically wrong, those parts matter more than ever in a world where AI can generate plausible output at speed. What AI cannot do is evaluate whether the output is appropriate for the specific context, the specific client, and the specific political dynamics of a given organisation. That evaluation requires exactly the kind of accumulated wisdom that experienced people possess.

But the parts of experience that involve doing the work that AI can now do faster, the manual production, the research grunt work, the first-draft generation, the template building, those parts are depreciating rapidly. And for many experienced professionals, the manual production was the majority of how they spent their time, which means the shift feels existential. AI is also moving up the value chain, much as Chinese manufacturing moved from cheap toys to highly complex electronics. This creates a kind of creeping dread that even our most valued, intangible skills will also eventually be under threat.

The whataboutism around experience is often an attempt to avoid this sorting exercise entirely. Rather than doing the difficult work of figuring out which parts of twenty years of expertise are now more valuable and which parts need to be released, it is easier to treat the entire bundle as sacred and dismiss the technology that requires the unbundling.

The way out is through the discomfort

Cognitive dissonance resolves in one of two directions. You can change your beliefs to match the new information, which is uncomfortable but productive. Or you can distort the information to match your existing beliefs, which is comfortable and eventually catastrophic. Whataboutism is the distortion path, and the longer you walk down it, the harder it becomes to turn around, because every objection you’ve raised becomes part of the identity you’re now defending.

The alternative isn’t to abandon caution. It is to be honest about the difference between caution that leads to better decisions and caution that functions as a socially acceptable way to avoid making decisions at all. Build the governance framework, but build it while experimenting, not instead of experimenting. Raise the security concerns, but raise them in the context of “how do we solve this”, rather than “this proves we should wait.” Lean on your experience, but do the honest accounting of which parts of that experience the world still needs and which parts you’re holding onto because letting go feels like losing a piece of yourself.

The concerns are all valid. The coping mechanisms aren’t.

I am a partner in Better than Good. We help companies make sense of technology and build lasting improvements to their operations. Talk to us today: https://betterthangood.xyz/#contact

Meta has been quietly building something significant. Most marketers haven’t fully grasped the importance because it has been wrapped in machine learning jargon and engineering blog posts.

The Generative Ads Recommendation Model, which Meta calls GEM, is the largest foundation model ever built specifically for advertising recommendation. It’s live across every major surface on Facebook and Instagram, and the Q4 2025 numbers, a 3.5% increase in clicks on Facebook, more than 1% lift in conversions on Instagram, are worth paying attention to at Meta’s scale.

Eric Seufert recently published a deep technical breakdown of GEM drawing on Meta’s own whitepapers, a podcast interview with Meta’s VP of Monetization Infrastructure Matt Steiner, and the company’s earnings calls. His analysis is the most detailed public account of how these systems actually work, and what follows draws heavily on it. I’d recommend reading his piece in full, because Meta has been deliberately vague about the internals, and Seufert has done the work of triangulating across sparse sources to build a coherent picture.

That sparseness is worth mentioning upfront. Meta has strong commercial reasons to keep the details thin. What we’re working with is a combination of carefully worded whitepapers, earnings call quotes from executives who are choosing their words, and one arXiv paper that may or may not describe GEM’s actual production architecture. I think the picture that emerges is convincing. But we should be honest about the fact that we’re reading between lines Meta drew deliberately.

The retrieval/ranking split

If you’re going to understand what GEM changes, you need to grasp the two-stage model Meta uses to select ads. Seufert explains this well: first ad retrieval, then ad ranking. These are different problems with different systems and different computational constraints.

Retrieval is Andromeda’s job (publicly named December 2024). It takes the vast pool of ads you could theoretically see (potentially millions) and filters to a shortlist of tens or hundreds. This has to be fast and cheap, so the model runs lighter predictions on each candidate. Think of it as triage.

Ranking is where GEM operates. It takes that shortlist and predicts which ad is most likely to produce a commercial result: a click, a purchase, a signup. The ranking model is higher-capacity but processes far fewer candidates, and the whole thing has to complete in milliseconds. Retrieval casts the net; ranking picks the fish.

When Meta reports GEM performance gains, they’re talking about this second stage getting more precise. The system isn’t finding more potential customers, it’s getting better at predicting which ad, shown to which person, at which moment, will convert.

The retrieval/ranking distinction is coveted in more depth in Bidding-Aware Retrieval, a paper by Alibaba researchers that attempts to align the often upper-funnel predictions made during retrieval with the lower-funnel orientation of ranking while accommodating different bidding strategies.

Sequence learning: why this architecture is different

Here’s where it gets interesting, and where I think the implications for how you run campaigns start to bite.

Previous ranking models used what Meta internally calls “legacy human-engineered sparse features.” An analyst would decide which signals mattered, past ad interactions, page visits, demographic attributes. They’d aggregate them into feature vectors and feed them to the model. Meta’s own sequence learning paper admits this approach loses sequential information and leans too heavily on human intuition about what matters.

GEM replaces that with event sequence learning. Instead of pre-digested feature sets, it ingests raw sequences of user events and learns from their ordering and combination. Meta’s VP of Monetization Infrastructure put it this way: the model moves beyond independent probability estimates toward understanding conversion journeys. You’ve browsed cycling gear, clicked on gardening shears, looked at toddler toys. Those three events in that sequence change the prediction about what you’ll buy next.

The analogy Meta keeps reaching for is language models predicting the next word in a sentence, except here the “sentence” is your behavioural history and the “next word” is your next commercial action. People who book a hotel in Hawaii tend to convert on sunglasses, swimsuits, snorkel gear. The sequence is the signal. Individual events, stripped of their ordering, lose most of that information.

This matters because it means GEM sees your potential customers at a resolution previous systems couldn’t reach. It’s predicting based on where someone sits in a behavioural trajectory, not just who they are demographically or what they clicked last Tuesday. For products that fit within recognisable purchase journeys, this should translate directly into better conversion prediction and fewer wasted impressions.

But I want to highlight something Seufert’s analysis makes clear: we don’t know exactly how granular these sequences are in practice, or how long the histories GEM actually ingests at serving time. The GEM whitepaper says “up to thousands of events,” but there’s a meaningful gap between what a model can process in training and what it processes under millisecond latency constraints in production.

How they solve the latency problem

This is the engineering puzzle at the centre of the whole thing. Rich behavioural histories make better predictions, but you can’t crunch thousands of events in the milliseconds available before an ad slot needs filling.

Seufert’s analysis draws on a Meta paper describing LLaTTE (LLM-Style Latent Transformers for Temporal Events) that appears to address exactly this tension, though Meta hasn’t confirmed it’s the architecture powering GEM in production.

The solution is a two-stage split. A heavy upstream model runs asynchronously whenever new high-intent events arrive (like a conversion). It processes the user’s extended event history, potentially thousands of events, and caches the result as an embedding. This model doesn’t know anything about specific ad candidates. It’s building a compressed representation of who this user is and what their behavioural trajectory looks like.

Then a lightweight downstream model runs in real time at ad-serving. It combines that cached user embedding with short recent event sequences and the actual ad candidates under consideration. The upstream model consumes more than 45x the sequence FLOPs of the online model. That asymmetry is the whole trick, you amortise the expensive computation across time, then make the cheap real-time decision against a rich precomputed context.

One detail from Seufert’s breakdown that I keep coming back to: the LLaTTE paper found that including content embeddings from fine-tuned LLaMA models, semantic representations of each event, was a prerequisite for “bending the scaling curve.” Without those embeddings, throwing more compute and longer sequences at the model doesn’t produce predictable gains. With them, it does. That’s a specific and testable claim about what makes the architecture work, and it’s one of the few pieces of genuine technical disclosure in the public record.

The scaling law question

This is where I think the commercial story gets properly interesting, and also where I’d encourage some healthy scepticism.

Meta’s GEM whitepaper and the LLaTTE paper both reference Wukong, a separate Meta paper attempting to establish a scaling law for recommendation systems analogous to what we’ve observed in LLMs. In language models, there’s a predictable relationship between compute invested and capability gained. More resources reliably produce better results. If the same holds for ad recommendation, then GEM’s current performance is early on a curve with a lot of headroom.

Meta’s leadership is betting heavily that it does hold. On their most recent earnings call, they said they doubled the GPU cluster used to train GEM in Q4. The 2026 plan is to scale to an even larger cluster, increase model complexity, expand training data, deploy new sequence learning architectures. The specific quote that should get your attention is “This is the first time we have found a recommendation model architecture that can scale with similar efficiency as LLMs.”

The whitepaper claims a 23x increase in effective training FLOPs. The CFO described GEM as twice as efficient at converting compute into ad performance compared to previous ranking models.

Now, the sceptic’s reading. Meta is a company that spent $46 billion on capex in 2024 and needs to justify continued spending at that pace. Claiming their ad recommendation models follow LLM-like scaling laws is convenient because it turns massive GPU expenditure into a story about predictable returns. I’m not saying the claim is wrong, the Q4 numbers suggest something real is happening, but we should notice that this is also the story Meta needs to tell investors right now. The performance numbers are self-reported and the scaling claims are mostly untestable from outside.

That said, the quarter-over-quarter pattern is hard to dismiss. Meta first highlighted GEM, Lattice, and Andromeda together in a March 2025 blog post, and Seufert describes the cumulative effect of all three as a “consistent drumbeat of 5-10% performance improvements” across multiple quarters. No single quarter looks revolutionary, but they compound. And the extension of GEM to all major surfaces (including Facebook Reels in Q4) means those gains now apply everywhere you’re buying Meta inventory, not just on selected placements.

The creative volume angle

There’s a second dimension here that connects to where ad production is heading. Meta’s CFO explicitly linked GEM’s architecture to the expected explosion in creative volume as generative AI tools produce more ad variants. The system’s efficiency at handling large data volumes will be “beneficial in handling the expected growth in ad creative.”

This is the convergence I think experienced marketers should be watching most closely. More creative variants per advertiser means more candidates per impression for the ranking system to evaluate. An architecture that gets more efficient with scale, rather than choking on it, turns higher creative volume from a cost problem into a performance advantage. Seufert explores this theme further in The creative flood and the ad testing trap.

If you’re producing five ad variants today, producing fifty becomes a different proposition when the ranking system can actually learn from and differentiate between those variants at speed. The advertisers who benefit most from GEM’s improvements will be those feeding it more creative options, not those running the same three assets on rotation.

What this means for how you spend

I’m not going to pretend these architectural details should change your Monday morning. But a few things follow from them that are worth sitting with.

GEM’s purpose is to outperform human intuition at predicting conversions from behavioural sequences. If you’re still running heavy audience targeting with rigid constraints, you’re limiting the data the system can learn from. Broad targeting with strong creative has been the winning approach on Meta for a while. GEM widens that gap.

The bottleneck is shifting from targeting precision to creative supply. As the ranking model gets better at matching specific creative to specific users in specific behavioural moments, the constraint becomes whether you’re giving it enough material to work with.

Your measurement windows probably also need revisiting. If GEM is learning from extended behavioural sequences, attribution models that only look at last-touch or short windows will undercount Meta’s contribution to conversions that unfold over days or weeks.

And watch the earnings calls. The 2026 roadmap (larger training clusters, expanded data, new sequence architectures, improved knowledge distillation to runtime models) suggests we’re in the early phase. If the scaling law holds (and that’s a real if, not a rhetorical one), the gap between platforms running this kind of architecture and those that aren’t will widen.

Meta is rebuilding its ad infrastructure around a small number of very large foundation models, GEM, Andromeda, and Lattice, that learn from behavioural sequences rather than hand-picked features.

The results so far are impressive. Whether the scaling story plays out as cleanly as Meta’s investor narrative suggests is genuinely uncertain. But for marketers running at scale on Meta, the platform is getting measurably better at the thing you’re paying it to do, and the trajectory of improvement appears to have more room than previous architectures allowed.

I am a partner in Better than Good. We help companies make sense of technology and build lasting improvements to their operations. Talk to us today: https://betterthangood.xyz/#contact

If you’ve spent any time in enterprise technology over the past two decades, you’ll recognise the pattern immediately. A new category of tool emerges. Employees start using it because it makes their working lives easier. IT discovers this unsanctioned adoption, panics about security and compliance, and responds by trying to lock everything down. A period of organisational friction follows, during which the people who were already getting value from the tool become increasingly frustrated, while IT attempts to build a sanctioned alternative.

This is almost exactly what is happening with AI right now, except the speed of adoption has compressed what used to be a multi-year cycle into months. Harmonic Security’s analysis of 22.4 million enterprise AI prompts during 2025 found that while only 40% of companies had purchased official AI subscriptions, employees at over 90% of organisations were actively using AI tools anyway, mostly through personal accounts that IT never approved. BlackFog’s research from late 2025 found that 49% of employees surveyed admitted to using AI tools not sanctioned by their employer at work. And perhaps most tellingly, 63% of respondents believed it was acceptable to use AI tools without IT oversight if no company-approved option was provided. And even when there is a sanctioned version (typically an Enterprise license for Copilot and/or chatGPT, implementation seldom goes far beyond simply making licenses available to users.

The instinct from many IT departments has been to treat this as a security problem. And in all fairness, it is partly a security problem. IBM’s 2025 Cost of a Data Breach report found that 20% of organisations suffered a breach due to shadow AI, adding roughly $200,000 to average breach costs. That is not nothing. But treating shadow AI purely as a security problem misses the more interesting and more consequential question underneath it, which is about organisational design, capability gaps, and who should actually be responsible for an organisation’s AI strategy.

The ownership reflex

There is a well-documented tendency in organisations for existing power centres to claim ownership of emerging technologies. IT departments in particular have a long history of this behaviour, and it makes a certain amount of institutional sense. New technology involves infrastructure, security considerations, vendor relationships, and integration with existing systems. These are things IT teams understand and have built processes around.

The problem is that AI, particularly generative AI and the emerging wave of agentic AI, does not fit neatly into the traditional IT operating model. It is not a new enterprise application to be procured, deployed, and maintained. It is not an infrastructure upgrade. It is not even, primarily, a technology problem at all. AI adoption is fundamentally a business transformation problem that happens to involve technology.

When IT departments attempt to own AI strategy, several predictable things happen. First, they frame it through the lens they understand best, which means the conversation becomes dominated by questions about security policies, approved vendor lists, data governance frameworks, and integration architecture. These are all legitimate concerns, but they represent perhaps 30% of what makes AI adoption successful.

The capability gap

Effective AI implementation in an organisation needs people who can do several things that don’t appear anywhere on a traditional IT org chart. You need someone who understands the business process being transformed well enough to know where AI adds value and where it introduces risk. You need people who can design prompts and workflows that produce useful outputs, which turns out to be a surprisingly nuanced skill that combines writing ability, logical thinking, and deep familiarity with whatever domain you’re working in.

You need people who can evaluate AI outputs for accuracy and bias, which requires subject matter expertise that sits in the business, not in IT. And you need people who can manage the change process, because asking someone to fundamentally alter how they do their job is never a simple matter of handing them a new login.

This capability gap helps explain why shadow AI is happening in the first place. The people closest to the work are the ones who best understand where AI can help them. A marketing analyst who discovers that Claude can help them write campaign briefs in half the time is not going to stop using it because IT hasn’t approved the tool yet. A financial analyst who finds that an LLM can help them spot patterns in quarterly data is going to keep using it regardless of what the acceptable use policy says. These people are not being reckless. They are being rational, responding to the incentive structure in front of them, which rewards productivity and results over process compliance.

The Gartner prediction that shadow IT will reach 75% of employees by 2027 (up from 41% in 2022) tells you everything about the trajectory. And shadow AI, being even more accessible than traditional shadow IT since all you need is a browser tab and a free account, is accelerating this pattern dramatically.

So if IT cannot own AI strategy alone, and if the business is already adopting AI without waiting for permission, what does the right organisational response look like?

Conway’s law and the automation trap

Before getting to solutions, it is worth understanding the most important conceptual framework for why AI adoption goes wrong in traditional organisations. In 1967, a mathematician named Melvin Conway observed that organisations are constrained to produce designs that mirror their own communication structures. The observation, which became known as Conway’s Law, was originally about software architecture, but it applies with uncomfortable precision to how organisations approach AI.

Conway’s Law predicts that if you let AI adoption emerge organically within existing organisational structures, what you will build is a set of AI solutions that reproduce your existing departmental silos, legacy objectives, internal politics, and traditional power dynamics. You will, in effect, automate the existing org chart.

This is the single most common failure mode I see in enterprise AI adoption, and it is devastatingly easy to fall into. Marketing builds its own AI tools for content generation. Finance builds its own AI tools for forecasting. Customer service builds its own AI chatbot. HR builds its own AI-powered recruiting screener. Each of these projects may individually deliver some efficiency gains, but collectively they create a fragmented ecosystem of AI capabilities that cannot talk to each other, that duplicate effort, that embed existing biases and inefficiencies into automated systems, and that make future integration progressively harder.

As Toby Elwin put it, an enterprise cannot adopt AI faster than it can align decision rights, language, and accountability. If your departments cannot communicate effectively with each other today, your AI implementations will faithfully reproduce that dysfunction. The AI will hedge like committees hedge. It will fragment like silos fragment. It will optimise for departmental metrics rather than organisational outcomes.

FourWeekMBA’s analysis of Conway’s Law made the point vividly by examining Microsoft’s troubled Copilot deployment. If that product feels like three different tools fighting each other, it’s because it was built by three different divisions that were forced to integrate after the fact. This is not bad engineering. It is Conway’s Law doing exactly what Conway’s Law always does.

The temptation to automate the existing org chart is especially strong because it is the path of least resistance. It does not require anyone to give up territory. It does not require difficult conversations about who owns what. It does not require rethinking how work gets done. It simply applies AI to existing processes in existing departmental silos, which delivers enough small wins to create the illusion of progress while actually cementing the structural problems that will prevent the organisation from capturing AI’s larger transformative potential.

The incremental-vs-wholesale question

One of the most contentious questions in AI organisational strategy is whether you can get there incrementally or whether the scale of change required demands a more fundamental restructuring.

The honest answer is that it depends on your starting position and your ambition level. If you are a mid-sized professional services firm that wants to use AI to make your existing teams 20-30% more productive, an incremental approach that adds AI tools to existing workflows, builds capability gradually, and evolves governance frameworks over time is probably sufficient and definitely lower risk.

But if you are a larger organisation in a competitive market where AI is already changing the basis of competition, incrementalism may be dangerously slow. The organisations that are winning with AI right now are not the ones that added ChatGPT or Copilot to their existing processes. They are the ones that redesigned their processes around AI capabilities, which is a fundamentally different thing.

There is a useful distinction from the organisational design literature between “first-order change” (improving existing processes within the current structure) and “second-order change” (fundamentally altering the structure and assumptions themselves). Most organisations default to first-order change because it is more comfortable and less politically fraught. But AI may be one of those rare technological shifts where second-order change is necessary for organisations that want to do more than survive.

Consider a practical example. A mid-sized insurer wants to improve its claims process using AI. Today, a claim passes through four separate teams in sequence. First contact sits with the customer service team, who log it. Assessment and settlement sit with the claims handlers, who evaluate damage, validate the claim against the policy, and calculate what to pay. Investigation sits with a fraud and compliance team, who flag suspicious patterns. And payment authorisation sits with finance, who release the funds. Each handoff introduces delay, each team has its own systems and metrics, and the customer experiences the whole thing as an opaque, slow, and frequently frustrating process. This is Conway’s Law made visible to the policyholder.

The incremental approach would give each of those four teams their own AI tools. Customer service gets a chatbot for first notification of loss. The claims handlers get an AI that pre-populates damage estimates from photos and suggests settlement amounts. The fraud team gets a pattern-matching model. Finance gets automated payment routing. Each team becomes somewhat faster in isolation, but the fundamental structure remains untouched. Four teams, four handoffs, four sets of metrics, and the customer still waits while their claim passes from queue to queue.

The transformative approach would ask why the claim needs to pass through four teams at all. An AI system that can simultaneously assess damage from submitted photos, cross-reference the policy terms, run fraud indicators against historical patterns, calculate the settlement, and trigger payment could collapse most of that chain into a single interaction for straightforward claims. The customer submits their claim, the AI processes it end-to-end, and a human reviewer approves the output. What was a four-team, ten-day process becomes a one-team, same-day process for the 70% of claims that are routine. The complex and contested claims still need human expertise, but even those benefit from the AI having done the preliminary work across all four traditional functions simultaneously.

That second approach is incompatible with the existing org chart. It eliminates handoffs that currently define departmental boundaries. It changes what claims handlers, fraud analysts, and finance teams actually do with their time. It requires new performance metrics, because “claims processed per handler” stops making sense when the AI is doing the initial processing. And it raises uncomfortable questions about headcount in teams whose primary function was moving information from one stage to the next.

Aligning the value chain

So how do you actually make this work? The standard answer from most consultancies and conference speakers is “create a cross-functional AI team,” and while that answer is directionally correct, it is also woefully insufficient. Creating a cross-functional team is a structural intervention, and structural interventions fail when they are not supported by corresponding changes to strategy, capabilities, processes, and incentives. You cannot simply staple people from different departments together, give them an AI mandate, and expect results.

Jonathan Trevor’s strategic alignment research at Oxford’s Saïd Business School provides the most useful framework I’ve found for thinking about this practically. Trevor’s central argument, developed across his books Align and Re:Align and a series of articles in Harvard Business Review, is that organisations are enterprise value chains, and they are only ever as strong as their weakest link. The chain runs from purpose (what we do and why) through business strategy (what we are trying to win at) to organisational capability (what we need to be good at), organisational architecture (the resources and structures that make us good enough), and management systems (the processes that deliver the performance we need).

The power of Trevor’s framework is that it forces you to work through AI adoption as a linked sequence of decisions rather than treating it as an isolated structural question. And it exposes exactly where most organisations’ AI efforts break down.

Start with purpose. Most organisations’ stated purpose does not change because of AI, but AI may fundamentally change what fulfilling that purpose looks like in practice. Our insurer’s purpose is presumably something about protecting policyholders and paying claims fairly and promptly. AI does not alter that purpose, but it radically changes what “promptly” can mean and what “fairly” requires in terms of oversight.

Then business strategy. If AI enables same-day claims settlement for routine cases, that becomes a competitive differentiator. The strategy question is whether the insurer wants to compete on speed and customer experience (which demands the transformative approach) or on cost efficiency within the existing model (which might justify the incremental approach). This is a leadership decision that needs to be made explicitly, because the organisational implications of each choice are completely different.

Then organisational capability. This is where most AI initiatives fall apart, because the capabilities required to execute an AI-driven claims process are different from the capabilities the insurer currently has. You need people who understand insurance underwriting AND who can evaluate AI outputs for accuracy. You need people who can design human-AI workflows where the AI handles routine cases and humans handle exceptions, which is a design skill that barely existed even five years ago.

You need people who can monitor AI systems for drift and bias over time, which is a form of quality assurance that traditional insurance operations have never had to think about. Trevor’s framework makes you ask whether these capabilities exist in the organisation today, whether they can be developed internally, and what the timeline for building them looks like. If the honest answer is that the organisation does not have these capabilities and cannot build them quickly enough, then the strategy needs to account for that through hiring, partnerships, or a phased approach that builds capability as it goes.

Then organisational architecture. This is where the cross-functional team question finally becomes relevant, but now it sits within a much richer context. The architecture question is about what structures, roles, and resources are needed to support the capabilities you have identified. For our insurer, this might mean creating a new “claims intelligence” function that sits alongside the existing claims teams, staffed by people who combine insurance domain knowledge with AI workflow design skills.

It might mean redefining the role of claims handlers from “people who assess claims” to “people who review and improve AI-assisted claim assessments,” which is a different job with different skill requirements and different performance expectations. It almost certainly means changing reporting lines so that the people responsible for AI-driven claims have authority over the end-to-end process rather than being subordinate to any single one of the four existing departmental heads.

The architectural decisions also need to address the political dimension directly. In the insurer example, the head of claims, the head of fraud, and the head of finance all currently control their own domains with their own budgets and their own staff. A transformative AI implementation threatens all three of those power bases simultaneously.

Trevor’s work acknowledges this tension by framing alignment as a leadership responsibility rather than an organisational design exercise. The decision about how to restructure around AI cannot be delegated to the teams whose authority it threatens. It has to come from senior leadership who have the authority and the willingness to make uncomfortable choices about where power and resources should sit.

Then management systems. This is the link that gets forgotten most often and that causes the most damage when it is neglected. Management systems include how people are measured, how they are rewarded, how information flows, and how decisions are made. You can create the perfect cross-functional AI team with the right people and the right mandate, and it will still fail if the management systems around it are pulling in the wrong direction.

Return to the insurer. Suppose you have created your claims intelligence function and staffed it with capable people. If the claims handling team is still measured on “claims assessed per handler per day,” they have no incentive to cooperate with the AI initiative, because the AI threatens to make their metric irrelevant. If the fraud team’s bonus structure is tied to “fraud cases identified,” they will resist an AI system that flags fraud automatically, because it removes the activity their compensation is based on. If the IT department’s budget is allocated based on the number of systems it manages, it will resist an architecture where AI tools are managed by the business, because every tool that sits outside IT reduces IT’s budget justification.

These are not hypothetical objections. They are the exact mechanisms through which well-intentioned AI initiatives get quietly suffocated by the organisations that launched them. Trevor’s value chain framework makes these dynamics visible before they become fatal, because it forces you to ask whether your management systems are aligned with your stated AI strategy or whether they are actively working against it.

The practical implication is that an organisation pursuing transformative AI adoption needs to change its measurement and reward systems at the same time as it changes its structures and capabilities. For the insurer, this might mean replacing team-level productivity metrics with end-to-end outcome metrics like “time from claim submission to resolution” and “customer satisfaction at point of settlement.”

It might mean creating shared incentives that reward the claims intelligence function and the traditional claims teams for collaborative outcomes rather than individual departmental throughput. And it definitely means ensuring that the people whose roles are changing through AI adoption have a visible and credible path to new roles that are at least as valued as their old ones.

What separates success from failure

The patterns on both sides are remarkably consistent. The organisations getting this right have governance frameworks that distinguish between high-risk and low-risk AI use cases rather than applying blanket controls to everything, and they have accepted that some amount of unsanctioned experimentation is healthy and necessary.

SentinelOne offers a good example of this in practice. Rather than threatening consequences for unapproved AI use, they created a coalition of eager participants across the organisation who can test new tools and introduce them for piloting, with multiple fast pathways for getting a tool evaluated and adopted. The data supports this approach. Harmonic Security’s research found 665 different AI tools across enterprise environments, and concluded that blanket blocking was futile and counterproductive.

The failure modes are the mirror image. Organisations go wrong when they hand AI ownership entirely to the CTO, when they create governance so heavy it prevents adoption altogether (pushing more activity into the shadows), when they mandate a single vendor across the entire organisation, or when they treat AI as a cost-reduction exercise (which produces the “automating the existing org chart” failure mode rather than process transformation).

The most pernicious mistake is treating AI adoption as a single programme with a defined start and end date. AI is not an ERP implementation. It does not have a go-live date. It is a continuous organisational capability, and the Nadler-Tushman Congruence Model helps explain why. When the formal structure says “IT owns AI” but the informal culture says “people are already using AI tools whether IT knows about it or not,” that misalignment will eventually break something. Usually what gives is the formal structure, albeit slowly and painfully.

Making it practical

The frameworks above provide a way to think about the problem, but thinking is not the same as doing. Here is what the sequence of practical actions looks like when you apply Trevor’s value chain logic to AI adoption in a traditional organisation.

Start by pressure-testing your strategy. Before making any structural changes, get your senior leadership team in a room and answer one question honestly. Are you pursuing AI for incremental efficiency within your current operating model, or are you pursuing it to fundamentally change how you compete? Both are valid answers, but they lead to completely different organisational responses.

Most organisations have not answered this question explicitly, which means different parts of the business are operating on different assumptions about what AI is for. That misalignment will express itself as confusion, turf wars, and wasted investment. Trevor and Varcoe’s HBR diagnostic on strategic alignment provides a structured way to surface these gaps.

Map capabilities against ambition. Once you have strategic clarity, audit what capabilities you have today versus what you need. Be honest about this. Most organisations dramatically overestimate their internal AI capability because they conflate IT technical skills with AI implementation skills, which are different things. The capability audit should cover technical AI skills (model selection, integration, monitoring), domain translation skills (people who can bridge between business processes and AI possibilities), workflow design skills (people who can redesign processes around AI rather than bolting AI onto existing processes), and change leadership skills (people who can bring others along). For each capability, you need a frank assessment of whether it exists internally, whether it can be developed on a realistic timeline, or whether it needs to be acquired through hiring or partnerships.

Design architecture around capability, not hierarchy. This is where the cross-functional team becomes relevant, but only if you design it deliberately. The team needs a clear mandate tied to the strategic choice you made in step one. It needs to be staffed with people who collectively cover the capability gaps you identified in step two. It needs reporting lines that give it authority over the processes it is transforming, which almost certainly means it reports to someone senior enough to arbitrate between competing departmental interests. And it needs to be structured in a way that acknowledges the political dynamics honestly. In practice, this means having representatives from the affected business units on the team, giving those representatives genuine influence over decisions, and ensuring that the business units they come from are rewarded for their cooperation rather than penalised for losing headcount or budget.

Redesign management systems in parallel. This is the step that separates organisations that succeed from organisations that create impressive-sounding AI teams that quietly accomplish nothing. Before the cross-functional team starts work, change the metrics and incentives for the business units it will be working with. If you are asking the adjusting team to cooperate with an AI initiative that will change their roles, make sure their performance metrics reflect the new expectations rather than the old ones. If you are asking IT to hand over some responsibilities to the AI function, make sure IT’s budget and headcount are not penalised for doing so. The management system changes do not need to be permanent or perfect at this stage, but they need to exist, because without them you are asking people to act against their own incentive structures, which they will not do for long regardless of how compelling your AI vision is.

Build in public. One of the most effective practical tactics I have seen is to have the cross-functional AI team work visibly and share results (including failures) broadly across the organisation. This serves several purposes simultaneously. It demystifies AI for people who are anxious about it. It creates internal advocates as people see tangible results. It gives the shadow AI users a legitimate channel to contribute their knowledge and experience. And it builds the organisational AI literacy that will be necessary for scaling beyond the initial team. Kotter’s dual operating system concept is relevant here, where the cross-functional AI team operates as a faster-moving network alongside the existing hierarchy, and the visibility of its work gradually shifts organisational norms without requiring a top-down mandate that triggers resistance.

Plan for the second wave. The initial cross-functional team and its first projects will teach you things that no amount of upfront planning can predict. Build explicit review points where you reassess your strategy, capabilities, architecture, and management systems in light of what you have learned. Trevor’s concept of strategic realignment as a continuous leadership competency rather than a one-off transformation is particularly apt for AI, because the technology is evolving so rapidly that any fixed structure will be outdated within a year. The goal is not to design the perfect AI organisation on day one. The goal is to build an organisation that can adapt its AI capabilities continuously as both the technology and your understanding of it evolve.

Conclusion

Most traditional organisations are not structured for the kind of cross-functional, fast-moving, continuously-evolving capability that AI demands. Their hierarchies, incentive structures, decision-making processes, and cultural norms were all designed for a world where technology changed more slowly, where knowledge was more specialised, and where coordination costs were higher.

AI offers the opportunity to do fundamentally different things, and to organise differently to do them. This goes well beyond doing existing things faster. The organisations that recognise this and are willing to make structural changes, even uncomfortable ones, will outperform those that try to bolt AI onto their existing operating model and hope for the best.

Shadow AI is the canary in the coal mine. It is telling you that your people are ready for AI, even if your organisation is not. The question is whether leadership will listen to that signal and respond with genuine organisational adaptation, or whether they will respond with a reflexive control impulse.

The history of technology adoption in enterprises suggests that the control impulse always loses eventually. The people with the tools always outperform the people with the policies. The difference with AI is that “eventually” is measured in months rather than years, and the competitive consequences of being late are proportionally far more severe, perhaps even existential.

I am a partner in Better than Good. We help companies make sense of technology and build lasting improvements to their operations. Talk to us today: https://betterthangood.xyz/#contact