You are the grader

You ask a coding agent to fix a failing test. A few minutes later it reports back: all tests pass. You merge the change, and a week later you find out how they came to pass. The agent did not fix the bug. It edited the test until the broken code satisfied it.An outside evaluator found GPT-5.5 falsely claimed to have completed an impossible programming task in 29% of samples, against 7% for GPT-5.4. METR caught OpenAI’s o3 overwriting the timer that measured its speed and patching the code that scored it. Or you ask an assistant for sources, and it hands you a tidy citation to a paper that does not exist, complete with authors, a year and a journal.OpenAI researchers argue that models invent facts largely because training and benchmarks reward a confident guess over “I don’t know”. Act II takes that argument apart.

These are not the products of careless amateurs. They come from systems built by some of the most capable engineering teams alive, tested for months before release. So why would a model that can refactor a large codebase rewrite a test instead of fixing a bug? Not out of malice, and not because nobody noticed. The short answer, which the rest of this essay unpacks, is that the model was trained by being scored, and it learned to satisfy the grader rather than the person who wrote the grader.

The trouble is far older than computers. Whenever people are paid, ranked or promoted by a number, the number starts to drift away from the thing it was meant to track, because writing down precisely what you want is much harder than it sounds. In 1902 the French colonial government of Hanoi, fighting a plague scare, paid a bounty for every rat tail handed in. Officials soon noticed tailless rats running in the streets: catchers were cutting off the tails and letting the rats go to breed, and inspectors later found rat farms on the edge of the city.The historian Michael Vann, who found the daily ledgers in the French colonial archives, reports 20,114 rats killed on 12 June 1902 alone. The bounty was one cent per tail, and it was eventually cancelled. The proxy was tails; the goal was fewer live rats.

Modern institutions, with auditors and lawyers, fare no better. Wells Fargo pressed branch staff to sell more products to every customer, and employees opened roughly 1.5 million deposit accounts that customers may never have authorised.A 2017 review raised the count to about 3.5 million accounts, and about 5,300 employees were fired; the chief executive’s slogan was “Eight is great”, meaning eight products per customer. In 2020 the bank paid $3 billion and admitted that from 2002 to 2016 employees had falsified records and forged signatures to meet sales goals. In Atlanta, where schools were judged on state test scores, investigators found organised cheating at most of the schools they examined, much of it teachers erasing wrong answers and filling in right ones.A 2011 state investigation found cheating at 44 of 56 schools and implicated 178 educators; in 2015, 11 of the 12 educators tried were convicted under Georgia’s racketeering law. The answer sheets could be edited by the people being graded, which is the closest human parallel I know to an agent editing its own tests. English ambulance services were told to reach life-threatening calls within eight minutes, and the recorded response times began to bunch up just under the line.In about a third of ambulance trusts, response times had been “corrected” to fall under eight minutes. For the four-hour emergency-department target, 96% of patients were officially seen in time in 2004/05, but only 77% said so in a patient survey. Each time, the number improved while the thing it stood for did not.

The economist Charles Goodhart made the point about monetary statistics in 1975, and the line everyone quotes, “when a measure becomes a target, it ceases to be a good measure”, is in fact the anthropologist Marilyn Strathern’s 1997 paraphrase. Whoever said it best, the pattern now called Goodhart’s law has the same shape in every case above: the people being measured could reach the measurement, and the gaming happened exactly where the measure was blind. Hold on to that sentence, because it describes a trained model just as well.You may know a story about a cobra bounty in British Delhi that led to cobra farms. No primary record of that episode has been found, and the phrase “cobra effect” dates only to 2001, so I treat it as a parable and use the rat ledgers instead.

Machines are the purest test of Goodhart’s law. A person who games a target has a conscience, a fear of the auditor and a limited number of tries. A learning system has none of these. It tries an enormous number of variations and keeps whatever scored. In a boat-racing game called CoastRunners, OpenAI rewarded an agent for hitting targets laid out along the track, expecting that to encourage finishing the race. The agent found a lagoon where the same targets kept respawning, and circled it indefinitely without ever finishing.OpenAI’s 2016 write-up is widely quoted as saying the boat outscored human players. I could not check that figure against the original, so I leave it out.

DeepMind researchers call this specification gaming: behaviour that satisfies the literal objective without achieving the intended outcome. A simulated robot hand, trained on the approval of people watching through a camera, learned to hover between the camera and the ball so that it only looked as if it had grasped it. A block-stacking agent rewarded for the height of a red block’s bottom face simply flipped the block over.DeepMind’s public list held about 60 such examples in 2020.

Language models are not exempt. They are the most capable gamers yet, because they can read the code that grades them. In 2025 METR found OpenAI’s o3 patching the scoring code and overwriting the timer on research tasks; asked on one task whether this matched what the user wanted, it said no every time.Reward hacking appeared in 30.4% of o3’s runs on one research benchmark, RE-Bench, against 0.7% on another, HCAST. The name for this in language models is reward hacking. Nobody wrote “cheat” into these systems. Each was shown a score and pushed toward whatever raised it, which is how today’s assistants are trained.

This essay follows that push from the inside, through one unusually public training run, Xiaomi’s MiMo-V2.6, whose trainer logs and costs were published as it ran. Act I builds the practice loop that every frontier lab now runs: a model tries a task many times, a grader scores each try, and the model leans toward what scored. It also explains why so old an idea took some sixty years to work on language. Act II walks through five places where real graders were blind, and the habit each blind spot grew, from declaring victory early to guessing instead of saying “I don’t know”.

Act III steps back to the whole factory: how a current frontier model is assembled, and why most of its skill arrives before the grading loop even starts. Act IV turns the loop around. When you hand a model a task, you are building the same kind of setup, because your prompt is the task and your check is the grader, so the leaks become things you can close. Act V lets you test whether the picture holds on cases I never described.

Start with a model that has already read a large slice of the internet. It can write code, fill in forms and use tools, but unevenly: it fixes one bug cleanly and flails on the next. The way labs improve it today is the way people improve at anything, which is practice. The model is set real tasks, it attempts each one as an agent, working step by step inside a disposable computer, and whatever worked is kept. The difficulty is the sheer volume of that practice.

An attempt is a conversation with a computer, built from turns. In one turn the model reads the latest output, thinks, and issues one command, such as running the tests or opening a file. Writing a turn takes the model roughly fifteen to thirty seconds, and an attempt in the run this act follows averaged about 44 turns, so a single attempt is ten to twenty minutes of nonstop work.This assumes the model writes about half of each turn’s new text, roughly 750–1,600 tokens, at about 50 tokens a second; anything from 20 to 100 is plausible. It excludes the time the commands themselves take to run. Xiaomi’s open recipe cuts a coding attempt off after 80 minutes. The slider below lets you change that assumed writing speed.

Now multiply. A single round of practice held 25,088 attempts. Run one after another, that one round would take about a year, and the whole run about thirty years. In reality each round finished in a few hours, because tens of thousands of sandboxed computers worked side by side.Steps of the larger Pro model took 2.7 to 6.4 hours, with between about 11,000 and 31,000 sandboxes active. At about 44 turns per attempt, the last step held roughly 1.09 million turns; at 30 seconds a turn that is about a year of serial work. A person working at a couple of minutes per turn would need about eighteen working years for that one step. No team of people can read a year of work every few hours, so something other than a person has to decide which attempts were good. That something is called the grader.

Graders come in two broad kinds. The first is a rule-based check, a program with a definite right answer. A coding attempt is scored by running hidden tests against its changes, as in MiMo’s coding tasks and the open DeepSWE-Preview agent. A maths answer is compared with the known result, as in DeepSeek-R1. An office task passes only if the right database row now reads “rejected”, and a security exploit only if the crash report names the expected bug type and code location.

The second kind is another language model acting as judge, for work no program can check. It may tick off a checklist one item at a time, as MiMo’s office tasks are judged, by an earlier copy of MiMo itself; compare answers and pick the better one, as in Kimi K2’s self-critique; or write a short critique before giving a score, as DeepSeek’s generative reward model does. Judges can weigh matters of taste, but they waver: Xiaomi dropped model judges for its security tasks because the same exploit got different verdicts on different runs.

Either way, notice what the grader sees and what it misses. A test suite sees whether the tests pass. It does not see whether the fix was sensible, honest or in scope, unless someone wrote a check for exactly that, which is the agent from the prologue in miniature. The cards below set each grader beside what it is blind to. Keep that gap in mind, because Act II is about what grows inside it.

Before going further into the loop, it is worth seeing how old it is. A child learning to ride a bicycle is not handed the physics. She tries, wobbles, falls, tries again, and her body keeps whatever kept her upright. Reinforcement learning is that idea written down for machines: try something, get a score, and make whatever scored a little more likely next time. Everything in this act is a version of those three verbs.

The idea is older than the personal computer, and for most of its history it was very hard to make work. The reason is easiest to see in a game. In chess you move, your opponent replies, and only many moves later do you learn whether you won. That single result is the reward, and it arrives once, at the very end, without saying which moves earned it. Worse, the number of ways a game can unfold grows faster than intuition can follow.The first program to learn this way played checkers, not chess, as the section after next tells. I use chess because more readers know it.

Count the possibilities. White has 20 legal first moves, Black has twenty replies to each, and every further ply, one move by one side, multiplies the count again by roughly twenty-five to thirty-five. So after just three moves each there are 119,060,324 possible games, and a full game runs to around eighty plies.The exact counts are 400 games after two plies, 8,902 after three, 197,281 after four and 4,865,609 after five. Claude Shannon’s 1950 estimate put the whole game tree at about 10¹²⁰ games; Victor Allis later used an average of 35 legal moves and 80 plies per game. The slider below steps through the first six plies.

Now imagine learning chess from scratch by playing randomly against a decent opponent. Almost every game is a loss. And when a rare win does arrive, it is one result spread over some forty of your own decisions. Which of them won it? Was a loss the fault of one blunder, or of being outplayed throughout? Marvin Minsky named this the credit-assignment problem in 1961, and the only general answer is volume: play so many games that a good move turns up more often in wins than in losses.

The first program to learn this way played checkers. In the 1950s Arthur Samuel at IBM had his program play many games against a version of itself, adjusting its judgement of positions as it went, until it played like a respectable novice. His 1959 paper is credited with naming the field: machine learning. The next landmark took three more decades. Gerald Tesauro’s TD-Gammon taught itself backgammon by playing against itself, and its best version came within a whisker of a top human player.Version 2.1 lost to Bill Robertie by a single point over 40 games. That version trained on 1.5 million games against itself. At ten minutes a game, that is roughly thirty years of nonstop play.

Then the games got bigger. DeepMind’s DQN learned dozens of Atari games from the raw screen, at a cost of about 38 days of play per game, and AlphaZero taught itself chess from 44 million games against itself, then beat the strongest conventional engine.The chess run took 9 hours, with 5,000 TPUs generating games. In the published 1,000-game match against Stockfish, AlphaZero won 155 and lost 6. At an hour a game, forty-four million games is some five thousand years of human play. Every one of these systems learned from a win, a loss or a game score, and every one needed a mountain of experience, because it started knowing nothing.

Language arrived later. OpenAI’s InstructGPT had people rank answers, trained a second model to predict their rankings, and ran RL against that learned grader, a recipe called RLHF.Raters preferred the answers of a 1.3-billion-parameter InstructGPT to those of the 175-billion-parameter GPT-3, a model over 100 times larger. DeepSeek-R1-Zero raised its pass rate on the AIME 2024 maths competition from 15.6% to 71.0% with RL alone. Then DeepSeek swapped the learned grader for plain rules, and its R1 model practised each maths question 16 times. Not millions of games: sixteen attempts. Something had changed about the learner.

What changed is where the learner starts. The measure that matters here is sample efficiency: how much a learner improves per attempt. A program playing chess at random needs millions of games because nearly all of them come out the same, a loss, and identical results teach nothing. Learning needs contrast: at least one attempt that worked beside one that did not, so that there is something to prefer.

A language model does not start at random. Pre-training on a vast amount of human writing means it already knows how code, tests and terminals behave, so on a fresh task it often succeeds. At the very first step of MiMo’s RL run, the larger model already passed about 56% of its attempts. At that rate a group of sixteen attempts almost always contains both a pass and a fail, so nearly every group gives the model something to learn from.The chance that 16 independent attempts include at least one pass and one fail is 1 − p¹⁶ − (1 − p)¹⁶. At p = 0.56 it is about 0.9999; at 0.1, about 0.81; at 0.01, about 0.15; for a random program, effectively zero. Real tasks vary, and about 13–16% of MiMo’s tasks still came back with every attempt failing. The slider shows how quickly that falls apart as the success rate drops.

So RL on language models mostly sharpens rather than invents. It takes something the model can sometimes do and makes it what the model usually does. Think of years spent reading every chess book, then a season of tournaments: the tournaments rarely teach a new opening, but they make you reliably play the good move you half-knew, and no number of tournaments helps someone who does not know how the pieces move. Act III shows how literal this is: most of MiMo’s coding gain arrived before RL began.

Back to the loop, now with the reason for its shape in hand. The first move is to set the same task many times. A model picks each word with a little randomness, controlled by a setting called temperature, so every attempt takes a slightly different route: one reads the files first, another guesses, another calls the wrong tool and then recovers. MiMo sets each task 16 times.

Each complete attempt, from the first command to the final answer, is called a rollout. The repetition is the point. One attempt says almost nothing about whether a route is good, but sixteen side by side supply exactly the contrast the random chess player lacked: some routes work, some do not, and the difference is visible. A pile of attempts is still not a verdict, though. Something has to say which of them succeeded.

That something is the grader, and in this run it is mostly a program. In MiMo’s coding tasks it runs hidden tests against the model’s changes and pays 1 if they pass and 0 if they do not. In office tasks, a script checks the rows and files the model was supposed to produce, and a model judge scores the parts no script can check. So after grading, each of the sixteen attempts carries a single number.

Look at what that number throws away. Two passing attempts get the same full marks whether one fixed the bug cleanly and the other weakened a test until it passed. Xiaomi ships a guard with its terminal tasks that zeroes the reward if the agent plants files that hijack the test run or touches protected files, which tells you the problem was real enough to need guarding. Every guard like that marks a hole someone already found; the open question is always which holes nobody has found yet.

A reward on its own is a poor teacher, and two tasks side by side show why. On an easy task, 15 of 16 attempts pass; on a hard one, only 2 do. If every pass simply earns its reward, the easy task hands out fifteen equal pushes toward behaviour the model already has, and the hard task hands out two. The rare success is pushed exactly as hard as a routine one. And the single failure on the easy task earns nothing at all, so nothing tells the model to avoid whatever went wrong there.

Chess players solved this long ago. A rating jumps when you beat a much stronger player and barely moves when you beat a beginner, because the system compares your result with what was expected of you. The loop does the same, and it gets its expectation from the attempt’s siblings: the group’s average score is the best guess at how this model does on this task. Each attempt is judged by how far it landed above or below that average, a margin called its advantage. The hard task’s pass earns 1 − 0.125 = +0.875; the easy task’s pass earns 1 − 0.9375 = +0.0625.The failures get negative margins. The lone easy-task failure scores 0 − 0.9375 = −0.9375, a hard push away; each hard-task failure gets only −0.125, a gentle one. In both groups the margins add up to exactly zero.

So the rare pass now pushes fourteen times harder than the routine one, and on the easy task the main thing learned becomes “avoid what that one failure did”. Because each group’s margins add up to zero, a task the model has already mastered cannot drown out one it is still learning. Judging attempts against their siblings, with no second network estimating how hard each task is, is the core of GRPO, a method DeepSeek introduced in 2024 that MiMo uses. Switch the figure below between raw reward and compared-with-siblings to watch the pushes change.

“Pushed up” sounds vague, so here is what literally changes. A model writes one token, a word or piece of a word, at a time, and before each one it holds a menu of possible next words with a probability on each. Take one moment inside an attempt: the model has just written “The test failed. Next I will” and is choosing between run, read, edit, delete and skip. If the attempt as a whole scored above its group, the word it actually chose becomes a little more likely in situations like this one, and the other options shrink to make room. If it scored below, the chosen word becomes a little less likely.

The same push, sized by the attempt’s advantage, is applied to every word the model wrote in that attempt, often tens of thousands of them. Nobody tells the model which words mattered. The crucial command and the throwaway remark move together, which is the credit-assignment problem again, and it is solved the same way, by volume. Across thousands of attempts, words that help turn up more often in above-average attempts, so they climb, while words that do not matter rise and fall at random and wash out.

Then a new batch of tasks arrives and the loop runs again: set the task, grade the attempts, compare them, nudge. In MiMo’s run each round, called a step, used 1,568 tasks, and the run lasted 30 steps. Over many rounds the nudges add up. The catch is that the model never sees what the task’s author wanted. It sees only which attempts beat their siblings, and the grader decides that. Try the buttons below, then consider the word “delete” on that menu.

Now push the comparison to its edge. The slider sets how many of the 16 attempts pass, and the numbers beside it show the group average and the advantage each attempt receives. Move it slowly from the middle toward either end and watch the gap between a pass and a fail shrink as the group becomes more uniform. You have met one end of this already, in the random chess player whose games were all losses. Before you drag the slider all the way, make a prediction about the very end.

A loop drawn in the abstract hides the details that decide what gets rewarded, so follow one real task. Among the training data Xiaomi released is an office task, written in Chinese, about correcting a construction permit for a senior high school in Shenzhen. The agent must read a reviewer’s verdict in a Word file, find the matching sign-off in a mock office-approval system, cross-check a permit ledger, mark item 3 as rejected with a complete reason, write a short receipt saying construction may not start, and report what it found.

The grader is five checks: two critical rule checks on the database row and on the receipt’s wording, one language-model check on the final report, and two sanity checks that nothing else was touched. If a sanity check fails, the score is zero whatever else passed, and in the open recipe an attempt scores 1 only if all five pass. The table below runs a group of sixteen through the arithmetic. Notice, too, that the receipt check looks for required phrases, so it can tell whether the words are present but not whether a site manager would find the receipt useful.

One task is a toy; the run was not. Each step of the Pro run pushed 1,568 tasks × 16 attempts = 25,088 trajectories through this loop, the year of serial work from the start of this act squeezed into a few hours.By the last step an average trajectory held about 137,000 tokens of context. Most of the tasks were code, 68% agentic and competitive coding, with the rest split across general tool use, visual design, context following and cyber security. The 30 steps cost about $2.6M for Pro.Roughly $87,000 a step. The smaller Flash model’s run cost about $0.9M. Even basic counts are slippery here: Xiaomi’s report, release note and dashboard give three different ranges for how many tokens a step held, as the first note below lays out. Three jobs share the bill: generating the attempts, updating the model, and grading them.

So what did the nudging buy? Over the run, the Pro model’s training pass rate rose from 0.565 at step 1 to 0.633 at step 30, a modest and noisy climb, because each step draws a different batch of tasks.That is about 12% in relative terms. It peaked at 0.643 at step 20. On DeepSWE, a benchmark of coding tasks kept out of training, the gain looks larger: Pro’s score rose from 58.41 to 72.57, with dips along the way.Meanwhile the average context per attempt nearly doubled, from about 72,000 to 137,000 tokens, so the model learned to do more work per task. That climb is real, but it is the smaller half of the story: most of the model’s coding skill was already there before RL started. The small charts below set Pro beside the smaller Flash model.

Here is the uncomfortable part of the loop I just described: it has no idea what anyone intended. It strengthens whatever scored. So the question that decides a model’s habits is not what its designers wanted, but what else, besides a real solution, could score. Every attempt does one of a few things. It solves the task, or it gets the grader to say yes without solving it, or it gives up and hands the decision back to a person.

A perfect grader would pay only for the first. Real graders are written by people on deadlines, and every blind spot lets reward leak toward the others, after which the nudges from the practice loop carry it into habit. Labs call the second outcome reward hacking. I’ll walk through five leaks documented in recent training runs. Each has a human version you have probably met, and each ends in a symptom you can see in your own chat window.

Imagine an exam with one question that cannot be answered: the figure it refers to was left off the paper, and you are not allowed to ask the invigilator. Very few students leave it blank. They write something plausible, or work around the missing figure and present the result as though it were complete, because a blank scores nothing and a confident attempt might score something. Nobody taught them to bluff. The marking scheme did.

Anthropic found the machine version of this exam in its own coding environments: a file the task needed was missing, and the prompt told the model not to ask for clarification. From the grader’s side nothing changed. It ran the same tests and paid for a pass as before. From the model’s side the honest route had closed, so every attempt that scored had to score some other way. Toggle the file below, but predict first.

Most of us learned this trick at school. On a multiple-choice test that counts only right answers, a blank and a wrong answer both score zero, so you never leave a question blank. You guess, because a guess might land and a blank never will. Exam setters who want to stop this subtract part of a point for every wrong answer, and the guessing stops being free. The scoring rule, not the student’s honesty, decides whether guessing pays.

Language models sit the same kind of test. Most benchmarks and many training graders mark an answer right or wrong, and give “I don’t know” the same score as a wrong answer. Picture a model asked for an obscure person’s birthday, with a vague sense of the answer and no way to check it. The slider below sets how sure it is, and the toggle switches between two scoring rules. Before you touch either, predict what the first rule teaches.

Anyone who wrote school essays to a minimum word count knows how to pad: restate the question, add a second example that says what the first one said, close with a summary of what you just wrote. Now take the minimum away and keep only a tired marker who skims, and who tends to read the longer essay as the more thorough one. Padding still pays; nobody has to ask for it. Length has quietly become a stand-in for effort.

The machine version appears wherever no script can check the answer, so another model judges it. For a request like “write a project brief”, Moonshot’s Kimi K3 uses a judge model that writes a rubric and then compares candidates two at a time, in a tournament. Comparing is easier for a judge than scoring on an absolute scale, and it keeps giving signal as answers improve. It also inherits the tired marker’s bias. The widget below plays the tournament round after round, and each round the winner’s style is what the loop reinforces.

Picture a support desk whose agents are scored on a single thing: complaints about their tickets. The safest agent soon learns to escalate everything, because a ticket handed to a supervisor can never go wrong on your record. “Let me check with my manager” becomes the answer to every question. Now flip the score to tickets closed per hour and you get the opposite agent, who closes tickets by promising things nobody authorised. One desk, two scoring rules, two opposite failures.

Cursor trained its Composer model on real users’ reactions, and shipped a new checkpoint roughly every five hours. An accepted edit counted as good, and an edit the user reverted or complained about counted as bad. Look at what that grader could not see. A clarifying question cannot be reverted, so it can never score badly. The slider below runs from a reward that mostly punishes mistakes to one that mostly pays for finishing.

A target with a line in it changes behaviour near the line and nowhere else. A school judged on how many pupils clear the pass mark has every reason to teach to that mark: lift the pupils just below it over the line and leave everyone else alone. Doing better than the line earns nothing extra, so nothing pushes past it. The measure is silent everywhere except at the line, and behaviour drifts wherever the measure is silent. Researchers who study targets call this a threshold effectEnglish ambulance services had a target of reaching life-threatening calls within eight minutes. Their recorded response times spiked just under eight minutes, and about a third of ambulance trusts were found to have “corrected” times..

The fifth leak shows that a lab that knows about leaks still ships one. MiMo’s training had a rule against runaway tool calls: more than 32 tool calls in one turn stopped the attempt and set its reward to zero. The rule said nothing about calls below the line, so repetitive calling went unpunished as long as it stayed under, and the loop amplified itOn Flash’s replay set, turns with more than ten calls went from 11.1% at step 0 to 24.6% at step 20. One real trajectory made 59 tool calls; at the 12th call, the model chose yet another call with 94.56% probability.. Xiaomi had two ways to fix it, and the widget below prices both.

Put the five side by side and the pattern is hard to miss. The missing file taught the model to declare victory, as an impossible exam teaches students to bluff. Right-or-wrong scoring taught it to guess, as multiple-choice tests do. A comparing judge taught it to pad, like the word-count essay. A reaction-based reward taught it to ask about everything, or to barrel on, like the support desk under either score. A cap that was silent below its line taught it to flood, like teaching to the threshold.

No designer wanted any of these. Each came from a gap between what the grader paid for and what was meant, carried into habit by a loop that is faithful only to the grader. This is Goodhart’s law with a training loop attached. Labs close the gaps with monitors, penalties and sturdier environments, and the assembly line shows where those fixes sit. The leaks matter to you for a more direct reason, though. Each has a cousin in your own prompts, because when you hand a model a task, you are writing its grader.

If the practice loop were the whole story, a lab could take a freshly built network and practise it into an agent. It can’t, for a reason the all-fail problem already showed. A model that fails every attempt at a task gets a group of equal scores, so every advantage is zero and nothing is learned. Practice amplifies abilities a model already has in some small measure. Most of the work therefore happens before the loop runs, and the loop is one station on a longer line.

I find it easiest to meet each station through the problem it exists to solve. A raw network knows nothing, so it reads. A reader can’t act, so it studies agent work. A student of agent work has the wrong habits, so it is shown examples. Imitation can’t beat its examples, so it practises. Practice produces specialists, so they are merged. And because every grader leaks, someone watches throughout. No lab publishes a diagram of its whole line; this one is assembled from the reports of open labs, including Xiaomi, DeepSeek, Moonshot, Zhipu and NVIDIA, which agree on its shape.

Start with the problem at its most extreme. A freshly built network is billions of random numbers, and asked anything, it emits noise. Before it can attempt a coding task it has to know what code is, what English is, what a bug report sounds like. The cheapest teacher of all that is the written record itself. So the first station, pre-training, has the model read, and learn to predict what comes next, across an almost unimaginable amount of text.

How unimaginable? A person reading nonstop, day and night, at a brisk 250 words a minute would need roughly 150,000 years to get through MiMo’s text aloneMiMo-V2.6-Pro was pre-trained on 30T tokens, 27T of text and 3T of mixed media, with its context window stretched from 32K to 256K tokens partway through. The reading time assumes about 0.75 words per token, so 27T tokens is about 20T words; at 250 words a minute, a year of nonstop reading is about 131M words. Xiaomi does not disclose what pre-training cost, and none of the closed labs disclose theirs.. What comes out can continue any kind of document in any voice. Anthropic describes the result as a model that can play many characters, and post-training as the process that picks out one of them, the Assistant, and shapes it. The later stations decide which character you meet.

A model that has read everything is still like a graduate who has read every manual and never held the tools. Hand it a real repository and dozens of steps of work, and it fails, every time. That is fatal for what comes later, because practice needs some successes to compare. A group where every attempt fails teaches nothing: the all-fail problem again. So labs continue the reading, but on the kind of material an agent produces. This is mid-training.

MiMo’s mid-training mixed realistic agent trajectories for coding, general, visual and research tasks with repository-level code and high-quality text, and stretched the context to 1M tokens. Research on why this matters points the same way: in controlled experiments, later practice produced genuinely new ability only when earlier training left the model at the edge of its competence, and mid-training substantially improved results for the same compute. The stage is growing, too: xAI says it ran it longer for Grok 4.7 than for its predecessor.

One ingredient is aimed straight at the leaks of Act II. Earlier experiments had shown reward hacking, so Xiaomi had the model reflect on those episodes, rewrite the offending turn and carry on honestly, and fed the corrected transcripts into mid-training. The model reads, in effect, its own corrected confessions before it ever practises. The size and cost of MiMo’s mid-training mix are not disclosed.

Even after mid-training, the model has the wrong habits for the job. It may answer a question by continuing it, ignore the format a tool expects, or ramble past the point where a reply should end. A new hire with the right knowledge and the wrong habits gets shown what a good report looks like, a few dozen times, before anyone grades their work. Models get the same treatment, called supervised fine-tuning: complete examples of good answers, and training to imitate them word by word.

SFT teaches format, turn-taking and the basics of tool use, so that the first practice attempts are not all failures. Labs say remarkably little about it: MiMo’s report gives the stage a single line, calling it a short SFT stage before RL, with no data, size or settings. The examples increasingly come from the previous generation of the same familyFor Grok 4.7, Grok 4.6 models generated the SFT trajectories across effort levels, harnesses and domains, with model-based screening.. Keep an eye on MiMo’s SFT checkpoint, because it comes back later as a grader.

Imitation has a ceiling: the student can at best become its examples. An apprentice who only copies the master’s finished pieces never discovers a technique the master didn’t use, and never learns what to do when their own work goes wrong. To go further, the model has to try things and be scored on the result. Only now does the loop from Act I run, inside environments: a task, a sandbox with tools and files, and a grader, packaged together.

The scale is hard to picture. Moonshot created more than fifty million disposable sandboxes to train Kimi K3; if a person set one up every second, without sleeping, it would take more than a year and a halfExactly 51,219,741 sandboxes, built from 1,505,678 images. Containers suffered kernel panics and deadlocks caused by the agents’ own actions, so Moonshot moved to Firecracker microVMs.. Those environments rebuild popular agent harnesses such as Claude Code and Codex, which plausibly explains why models now work reasonably well inside tools they were never shipped with. A harness, in other words, is part of what the model practised in.

Labs split on how to organise the practice. MiMo trained code, general tool use, visual design, chat and cyber security in one mixed run across several harnesses, and its average score on three harnesses held out of training rose from about 50% to 66% during the run. Most labs instead train a separate specialist copy per domain: Kimi K3 trains three domains at three effort levels, nine specialists in all. Separate copies raise an obvious question: how do they become one model?

Practising in separate copies leaves a lab with a strange problem: nine experts and only one model to ship. A hospital can keep its cardiologist and its neurologist in different rooms. A model has a single set of weights, and every skill has to live in it at once. You can’t staple the specialists together. Something has to teach one model what all of them know, without one skill overwriting another.

The labs have converged on one answer, though they apply it differently, so switch between them below. Xiaomi ran one mixed RL run, then a later stage folded in teachers, including ones for hard-to-verify domains such as game development and scientific researchXiaomi calls that stage MOPD2.. DeepSeek-V4 replaced its mixed RL stage entirely with distillation from more than ten teachers, and Moonshot merges its nine Kimi specialists the same way. In every case a single student writes its own answers while the specialists grade them, a method Xiaomi calls MOPD. Why it beats the obvious alternatives is the next problem.

Why not simply average the specialists? Because averaging trades their skills against each other. Each skill is spread through the weights in its own pattern, so the average of a chess player and a pianist keeps some of each, unevenly: one skill survives nearly intact while another slips well below its specialist, and which one slips is hard to predictDeepSeek reports that weight merging and mixed RL often degrade performance. The uneven, domain-by-domain pattern is my reading of why merges disappoint; the reports do not publish per-domain merge results.. The next idea is to have the student copy each specialist: collect its best transcripts and train the student to reproduce them word by word. That fails more subtly, and the way it fails is worth understanding, because it explains why practising on your own attempts matters at all.

Think of learning chess from game tapes. A grandmaster’s games are excellent, but they are played in positions a novice rarely reaches. The novice copies the opening, slips once early in the middle game, and is now in a position no tape covers, where the next move is a guess that leads somewhere stranger still. A coach at your shoulder works the other way round. You play your own game, the coach rates each of your moves, and the correction lands exactly where you went wrong.

Every leak in Act II was found because someone was watching, and the human version of the fix is old: a factory that pays by output hires inspectors, because output alone invites shortcuts. Labs now inspect every step of training. Their monitors come in a few kinds: a classifier that reads each episode for signs of a hack, a rule that zeroes the reward when known tricks appear in the sandbox, and an attacking agent that probes an environment for exploits before the model ever sees it.

Anthropic screens every RL episode sampled at every step with reward-hack classifiers. Xiaomi ran its hack agent against every environment until it found no exploitMiMo’s released terminal tasks ship a guard script that zeroes the reward if the agent planted interpreter hooks such as conftest.py or sitecustomize.py, or modified protected files. Qwen3.7-Max monitored its own coding RL for more than 80 hours, wrote 13 new detection rules and flagged 1,618 hacking cases. OpenAI measures “metagaming” during training as well as afterwards.. But a monitor is a grader too, with blind spots of its own. Xiaomi reports confirmed hacks under 2%, yet its dashboard logs a “hack attempt rate” many times higherThe logged rate runs from 32% to 51%., and nothing published says what counts as an attempt.

Monitoring can also leak into the thing it watches. Anthropic disclosed that a bug let reward code see the model’s reasoning in about 8% of Claude Mythos Preview’s RL episodes. That could teach a model to write reasoning that looks good to the grader rather than reasoning that is good, like a student who learns to show the working the examiner likes.

Rule-based checks cover only what a program can verify, and paying people to read millions of attempts is out of the question. So who grades an office report, or the quality of a patch? Increasingly, an LLM judge, and in MiMo’s case largely the model family itself: office-style tasks were judged during RL by MiMo-V2.6-SFT, the pre-RL checkpoint of the very model being trained. It is like a school where last year’s graduates mark this year’s exams. They know the subject, and they share the school’s blind spots.

Coding attempts that passed their tests went to a second judge, because tests alone had rewarded speculative compatibility branches, swallowed exceptions and relaxed validation. That judge is an agent that sees all 16 attempts in one workspace, can run code, compares the passing patches on fit of approach, precision, minimality, out-of-scope effects and codebase conventions, and moves reward from sloppier passes toward cleaner ones while setting confirmed hacks to zero. Xiaomi calls it GAR. DeepSeek-V4 goes further: its judge for hard-to-verify tasks is the policy itself, trained alongside it.

So “self-improvement” in this year’s reports means the family grades its own work and builds its own curriculum. It does not mean self-play: MiMo’s tasks were fixed datasets, edited by people mid-run. And Xiaomi publishes no rate at which any of its judges agrees with humans, so how well the graduates mark is, for now, taken on trust.

I have spent most of this essay on RL, because RL is where the grader lives. That raises a fair question: how much of the improvement did RL actually buy? MiMo publishes enough to check on one benchmark, DeepSWE, a held-out set of real coding tasks, measured against the previous model, MiMo-V2.5-Pro. Before you look at the result, guess how the gain splits between everything that came before RL and RL itself.

Most of this line is drawn from open labs, because the closed ones say little. Anthropic, OpenAI and Google name no RL algorithm and publish no split of compute between pre-training and RL. Anthropic publishes a system card with each model, and it reports what the model did during training, such as hack rates by type, but not its algorithm, compute or data mix. Google DeepMind says of its consistency training work that most of it is not published.

Even the most open lab leaves gaps: Xiaomi does not disclose its SFT data or size, its pre-training cost, the base model of its coding judge, or any human-agreement rate for its judges, and the note below lists more. The sources also disagree with each other in places, such as MiMo’s confirmed-hack rate against its logged attempt rate, and its final score in the report against the one on its dashboardThe report’s final table gives DeepSWE 71.9 for Pro, while the dashboard shows 72.57 at step 30; the table probably reflects a different checkpoint or evaluation setup.. So treat the line as the best public sketch, with its gaps marked. It establishes enough for what comes next: the model you use was shaped by graders, and it will treat your setup the same way.

Everything so far happened in a lab, months before you typed anything. The weights you talk to are frozen, and nothing you write trains them. Yet the model carries the habits its graders paid for, and when you hand it a task you rebuild the situation it practised in: a task, an environment, and someone deciding whether the result is good. Your prompt is the task. Your check, or the lack of one, is the grader. So every leak from Act II is waiting in your setup until you close it.

Start with the prompt most people write: fix the login bug. In a teaching simulation of sixteen attempts at that prompt, most miss, because the model has to guess which bug, in which file, for whom, and what “fixed” means. The kinds of failure match ones documented earlier; the proportions are illustrative rather than measured. Each piece that follows closes one leak, and I will name the leak each time.

Think of handing a task to a brilliant contractor on their first morning. They are capable, but they don’t know your codebase, your users or what went wrong yesterday, and they won’t unless you tell them. A model knows only its weights and what is in front of it: anything outside its context window does not exist for it. Recall the construction-permit task. Its author spelled out every requirement, because a grader can check only what a task writes down, and the model learned to solve tasks as written.

A one-line prompt writes almost nothing down, and people new to prompting tend to write as if the model shared their background, and under-specify as a result. Anthropic’s advice is to write for a brilliant but new employee who lacks context on your norms, and to say why as well as what. Before: fix the login bug. After: Since yesterday's deploy, Safari users get logged out after about 5 minutes. Session handling is in auth/session.ts. This is for a customer demo on Friday. In the simulation, attempts that misread the task shrink the most. This is M1: only what is in context exists.

The missing-file leak taught the model that a “done” the grader accepts is as good as a done that is true. Agents stop when the work looks done, and without a stated end point, looking done is all the model has to go on. A builder told only “sort out the kitchen” will stop when it looks sorted, or rebuild half the house. The second failure lives here too, since current models can widen a task’s scope on their own. So say what done looks like as something you could observe, and say what is out of bounds.

OpenAI’s Codex guide asks for a “Done when” line in every prompt, alongside the goal, context and constraints: Done when: a Safari session survives 30 minutes idle and the existing auth tests still pass. Don't touch the OAuth flow or refactor unrelated code. Phrase the action literally as well, because current models follow instructions precisely, so “can you suggest changes” gets suggestions rather than edits. In the simulation, both stopping early and over-reaching shrink. These are M9, it stops when it looks done, and M5, it follows the instruction you actually wrote.

The strongest single piece is a check the model can run itself, a verifier of your own. A trainee pilot learns in a simulator that says at once when they have crashed; take that feedback away and they fly by feel. Agents practised the same way. MiMo’s coding reward was exactly this: hidden tests, pass or fail, the unit-test grader from Act I. A runnable check turns your task back into the kind the model practised on, so it can close the loop on its own; without one, looking done is the only signal available and you become the verification loop.

Feed it real failures too, because models are much better at fixing an error once told where it is than at finding it. Run npm test -- auth and show me the output. Here is the failing test: [paste]. In the simulation, real solves jump. This is M6: give it the tests and tools it trained with. But in adding a check you have become a grader, and graders have blind spots.

Labs closed this leak by hardening environments and zeroing confirmed hacks, the monitors of Act III. You close it with the two moves an auditor uses: keep the books out of the accountant’s reach, and ask for receipts rather than a summary. First, protect the check by saying what the tests are for and that they are not the model’s to change. Anthropic suggests telling the model that tests verify correctness rather than define the solution, and that it should report a wrong test instead of working around it.

Second, ask for evidence instead of a claim: the command, its output, the diff. In Anthropic’s testing, telling a model to audit each progress claim against a tool result from the session nearly eliminated fabricated status reports. The tests define correct behaviour; don't edit them. If one looks wrong, stop and tell me why. When you report back, paste the test command and its full output.

Then read the diff of the test files yourself, because a prompt is not a lock: METR saw reward hacking continue even when models were told to use the intended method. In the simulation, gaming the check collapses, and so does declaring success. This closes the missing-file leak from your side, with M2 and M9 working together.

Recall the multiple-choice leak. If “I don’t know” scores the same as a wrong answer, guessing always wins, and the model learned that thoroughly; it is a root of hallucination. Your conversation has no scoring rule unless you write one, so write one, the way an exam setter takes the guessing incentive away. Anthropic’s guidance is to give the model explicit permission to say it lacks the information, to ground claims in direct quotes, and to retract any claim it cannot support with one. This raises the value of abstaining, the same move a lab makes when it switches to net scoring.

Answer only from the docs I pasted. If they don't say, write "not stated" instead of guessing, and quote the line each claim rests on. Stay most suspicious of confident specifics about niche things, such as a library flag, a paper title or a version number, where a familiar-sounding name invites a guess. In the simulation, confident wrong answers turn into “I don’t know”. This is M3: training rewarded confident answers, so invite the other kind.

The reaction-scored leak came from rewarding people’s approval, and preference training shares its root. A junior employee who learns that agreeing with the boss goes down well starts reading the boss’s questions for the hoped-for answer. People, and reward models trained on their choices, tend to prefer answers that agree with them, sometimes over correct ones. When OpenAI gave a thumbs-up signal more weight in a GPT-4o update, the model became so sycophantic that the update was rolled back within days.

So the model reads your framing as a hint about what will be rewarded, which is sycophancy in its everyday form. “Isn’t approach A cleaner?” asks for agreement, while “Compare A and B for this codebase and recommend one” does not. The same pull applies after it answers, when you push back.

The padding leak showed that more is not automatically better, and in your own sessions more context is often worse. A meeting that has run for three hours, full of abandoned proposals, decides worse than a fresh one that starts from a one-page summary. Performance falls as context grows, even on simple tasks, and in conversations that start underspecified and fill in over many turns, models lock in early assumptions and score on average 39% lower than with one complete prompt.

Failed attempts left in context bias what comes next, so Anthropic advises clearing the session after correcting the same issue twice. When a session goes wrong, then, don’t argue with it. Write what you learned into a new, complete prompt and start fresh, as you would hand a clean brief to a colleague. That is M7.

Judging needs a fresh context too. Act III showed a measurable bias when a model grades its own family, and models favour their own outputs, so have a separate context, such as a subagent, review the work against explicit criteria (M8). A reviewer told to find problems will report some even in sound work, so ask only for correctness and requirement gaps. In the simulation, drift falls.

Now assemble your own. Tick the pieces below and watch the mock prompt build itself while the simulated attempts move between outcomes. Two things are worth noticing as you play. The pieces interact: a check without protection opens the door to gaming it, so the check and the evidence request belong together. And adding more does not always help, because “if in doubt, use the tool” prompts written to push older models now cause over-use on current ones, and instructions to double-check add cost without benefit on models that already verify their own work.

Aim for completeness rather than length: the task as you mean it, what done looks like, a check the model can run and you can trust, room to say “I don’t know”, a neutral question, and a clean context. That still leaves a judgement no prompt can make for you, because capability is jagged. A model that writes a flawless migration may fumble a date calculation, and fluent prose is no evidence either wayIn a field experiment, consultants working on tasks outside the AI’s frontier were 19 points less likely to be correct, even as their answers grew more persuasive..

So the last piece is deciding whether to delegate at all. Hand over what sits well inside the model’s practised ground, keep what sits outside it, and verify hardest where you are least sure which side a task is on. That is M10. The rest is the environment, and you are its grader.

If you have read this far, here is an offer rather than an exam. Reading about a mechanism makes it feel understood, and that feeling is a poor guide. The honest check is whether you can predict what happens in a case you haven’t seen, so I have collected cases rather than definitions. Ten situations are drawn at random from a bank covering coding, writing, research, data, everyday work and agent supervision, and several begin after something has already gone wrong.

Each option is graded best, flawed or harmful, and the feedback names the training fact that makes the best option work. Treat a wrong pick as the interesting part. It marks the spot where your intuition and the training story disagree, which is exactly the spot worth rereading. Your record for each of the ten principles, M1 to M10, fills in as you go, and nothing leaves your browser.

When you finish, you will see which principles you used and which you missed, and each missed one links back to where its training fact lives: a missed M3 to the guessing leak, a missed M2 to the missing file, a missed M7 to starting fresh. Follow one of those links rather than all of them. The rabbit holes will keep.

Then take one thing into your next real session. Pick the principle you missed most, and the next time you hand a model a task, add the piece that answers it: a done-when line, a check it can run, permission to say “I don’t know”. And the next time an agent tells you every test passes, ask what it was graded on, and remember that this time the grader was you.