Auditing MirrorCode

MirrorCode (Epoch AI and METR) hands an AI an execute-only binary plus its docs and asks it to rebuild the whole program, graded byte-exact against the reference. Their headline asks: “What’s the largest software project AI can complete on its own?” I ran it through the how-to-audit checklist.

Full audit, re-runnable, with a receipt for every claim: https://github.com/kimjune01/mirrorcode-audit

MirrorCode is a well-built instrument that measures scoped reimplementation from a working oracle, partly via recall of published specs the artifact can’t teach and of these specific programs, and is marketed as autonomous creation.

What it gets right

MirrorCode is better-built than most, on several axes better than the six benchmarks I audited before it. Full credit, sourced.

• Cheat-proofing is real. Four isolated containers, scoring where the agent can’t reach it, and an execute-only reference enforced with seccomp-BPF, Landlock, and RLIMIT_CORE=0 that blocks every byte-read path to the binary, and the scoring isolation defeated a live Gemini binary-wrap cheat.
• I/O-only grading sidesteps the frame sin. The oracle is (stdout, stderr, exit), not final environment state, so the destructive-completion trap that broke Terminal-Bench can’t arise.
• It did the two things most benchmarks skip: a human baseline and a memorization screen. Both incomplete, both worth attempting.
• Selection drained the recall surface. A complete per-target read finds 2 recall targets out of 25, and none is a hash, image, or media target.
• Not saturated. 8 of 25 targets were never solved to 100%, and the large ones sit near zero, so the instrument still discriminates at the frontier.
• The paper body is candid. It concedes the code is piecemeal, “would not be merged,” and that the results don’t show AI can do arbitrary implementation work. The overreach is in the title, not the methods section.

Some claims outrun the metric

• Construct validity. MirrorCode’s title claims autonomous whole-program builds. The metric scores reimplementation from a live reference oracle, something narrower than the title names. The gap, laid out.
• The human-labor claim is unmeasured belief. Every “a human would take weeks/months” is “we believe,” from four contributors with a sevenfold spread and no completed baseline. The targets are open-source, so their git histories anchor the real labor. Creation runs to tens of developer-weeks, while MirrorCode’s own baseline puts reimplementation at days. The marketing evokes the first and measures the second.
• Two targets are recall by construction (below), so a pass means recalling a spec rather than reconstructing it. Witnesses.
• Contamination, unquantifiable. MirrorCode’s own screen flags 17 of 25 targets as screen-positive for memorization. Whether recall inflates the 56% headline cannot be computed from released data, because the join that would show it was never published.
• The receipts weren’t shipped. The per-task, per-model outcome grid is a figure in the paper, so publishing it as data would cost almost nothing. The run records still aren’t public.
• “Entire program” flattens heterogeneous units. cal and uuidparse are tiny tools inside util-linux. bib2json is one format pair of pandoc. Each scope is legitimate, but the aggregate phrase hides how different the tested surfaces are.

The two recall witnesses

Each carries a re-fetchable receipt. Per-target audit.

• brotlid decodes a bundled Brotli stream to exact bytes. That needs the RFC-7932 decoder and its ~120 KB static dictionary, neither in the docs nor derivable from the visible examples. MirrorCode defends it as “fully determined by the documented format,” which conflates the RFC with the bundled docs.
• mailauth arc-seal must emit an exact RSA-SHA256 signature. Reproducing those bytes requires RSA, SHA-256, and the ARC canonicalization rules from the spec. Receipt.

What I’d report instead

MirrorCode asks three questions: how large a program, how much faster than a human, and how reliably. Collapsing them into “56%” is the problem. The constructive fix is a preregistered size-by-time-by-success profile. Plot whole-task success and time-to-success against program size, censor failures rather than dropping them, split out contamination, and show uncertainty so the curve can’t imply precision it lacks.

Wall-clock is the capability axis, not dollars, since a model’s compute is negligible against a human SWE’s time. The audit can’t build this profile, because the run records aren’t public. So the ask is the smaller one: publish the per-task, per-model results that already sit in the paper as a figure, and validate or dispute the findings against the repo.

Per-target, all 25

The complete read gives every target a verdict and sweeps the full rejection taxonomy from the prior audits: recall, implementation-pinned render, undiscoverable entry point, self-capturing golden, scale, and the non-determinism family. The tally across all 25: 2 recall, 7 scale, 13 clean, 3 private unread.

One-sided by design

The audit is one-sided by design. It can show a defect, never the absence of one. Every quantitative claim traces to a cited paper section, a public repo file, or a re-runnable script, with per-cell figure reads labeled as such. The recall column is verified from the graded I/O, while the memorization and solve columns are figure reads, exact only in aggregate. Corrections are welcome, and re-derivable against the receipts. MirrorCode is a good instrument for a real capability. It measures a narrower thing than its title, and the narrowing favors the headline.

Disclosure: I applied for a role at Epoch AI, which co-produced MirrorCode, and didn’t get it. This audit uses only public artifacts and is re-runnable, so every claim traces to a cited receipt and stands or falls independent of me. My own check 4 says a producer’s relationships are a conflict when the artifact is asked to be science, and that rule applies to the auditor too.