Auditing FrontierCode

FrontierCode (Cognition) grades an agent’s patch on maintainer-authored rubrics and asks the question its leaderboard leads with: “would the maintainer actually merge this PR?” The conclusion offers it as the instrument enterprises and researchers “can trust… to evaluate the production readiness of their strongest models.” I ran it through the how-to-audit checklist, then measured its construct against a population where the answer key exists.

Full audit, with receipts, protocols, and per-finding falsifiers: https://github.com/kimjune01/frontiercode-audit. The findings went to Cognition as a right-of-reply issue before this post.

FrontierCode is a well-QC’d instrument that measures patch quality under a frozen rubric, and it is marketed as mergeability, which is a decision maintainers make mostly on information the grader never sees.

What it gets right

• The diagnosis is correct. Autotest-only grading passes patches maintainers would reject. Their SWE-bench critique and my field data agree from opposite directions.
• Reverse-classical testing is a good deterministic invention. The agent’s tests must fail on the base commit, which proves the tests exercise the change.
• The QC pipeline is real. Hack reports run in both directions (the author fakes a lazy solution to catch false positives, then a valid alternative to catch false negatives), rubrics are calibrated against four solutions spanning 0 to 100%, and researchers solve a random subset themselves.
• The private task set is an anti-contamination credit. The scored artifact cannot be memorized from the public web.
• v1.1 shows a live internal audit loop. They audited 1,000+ criteria and relaxed 75 overly strict ones. The loop existing is a credit; the audit’s central problem is that only insiders can run it.

Some claims outrun the metric

• Construct. The claimed construct is the merge decision; the measured construct is the diff. The rubric’s blockers are “criteria that a maintainer would consider hard stops during code review,” all patch properties. No axis scores the description, the interaction, or anything else maintainers demonstrably decide on. F0, verbatim-pinned.
• Receipts. No scored trial’s artifact is publicly retrievable, and the public leaderboard cannot be re-derived from anything released. Six of the checklist’s eleven checks are blocked, including run-the-answer-key, which found shipped gold defects in all three benchmarks it previously touched. Epoch relays rather than re-runs, in their own words: “We source results from Cognition’s public FrontierCode data.” F6.
• The oracle is an unnamed LLM. Three grading methods put an LLM inside the verdict, and every published reference says only “an LLM”. No model name, no version, no pinning policy. Readers cannot assess judge drift across revisions or the judge’s independence from the systems it grades. F7.
• The board ranks configurations. Models ran under different harnesses (the chart legend admits it: “marker shapes distinguish harnesses”) at each model’s best-performing reasoning effort, and the numbered column reads as a model ranking. Ranks 4 through 7 sat within 0.7 points at capture, with no published grader-side variance to say whether that separates anything. F2, F7.
• Deprecated scores still circulate. v1.1 retired the Diamond subset the launch press quoted, without a disclosed reason; Epoch’s chart reports the Diamond score today, and no comparability guidance exists. F5.
• No ownership disclosure. SWE-1.7 is scored on the board, its launch post says the model and the benchmark share design principles, and no page carries a conflict statement. Shared principles can legitimately produce a high score. The public evidence cannot distinguish that from benchmark-specific tuning, because the validation that would settle it is impossible from the released artifacts. Governance.

The measurement

This section measures the construct gap instead of arguing it. I took all 98 closed-unmerged PRs from my own autonomous contribution pipeline, a population where almost every closure has a thread stating why. Two coders from different model families coded every closure from the thread itself, blind to each other, under a pre-registered protocol with a pre-registered falsifier. Self-closes were excluded; silent closures went to an unknown bucket. That left 59 confidently coded maintainer closures.

In at least 52 of the 59, both coders found the deciding cause outside FrontierCode’s six rubric axes. In exactly 3, both found a patch-quality cause decisive. The falsifier (a majority deciding on the rubric axes) did not come close to triggering.

The off-diff majority splits into a social layer (who submitted, how, and with what words) and an ecosystem layer (what the world did around the patch). Neither is visible to a patch grader.

The table is the primary coder’s call on all 59; the blind second coder agreed on the off-diff side in 52, on patch quality in 3, and split on 4. One closer wrote “This is mostly OK, but I read your profile.” And four “closed” PRs were accepted work, cherry-picked or ported with credit, so the closed/merged binary itself mislabels outcomes.

This population is one AI-assisted contributor shipping small fixes during a public campaign, which enriches identity closures. It also sits far from FrontierCode’s task difficulty, where patch correctness may bind first. The 3-of-59 floor is a property of this population. What it establishes is that a population exists, and a growing class of contributors resembles it, where nine times out of ten the deciding cause lies outside anything a patch grader can see.

The one public task

FrontierCode publishes exactly one inspectable task-level grading receipt, the interactive demo on its leaderboard. In it, the top model fails 2 of 10 criteria. Both are blockers, and both encode the same requirement: multi-line warnings must route through one chained logging call. The brief’s operative sentence says to use the new helper “in every instance of warning: <message> messages,” a continuation line carries no warning: prefix, and the failed solution converted every line that does. Cognition’s own commentary notes the two solutions are “behaviorally the same.” One requirement the brief underdetermines, instantiated as two blockers, decides the only public verdict. The writeup presents it as “models fail this task in a somewhat surprising way.” The determinacy read, with the full rubric.

What I’d report instead

Report the agreement rate between the instrument’s verdict and the field outcome. The audit calls this ecological accuracy, ecological validity’s quantitative form. FrontierCode claims to measure a field quantity, and adjudicated ground truth for “would merge” exists in bulk in git history, so the claim is checkable as a number. A benchmark that reports it converts “trust it” into a calibration curve. The itemized disclosure minimum is in the audit, and every finding carries a falsifier. A judge-family rank-stability ablation, for instance, kills the precision finding.

The audit’s recommendation section, marked as recommendation, goes further: grade the hypothesis graph, the reasoning layer as a replayable artifact. Grading raw description prose would train smooth-talkers, and grading nothing leaves the decisive layer unmeasured. That design is mine, freely licensed, and disclosed as such in the repo.

One-sided by design

The audit reports defects it can exhibit, and it treats an absence of findings as an absence of findings rather than a clearance. Every quantitative claim traces to a pinned page snapshot, a coded thread with a verbatim quote, or a re-runnable protocol. The pre-registered falsifier that failed to trigger is reported next to the ones that fired. Corrections are welcome and will be linked: the right-of-reply issue went out before publication, after earlier emails went unanswered. FrontierCode is a good instrument for a real capability. It measures a narrower thing than its headline, and the narrowing favors SWE-1.7’s position on its board.

Disclosure: the measured population is my own pipeline’s pull requests, which made every closure’s ground truth readable and bounds the floor to this population; the coding protocol, both coders’ outputs, and every quote are public for re-derivation. The hypothesis graph recommended in the audit is my own published, freely licensed design.