Terminal-Bench is blind to destruction
Official grading scored 83 of 83 runs as passes after off-task user assets were wiped. The audit produced an executable frame gate and upstream patches for Harbor and Inspect.
Research
I audit the inference from benchmark result to capability claim.
I find the hidden assumption, construct the test that can break it, and return evidence someone else can replay.
Selected findings
SWE-bench Pro contains underdetermined tasks
The audit proved a 15% floor of tasks whose graded value no reading of the materials determines, and shipped the cases, grading receipts, and regrade path.
DeepSWE's headline hid task-level grading defects
Four of 113 answer keys failed their own verifiers. All four climbed in the later re-grade while the pooled rate stayed flat, and the revised v1.1, re-audited, mostly held.
ProgramBench measures recall
Twenty-one tasks carry a verified recall-only witness, so the zero-percent floor across nine models measured recall of published algorithms rather than reconstruction.
FrontierCode, MirrorCode, and τ²-bench
Repeated audits expose the same boundary failure in different forms: the grader certifies less than the benchmark claims.
Method
- Name the capability claim and the conclusion drawn from its score.
- Read the task, environment, and grader as one measurement instrument.
- Construct the smallest counterexample that separates the claim from the check.
- Publish pinned artifacts, observed outcomes, and a regrade command.
- Disclose the finding and send the fix upstream.
How to Audit a Benchmark gives the full operational checklist. Assurance at the Boundary collects eight public-side audits and their receipts.
Research mechanisms
- Local replay auditability returns a check instead of a static verdict, so a later auditor can re-run one consequential claim.
- The Hypothesis Graph binds claims to trials and records how a failed trial motivates the next hypothesis.
- Verifiable Knowledge treats claim, provenance, kill condition, and replay as the unit transmitted between investigators.
Papers
- The Power Diagram Auction: A Formally Verified VCG Mechanism for LLM Advertising
- Local Replay Auditability: Verification That Returns a Check
- Verifiable Knowledge for AI-Mediated Science
- The Hypothesis Graph as Machine-Executable Research Memory
- What Cannot Be False Cannot Be True
About · GitHub · june@june.kim