Auditing SlopCodeBench

SlopCodeBench asks whether iteration degrades agent code, and answers that agents lack the design discipline iterative development demands. The instrument between question and answer is a pair of static code-shape metrics that usually rise as specifications expand. Connecting the two needs two links: that the metrics measure extension robustness, and that iteration moved them. The paper supplies neither. Construct receipt.

I audit public benchmarks to make them better. This one asks a question worth answering, so it gets the full treatment.

The ruler misses

Structural erosion is the share of complexity concentrated in functions above a cyclomatic-complexity threshold. The paper reports its correlation with passing the next checkpoint: −0.018. Lines of code reaches −0.212. For next-checkpoint cost, erosion reaches 0.167 and LOC 0.502. A plain line count reads the future better than the instrument built for it.

Verbosity has a different problem. Its 137 rules were developed partly from observed agent code, then used to conclude that agent code is 2.3 times as verbose as human repositories. A ruler built partly from one population’s characteristic marks will distinguish that population by construction. The comparison needs to survive on clone coverage alone, on rules developed without the evaluated agents, or under blinded human judgment. None is reported.

Iteration is not the treatment

Every checkpoint adds requirements. Later code has undergone more edits, has more work to do, and faces a harder specification. The experiment changes all three together.

The comparison against commits from 473 unrelated Python repositories does not separate them. Those commits are a calibration panel. Nobody reimplemented the cumulative specifications from scratch.

Under SlopCodeBench’s combination of repeated editing and expanding scope, its static metrics rise. “Iteration causes degradation” requires the missing counterfactual.

The frontier was selected

The authors removed proposed problems that frontier agents could solve in one shot, then reported that no agent solved a surviving problem end to end. Saturation filtering can preserve headroom. It also conditions the result on model failure.

The paper does not report the screening models and versions, the number removed, an unfiltered comparison, or an independent human baseline. The low score is performance on a set selected partly because frontier agents failed it, not an unconditioned estimate of iterative coding ability. Selection receipt.

The spec is not the contract

At each checkpoint the agent sees one spec file, its own prior code, and nothing else. The tests stay hidden. That design is sound when the tests grade what the spec states. They don’t always. In a 12-problem sample covering 65 checkpoints, 28 of 343 core tests assert exact values no visible spec states, implies by rule, or shows in an example, and one more is contested. Core tests alone decide checkpoint success. Each of the 28 survived an adversarial pass instructed to refute it under the paper’s own definition of core: “functionality explicitly mentioned or shown in the specification.” Oracle receipt.

• The trajectory_api answer key returns 200 on create when a toolpack is active. Every visible spec says creation returns 201. The special case exists in two places: the gold and the test that grades it.
• The dynamic_buffer spec offers three JavaScript interface shapes and mandates a usage pattern. The hidden harness instantiates a class. An implementation of the spec’s own mandatory pattern fails every JavaScript case.
• Three problems ship checkpoint-n tests that require conventions the spec introduces at checkpoint n+1. cfgpipe error tests demand the literal word “duplicate” one checkpoint before the spec first uses it. A dag_execution core test writes unquoted list syntax first shown a checkpoint later. datagate tests configure the cache through an environment variable the spec names a checkpoint later. An agent that implements exactly what it has been shown fails; one that guesses the next spec passes.
• The meshctl spec says “Vault error order is not part of the contract.” A test asserts the exact order. Determinacy receipt.

This is a guessing game. For example, a dynamic_buffer core test shows the agent this sample and asks it to infer the filter:

{id 1, Hello,      normal}  → kept        {id 2, Buy now,    spam}  → dropped
{id 3, Meeting,    normal}  → kept        {id 4, Free money, spam}  → dropped
{id 5, Report,     normal}  → kept

The spec’s grammar allows comparisons to constants under AND/OR, with no preference rule. Two hypotheses reproduce the sample exactly: category != "spam", and id != 2 AND id != 4. The hidden data contains ids 10 through 14, so the first passes and the second keeps every row and fails. Everything the agent can see supports both. The information that separates them exists only inside the test the agent is not shown.

For those checkpoints the benchmark grades agreement with unpublished author conventions, and one wrong guess fails the checkpoint.

Nobody disclosed a red team

The paper describes coauthor review and agent-assisted refinement of ambiguous tests. That is maker QC. It does not disclose an independent team asked to break the construct, run the answer keys, mutate passing golds, or rederive the scores. As far as the paper discloses, this audit is the first adversarial contact the evaluator has had.

• All 196 answer keys ran through the benchmark’s own harness in its own Docker image. Four are deterministically defective: dynamic_buffer checkpoints 2 through 4 and env_manager checkpoint 3. One is unstable: pwd_manager checkpoint 4 fails a different regression test on each run. Eight more grade TypeScript through an unpinned npx toolchain that resolves to a broken version today. Gold sweep receipt.
• A wrapper deletes a test-unreferenced file from the passing trajectory_api workspace and delegates to the unchanged gold. The probe confirms the file is gone; all 373 tests still pass. Frame receipt.
• No scored model workspace or per-checkpoint result is retrievable from the linked public artifacts. The paper reports GPT-5.5 at 29 of 196 strict checkpoints; the current leaderboard reports 28. Without trial receipts, that is version drift, not a proven arithmetic error. The embedded data also contains multiple rows with identical displayed configurations and different scores. Score receipt.

No independent adversarial validity audit is disclosed, and the internal process missed defects across the claim, selection, gold, oracle, spec, frame, and score clauses. Review-process receipt.

The next experiment

At checkpoint n, give one agent its checkpoint n−1 workspace. Give another an empty workspace plus the complete specification through n. Hold model, harness, budget, specification, and tests fixed. The difference estimates the cost of accumulated history; their shared decline estimates task growth.

Then validate erosion against an external outcome: defects in checkpoint n+1, maintenance time, or success by a blinded agent inheriting the code. Control for LOC. If the metric does not predict future maintenance cost, call it static code-shape drift.

Publish the sampling funnel: every candidate, exclusion, screening model, and one-shot result. Test preservation requirements, run mutation and frame-breaking probes, and verify every gold in the public harness. Release immutable benchmark versions, scored workspaces, per-checkpoint rows, and the leaderboard calculation.

Commission an independent team to attack the evaluator before release, then publish the attacks and fixes.

Until those controls exist, report static code-shape drift under iterative specification refinement. The next version needs a control group more than another model.

Reproduce the audit: pinned sources, scripts, tests, and finding-level receipts.

Disclosure: Claude challenged candidate findings during the audit. Its objections narrowed the claims; the receipts decide them.