Grade the defect.
Audit the claim.

Freeze realistic review work

Each task contains an exact diff, an inert repository checkout, source and license metadata, a declared context envelope, and zero or more hidden findings. Related revisions and clean twins share a family ID and stay in one split.

Public fixtures exist for reproduction and harness development. Held-out releases combine recent real changes, owned cases, and fresh mutations to reduce direct contamination risk.

Prove the answer key

A human comment is a candidate, not truth. Admission requires a defect statement, trigger, impact, and proof: a regression test, a fixing commit with a boundary argument, a primary specification, a reproducible trace, or explicit independent adjudication.

Clean controls are equally important. A useful reviewer must know when not to comment.

Match issues one-to-one

The scorer finds a maximum one-to-one assignment between reported findings and gold defects. Two comments about the same underlying bug cannot create two recall credits.

Unmatched findings become valid_extra, false_positive, or pending. Pending is visible work for adjudicators, not an automatic penalty.

Report quality before operations

Primary reporting includes task-macro recall, adjudicated precision, duplicate rate, clean-control pass rate, severity agreement, and detection frequency across attempts. Latency, tokens, and cost follow.

Malformed output, refusal, truncation, budget exhaustion, provider failure, and adapter failure remain separate operational outcomes. Unknown cost is null, not zero.

Keep two questions separate

The fixed-harness track asks which model performs better under one frozen prompt, tool set, budget, and provider policy. The end-to-end track asks which complete review product performs better. Those results are never pooled.

Publish a receipt, not private evidence

Official records carry exact product, runner, benchmark, and scorer SHAs plus aggregate quality and operations. Raw held-out prompts, output, labels, task identifiers, canaries, and adjudicator notes remain private.