Use early evidence without overselling it
The current board is a three-attempt-per-task model comparison. It is intended to choose useful models for the OpenRouter Review Bot, not to certify a universally best reviewer. Known-issue recall leads, while incomplete reviews and provider failures remain visible.
Unmatched findings begin as pending. Where the board shows AI-assessed precision, an AI review has classified those claims against frozen evidence; that assessment remains provisional. Findings with insufficient evidence stay unresolved. Neither a model's confidence nor agreement between repeated reviews establishes correctness.
Freeze realistic review work
Each task contains an exact diff, an inert repository checkout, source and license metadata, a declared context envelope, and zero or more hidden findings. Related revisions and clean twins share a family ID and stay in one split.
Public fixtures exist for reproduction and harness development. Held-out releases combine recent real changes, owned cases, and fresh mutations to reduce direct contamination risk.
Prove the answer key
A human comment is a candidate, not truth. Admission requires a defect statement, trigger, impact, and proof: a regression test, a fixing commit with a boundary argument, a primary specification, a reproducible trace, or an evidence-backed expert disposition.
Clean controls are equally important. A useful reviewer must know when not to comment.
Match issues one-to-one
The scorer finds a maximum one-to-one assignment between reported findings and gold defects. Two comments about the same underlying bug cannot create two recall credits.
Unmatched findings become valid_extra, false_positive, or pending. Pending is visible work for adjudicators, not an automatic penalty.
When the comparison labels a result AI-assessed provisional, an AI judge has checked the frozen code and evidence for unmatched findings. The private review distinguishes valid extras, scorer matcher misses, false positives, repeated reports, out-of-scope claims, and insufficient evidence. Valid extras and matcher misses count as correct for provisional precision; false positives, repeats, and out-of-scope claims count as incorrect; insufficient-evidence cases remain unresolved and outside the denominator. Only aggregate counts cross the publication boundary.
The board preserves the scorer's original seeded-issue recall and original pending count. A separately labeled assessed recall may credit a matcher miss only when its private gold identity was not already credited in that attempt. It never silently replaces the baseline recall.
Report quality before operations
Primary reporting includes task-macro recall, adjudicated precision, duplicate rate, clean-control pass rate, severity agreement, and detection frequency across attempts. Latency, tokens, and cost follow.
Malformed output, refusal, truncation, budget exhaustion, provider failure, and adapter failure remain separate operational outcomes. Unknown cost is null, not zero.
Provider routing and service tier affect price and latency. A row labeled Flex measures that tier, including provider waits and retries within the review. Its median is not a latency promise for standard service or future runs. Known cost includes upstream provider charges when a linked provider key is used.
Keep two questions separate
The fixed-harness track asks which model performs better under one frozen prompt, tool set, budget, and provider policy. The end-to-end track asks which complete review product performs better. Those results are never pooled.
Publish a receipt, not private evidence
Screening pages publish aggregate metrics and workflow links only. Future immutable records carry exact product, runner, benchmark, and scorer SHAs plus aggregate quality and operations. Raw held-out prompts, output, labels, task identifiers, canaries, and private notes remain private.
Read the canonical specification ↗