Contracts and calibration
completePublic schemas, neutral task format, deterministic scorer, planted defects, clean twin, private pilot shape, and exact-SHA product adapters.
Release gates
Slow enough
to be credible.
This is a hobby project with a practical goal: choose better code-review models. The roadmap favors useful, repeatable comparisons and transparent limitations over certification theater.
First model comparison
complete18 candidate models, three attempts across 11 held-out reviews, provider-reported time and cost, and operational failures preserved.
Harden the run path
completeRehearse publication without model calls, use per-run guards as runaway protection, diagnose provider failures, and ensure completed campaigns can reach the site.
Keep the board current
activeTest newly released models, rerun strong candidates when providers or harness behavior change, and keep partial runs separate from complete cohorts.
Expand the challenge set
nextAdd harder, evidence-backed tasks across more languages, frameworks, databases, frontend stacks, CSS, and cloud configuration without exposing the held-out answers.
Intentionally not now
No platform theater.
- No accounts, billing, or database for a static results site.
- No commercial-tool claims until an adapter can reproduce their actual review conditions.
- No paid large matrix before the exact publication path passes a no-spend rehearsal.
- No claim that AI-assessed provisional precision is human-certified.