Contracts and calibration
completePublic schemas, neutral task format, deterministic scorer, planted defects, clean twin, private pilot shape, and exact-SHA product adapters.
Release gates
Slow enough
to be credible.
This is a volunteer open-source project. The roadmap favors a small defensible benchmark over a large fragile leaderboard.
Real-PR corpus
activeSource license-compatible changes, reconstruct exact checkouts, validate defects with tests and fixing evidence, add clean controls, and blind the held-out split.
Judge calibration
nextCompare semantic matching against blinded human dispositions, publish a conformance set, and measure disagreement and variance.
Pilot runs
nextRun both products at least three times per task, adjudicate every unmatched finding, audit costs and provider routing, and rehearse private-to-public distillation.
First official release
nextFreeze the release, reproduce the run, publish reviewed aggregate records, and surface confidence intervals instead of overstating small score gaps.
Intentionally not now
No platform theater.
- No accounts, billing, or database for a static results site.
- No commercial-tool claims until an adapter can reproduce their actual review conditions.
- No paid large matrix before public smoke tests and cost bounds pass.
- No official ranking from the synthetic pilot corpus.