An open, practical comparison of code-review models in one real review harness. Quality first, then completion, speed, and cost.
Current recommendation
Sol is the best balanced candidate, Grok 4.6 is the complete-cohort recall leader, and Opus 5 has the strongest partial recall signal. Opus remains conditional because two reviews were unscoreable.
openai/gpt-5.6-sol · Azure
x-ai/grok-4.6 · xAI
anthropic/claude-opus-5 · Amazon Bedrock
A reviewer earns credit for identifying an evidence-backed defect once. Rephrasing it, guessing loudly, or padding the review does not help.
Defect recall, precision, clean-review restraint, severity judgment, and consistency across repeated attempts.
One-to-one matching, proof-backed gold findings, human adjudication, and visible uncertainty for incomplete labels.
Latency is reported after review quality, with failures and budget exhaustion kept separate from successful reviews.
Observed token and cost data where available. Unknown cost is null—never quietly treated as free.
Hold prompt, tools, budgets, context, scorer, and task release still. Change the model or provider endpoint.
Measure the whole review system: its model, prompt, tools, parsing, and orchestration. OpenRouter Review Bot and Grok Code Review Bot are first.
Current state
The first sweep is a small, one-pass screening run—not a universal measure of coding ability. It is already useful for choosing models in the OpenRouter Review Bot, and future runs will add harder cases, better provider controls, and repeated finalist tests.