Review Benchmark
OpenRouter model comparison · October 19, 2026

Quality first.
Failures included.

18 models, 3 attempts across 11 held-out code changes, inside the OpenRouter Review Bot. This is a practical product-selection comparison, not a universal model ranking.

Ranked recommendation

Which model should the bot run?

Recall leads. The rank starts with AI-assessed known-issue recall, then considers provisional assessed precision, completion, speed, and provider-reported cost.

AI-assessed, still provisional. Every previously unmatched finding has been reviewed against frozen evidence; 390 remain unresolved. Unresolved findings are excluded from provisional precision. The original scorer recall remains visible; these results are useful for product selection, not presented as human-certified.

Targeted repairs stay visible. 5 rows combine successful original reviews with reruns of only the missing cells. Repaired rows are labeled separately and report all-in cost.

Cost is observed. Row cost describes the displayed comparison evidence. Total spend represented by this snapshot—including billed operational failures and corrected reruns—is $186.23.

Run context. Rolling comparison of the same 11 changes, three attempts and a 50-tool-turn budget. Earlier rows retain their original product revisions; Argon uses the October 19 Gemini protocol fix. Most routes require ZDR; Astra Flex used an authorized non-ZDR route. AI assessment remains provisional. Review costs exclude assessment and smoke probes. A separate GLM/Together timeout remains unranked and outside exact spend because its cost is unknown.

01

Claude Opus 5

anthropic/claude-opus-5 · Amazon Bedrock

High known-issue recall, but high finding volume, low assessed precision, and the highest run cost in the cohort.
80% AI-assessed recall · baseline scorer recall 73.3% 12.0 unmatched findings/review · 380 initially pending 27/33 gold matched after matcher review AI-assessed provisional precision 25.4% · 0 unresolved 100% of initially unmatched claims resolved 77 valid extra · 2 matcher miss · 177 false positive · 5 repeated · 119 out of scope
33/33 Complete
6.1 minper attempted review
$41.42 provider reported Workflow run ↗
02

Muse Spark 1.2

meta/muse-spark-1.2 · Meta

Complete local run without ZDR; unusually high finding volume makes provisional precision especially important.
78.3% AI-assessed recall · baseline scorer recall 75% 7.2 unmatched findings/review · 216 initially pending 26/33 gold matched after matcher review AI-assessed provisional precision 21.6% · 0 unresolved 100% of initially unmatched claims resolved 29 valid extra · 1 matcher miss · 127 false positive · 2 repeated · 57 out of scope
33/33 Complete
1.5 minper attempted review
$3.71 provider reported Local Docker run
03

Grok 4.7

x-ai/grok-4.7 · xAI

Sep 23 xAI ZDR run: 15 original successes plus 18 targeted repairs. 167 claims remain unresolved after conservative AI review, so provisional precision covers only resolved claims. Cost includes the original campaign and completion.
73.3% AI-assessed recall · baseline scorer recall 73.3% 8.3 unmatched findings/review · 263 initially pending 25/33 gold matched after matcher review AI-assessed provisional precision 31.7% · 167 unresolved 36.5% of initially unmatched claims resolved 11 valid extra · 0 matcher miss · 23 false positive · 0 repeated · 62 out of scope
33/33 Complete after repair Initially 15/33 · repaired 18/18
13.2 minper attempted review
$16.16 all-in provider reported $9.46 targeted repair Preserved run + targeted repair
04

GPT-6 Astra (Flex)

openai/gpt-6-astra · OpenAI

Run September 5 via pinned OpenAI Flex without fallbacks; non-ZDR routing authorized. All 33 reviews confirmed Flex and complete cost, including upstream BYOK charges. Offline finalization repair repeated no reviews. Model-aware Codex assessment; 12 unresolved findings are excluded from assessed precision. Timing and cost include cache effects and describe this cohort, not standard service.
70% AI-assessed recall · baseline scorer recall 70% 1.8 unmatched findings/review · 59 initially pending 24/33 gold matched after matcher review AI-assessed provisional precision 80.1% · 12 unresolved 79.7% of initially unmatched claims resolved 30 valid extra · 0 matcher miss · 5 false positive · 0 repeated · 12 out of scope
33/33 Complete
30 sper attempted review
$2.97 provider reported Local Docker run
05

Gemini 4 Argon

google/gemini-4-argon · Google

October 19 Google Vertex ZDR run with the merged Gemini 4 protocol fix; earlier rows retain their original builds. 37 claims remain unresolved after conservative model-blind AI review, so provisional precision covers only resolved claims. All 33 reviews completed without salvage.
70% AI-assessed recall · baseline scorer recall 70% 2.1 unmatched findings/review · 66 initially pending 24/33 gold matched after matcher review AI-assessed provisional precision 70.5% · 37 unresolved 43.9% of initially unmatched claims resolved 6 valid extra · 0 matcher miss · 4 false positive · 0 repeated · 19 out of scope
33/33 Complete
1.6 minper attempted review
$4.76 provider reported Local Docker run
06

GLM 5.3

z-ai/glm-5.3 · Fireworks

Strong recall and moderate assessed precision at low cost, but the Fireworks run had the slowest median review.
70% AI-assessed recall · baseline scorer recall 70% 5.6 unmatched findings/review · 174 initially pending 24/33 gold matched after matcher review AI-assessed provisional precision 33.1% · 0 unresolved 100% of initially unmatched claims resolved 43 valid extra · 0 matcher miss · 78 false positive · 5 repeated · 48 out of scope
33/33 Complete
17.6 minper attempted review
$14.20 provider reported Workflow run ↗
07

GPT-5.6 Sol

openai/gpt-5.6-sol · Azure

Full completion, solid recall, moderate precision, and fast reviews at a midrange cost.
66.7% AI-assessed recall · baseline scorer recall 56.7% 2.5 unmatched findings/review · 81 initially pending 22/33 gold matched after matcher review AI-assessed provisional precision 55.3% · 0 unresolved 100% of initially unmatched claims resolved 30 valid extra · 3 matcher miss · 32 false positive · 0 repeated · 16 out of scope
33/33 Complete
1.6 minper attempted review
$7.13 provider reported Workflow run ↗
08

Claude Opus 5.5

anthropic/claude-opus-5.5 · Amazon Bedrock

Sep 23 Amazon Bedrock ZDR run. More known issues found than Sol, but high report volume and 89 unresolved claims limit the precision estimate.
66.7% AI-assessed recall · baseline scorer recall 66.7% 6.0 unmatched findings/review · 191 initially pending 23/33 gold matched after matcher review AI-assessed provisional precision 24.5% · 89 unresolved 53.4% of initially unmatched claims resolved 10 valid extra · 0 matcher miss · 3 false positive · 0 repeated · 89 out of scope
33/33 Complete
1.3 minper attempted review
$10.75 provider reported Parallel local Docker run
09

Muse Spark 1.3

meta/muse-spark-1.3 · Meta

Complete local run without ZDR; lower recall and slower reviews than Muse Spark 1.2 despite similarly low cost.
61.7% AI-assessed recall · baseline scorer recall 58.3% 8.3 unmatched findings/review · 264 initially pending 21/33 gold matched after matcher review AI-assessed provisional precision 23.4% · 0 unresolved 100% of initially unmatched claims resolved 43 valid extra · 1 matcher miss · 133 false positive · 2 repeated · 85 out of scope
33/33 Complete
2.8 minper attempted review
$3.71 provider reported Local Docker run
10

GPT-6 Sol

openai/gpt-6-sol · Azure

Sep 23 Azure US ZDR run. 58 claims remain unresolved after conservative AI review, so precision covers only resolved claims.
60% AI-assessed recall · baseline scorer recall 60% 2.8 unmatched findings/review · 91 initially pending 21/33 gold matched after matcher review AI-assessed provisional precision 60.6% · 58 unresolved 36.3% of initially unmatched claims resolved 14 valid extra · 0 matcher miss · 0 false positive · 0 repeated · 19 out of scope
33/33 Complete
57 sper attempted review
$2.26 provider reported Parallel local Docker run
11

Grok 4.6

x-ai/grok-4.6 · xAI

Balanced recall and assessed precision with full completion, though reviews were relatively slow.
60% AI-assessed recall · baseline scorer recall 51.7% 5.0 unmatched findings/review · 159 initially pending 21/33 gold matched after matcher review AI-assessed provisional precision 43.3% · 0 unresolved 100% of initially unmatched claims resolved 53 valid extra · 3 matcher miss · 75 false positive · 2 repeated · 26 out of scope
33/33 Complete
6.6 minper attempted review
$10.86 provider reported Workflow run ↗
12

GLM 5.3 Flash

z-ai/glm-5.3-flash · Fireworks

Complete after one targeted repair, with strong recall per dollar and moderate assessed precision.
51.7% AI-assessed recall · baseline scorer recall 41.7% 4.9 unmatched findings/review · 159 initially pending 18/33 gold matched after matcher review AI-assessed provisional precision 34.9% · 0 unresolved 100% of initially unmatched claims resolved 44 valid extra · 3 matcher miss · 82 false positive · 1 repeated · 29 out of scope
33/33 Complete after repair Initially 32/33 · repaired 1/1
5.7 minper attempted review
$1.08 all-in provider reported $0.0394 targeted repair Preserved run + targeted repair
13

Gemini 3.8 Flash

google/gemini-3.8-flash · Google

Complete after strict targeted repair using bounded parallel local Docker shards and a narrow provider-compatibility patch; earlier partial attempts remain counted in campaign spend only.
48.3% AI-assessed recall · baseline scorer recall 48.3% 4.2 unmatched findings/review · 133 initially pending 16/33 gold matched after matcher review AI-assessed provisional precision 46.7% · 0 unresolved 100% of initially unmatched claims resolved 57 valid extra · 0 matcher miss · 64 false positive · 0 repeated · 12 out of scope
33/33 Complete after repair Initially 32/33 · repaired 1/1
5.4 minper attempted review
$17.87 all-in provider reported $0.50 targeted repair Preserved run + targeted repair
14

Kimi K3

moonshotai/kimi-k3 · DigitalOcean

Completed on DigitalOcean with moderate recall and precision; an earlier Fireworks partial attempt is counted only in campaign spend.
46.7% AI-assessed recall · baseline scorer recall 40% 4.5 unmatched findings/review · 145 initially pending 17/33 gold matched after matcher review AI-assessed provisional precision 37.2% · 0 unresolved 100% of initially unmatched claims resolved 43 valid extra · 2 matcher miss · 73 false positive · 1 repeated · 26 out of scope
33/33 Complete
8.1 minper attempted review
$10.57 provider reported Workflow run ↗
15

GPT-5.6 Luna

openai/gpt-5.6-luna · Azure

Complete after three targeted repairs and extremely inexpensive, but recall and assessed precision remain low.
36.7% AI-assessed recall · baseline scorer recall 26.7% 1.7 unmatched findings/review · 55 initially pending 13/33 gold matched after matcher review AI-assessed provisional precision 44.1% · 0 unresolved 100% of initially unmatched claims resolved 18 valid extra · 3 matcher miss · 28 false positive · 0 repeated · 6 out of scope
33/33 Complete after repair Initially 30/33 · repaired 3/3
1.0 minper attempted review
$0.26 all-in provider reported $0.0202 targeted repair Preserved run + targeted repair
16

GPT-6 Luna

openai/gpt-6-luna · Azure

Sep 23 Azure US ZDR run. Low known-issue recall despite very low cost; 27 claims remain unresolved, narrowing the provisional precision estimate.
33.3% AI-assessed recall · baseline scorer recall 33.3% 1.1 unmatched findings/review · 36 initially pending 11/33 gold matched after matcher review AI-assessed provisional precision 76.7% · 27 unresolved 25% of initially unmatched claims resolved 4 valid extra · 0 matcher miss · 0 false positive · 0 repeated · 5 out of scope
33/33 Complete
1.7 minper attempted review
$0.16 provider reported Parallel local Docker run
17

GPT-5.6 Terra

openai/gpt-5.6-terra · Azure

Fast reviews and moderate assessed precision, but it found fewer than one-third of known issues.
31.7% AI-assessed recall · baseline scorer recall 28.3% 1.4 unmatched findings/review · 44 initially pending 11/33 gold matched after matcher review AI-assessed provisional precision 59.3% · 0 unresolved 100% of initially unmatched claims resolved 19 valid extra · 1 matcher miss · 20 false positive · 0 repeated · 4 out of scope
33/33 Complete
53 sper attempted review
$2.32 provider reported Workflow run ↗
18

Gemini 3.7 Flash

google/gemini-3.7-flash · Google

Complete after one targeted repair, but it has the lowest known-issue recall and low assessed precision in this cohort.
13.3% AI-assessed recall · baseline scorer recall 10% 1.3 unmatched findings/review · 42 initially pending 5/33 gold matched after matcher review AI-assessed provisional precision 44.4% · 0 unresolved 100% of initially unmatched claims resolved 15 valid extra · 1 matcher miss · 22 false positive · 0 repeated · 4 out of scope
33/33 Complete after repair Initially 32/33 · repaired 1/1
1.8 minper attempted review
$4.09 all-in provider reported $0.16 targeted repair Preserved run + targeted repair

What the comparison says

Claude Opus 5 leads the current board.
Muse Spark 1.2 follows.

Claude Opus 5 ranks first with 80% known-issue recall, 25.4% AI-assessed provisional precision, 33/33 completed reviews, and $41.42 observed run cost. Muse Spark 1.2 follows with 78.3% recall and $3.71 observed run cost. This is the current quality-first ordering; every row retains its completion, provider, speed, cost, and limitation context.

Experimental

This is a small, harness-specific comparison for choosing models in the OpenRouter Review Bot. It is useful directional evidence, not a certification of general code-review ability.

Initial subjects

Two products. Same frozen work.

OR

OpenRouter Review Bot

A multi-model OpenRouter harness with read-only repository tools and structured finding output.

View repository ↗
GX

Grok Code Review Bot

A Grok CLI review product running with a strict Linux sandbox and a deliberately limited read-only tool set.

View repository ↗

How to read the board

Quality leads. Known-issue recall comes first; completion, speed, and cost follow.

Precision is provisional. Valid extras and matcher misses count as correct; false positives, repeated reports, and out-of-scope claims count against precision. Unresolved findings stay outside the precision calculation.

Conditions matter. These rows compare one review product and one small held-out cohort over three attempts. Product revisions and provider reliability can affect results; the rolling run context above records those differences.