- Context
- 1Mtokens
- Released
- 3mo ago
Cost/1M tokens
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
#3 of 28
- 74.1%

- 73.8%

- 73.6%
Claude Opus 5 - 72.7%

- 69.9%

- 69.6%

- 69.0%

- 68.5%

- 67.5%

- 67.2%

Arena Text
Best Arena rating per model · Higher is better
Arena #15
- 1511
Muse WonderSimilar price - 1493
Muse Spark 1.2Nearby rank - 1491
Muse Spark 1.1Nearby rank - 1490
Claude Opus 5 - 1489
Gemini 3.7 FlashNearby rank - 1487
Kimi K3Nearby rank - 1485
GPT-5.6 SolSimilar price - 1482
Claude Opus 4.8Predecessor - 1481
GPT-5.5Similar price - 1346
GPT-4o (2024-05-13)Similar price
+8 vs Claude Opus 4.8 (predecessor) · Ahead of 2 of 4 similarly priced models
Arena WebDev
Best Arena rating per model · Higher is better
Arena #10
- 1742
Claude Fable 5.1Nearby rank - 1723
Muse Wonder FullNearby rank - 1691
Claude Opus 5 - 1688
Kimi K3.1 ProNearby rank - 1682
Gemini 4 ArgonNearby rank - 1649
Muse WonderSimilar price - 1615
GPT-5.6 SolSimilar price - 1555
Claude Opus 4.8Predecessor - 1512
GPT-5.5Similar price - 1462
GPT-5.4Similar price
+136 vs Claude Opus 4.8 (predecessor) · Ahead of 4 of 4 similarly priced models
Arena Vision
Best Arena rating per model · Higher is better
Arena #19
- 1288
GPT-5.6 SolSimilar price - 1286
Claude Opus 4.8Predecessor - 1285
Claude Opus 5 - 1285
GPT-5.5Similar price - 1283
GPT-6 AstraNearby rank - 1282
GPT-5.4Similar price - 1162
GPT-4o (2024-05-13)Similar price
Level with Claude Opus 4.8 (predecessor, overlapping 95% CIs) · Ahead of 1 of 4 similarly priced models
Arena Search
Best Arena rating per model · Higher is better
Arena #10
- 1255
GPT-5.6 SolSimilar price - 1245
GPT-6 AstraSimilar price - 1227Qwen3.8 MaxNearby rank
- 1227
Claude Fable 5.1Nearby rank - 1213
Claude Opus 5 - 1213
Claude Opus 5.5Nearby rank - 1211
GPT-6 LunaNearby rank - 1210
Claude Opus 4.7Similar price - 1194
Claude Opus 4.8Predecessor - 1142
Sonar ProSimilar price
+19 vs Claude Opus 4.8 (predecessor) · Ahead of 1 of 4 similarly priced models
Arena Document
Best Arena rating per model · Higher is better
Arena #8
- 1494
Claude Opus 4.7Nearby rank - 1486
GPT-5.6 SolSimilar price - 1486
Claude Opus 5 - 1485
Claude Opus 4.8Predecessor - 1481
GPT-5.5Similar price - 1474
GPT-6 AstraSimilar price - 1471
GPT-5.4Similar price
Level with Claude Opus 4.8 (predecessor, overlapping 95% CIs) · Ahead of 1 of 4 similarly priced models
Agent Arena
Best IPS score per model · Higher is better
Arena #12
- +0.195
Claude Opus 5.5Similar price - +0.145
GPT-6 AstraSimilar price - +0.090
Muse Wonder FullNearby rank - +0.073
Claude Fable 5Nearby rank - +0.072
Claude Opus 5 - +0.069
GPT-6 SolNearby rank - +0.053
Hy4Nearby rank - +0.046
GPT-5.6 SolSimilar price - +0.018
Muse WonderSimilar price
Ahead of 2 of 4 similarly priced models
ARC-AGI-2
Best score per model · Higher is better
#12 of 85
- 95.0%
GPT-6 AstraSimilar price - 92.5%
GPT-5.6 SolSimilar price - 89.2%
Claude Fable 5Nearby rank - 89.2%
Gemini 3.8 FlashNearby rank - 88.3%
Claude Opus 5 - 85.0%
GPT-5.5Similar price - 84.6%
Gemini 3.7 FlashNearby rank - 74.0%
GPT-5.4Similar price - 72.1%
Claude Opus 4.8Predecessor
+16.3 pts vs Claude Opus 4.8 (predecessor) · Ahead of 2 of 4 similarly priced models
FrontierMath-Tier-4-v2-Private
Best score per model · Higher is better
#15 of 72
- 97.6%
GPT-6 AstraSimilar price - 82.9%
GPT-5.6 SolSimilar price - 78.0%
Gemini 4 ArgonNearby rank - 78.0%
GPT-5.5 ProNearby rank - 73.2%
Claude Opus 5 - 72.5%
GPT-5.5Similar price - 70.7%
GPT-5.6 TerraNearby rank - 56.1%
Claude Opus 4.8Predecessor - 49.0%
GPT-5.4Similar price
+17.1 pts vs Claude Opus 4.8 (predecessor) · Ahead of 2 of 4 similarly priced models
FrontierMath-Tiers-1-3-v2-Private
Best score per model · Higher is better
#15 of 87
- 93.7%
GPT-6 AstraSimilar price - 89.1%
GPT-5.6 SolSimilar price - 86.0%
Gemini 4 ArgonNearby rank - 86.0%
GPT-5.6 TerraNearby rank - 85.6%
Claude Opus 5 - 85.3%
GPT-5.5Similar price - 82.5%
GPT-5.4 ProNearby rank - 80.0%
Claude Opus 4.8Predecessor - 78.6%
GPT-5.4Similar price
+5.6 pts vs Claude Opus 4.8 (predecessor) · Ahead of 2 of 4 similarly priced models
GPQA diamond
Best score per model · Higher is better
#14 of 161
- 94.0%
Grok 4.6Nearby rank - 93.9%
Claude Fable 5.2Nearby rank - 93.9%
Claude Opus 5 - 93.5%
GPT-5.6 SolSimilar price - 93.4%
Grok 4.5Nearby rank - 93.3%
GPT-5.4Similar price - 91.0%
Claude Opus 4.8Predecessor - 90.7%
GPT-5.5Similar price - 48.9%
GPT-4o (2024-05-13)Similar price
+2.8 pts vs Claude Opus 4.8 (predecessor) · Ahead of 4 of 4 similarly priced models
OSWorld 2.0
Best score per model · Higher is better
#1 of 9
- 31.4%
Claude Opus 5 - 27.3%
GPT-5.6 SolSimilar price - 18.5%
Claude Opus 4.8Predecessor - 13.9%
Claude Opus 4.7Similar price - 13.0%
GPT-5.5Similar price - 9.3%
Claude Sonnet 4.6Similar price
+12.9 pts vs Claude Opus 4.8 (predecessor) · Ahead of 4 of 4 similarly priced models
OTIS Mock AIME 2024-2025
Best score per model · Higher is better
#19 of 158
- 100.0%
GPT-5.6 SolSimilar price - 99.2%
Grok 4.6Nearby rank - 98.9%
Claude Haiku 5.5Nearby rank - 98.9%
Claude Opus 5 - 98.9%
Gemini 3.8 FlashNearby rank - 98.9%
GPT-6 LunaNearby rank - 98.3%
Claude Opus 4.8Predecessor - 97.8%
GPT-5.4Similar price - 84.4%
GPT-5.5Similar price - 6.3%
GPT-4o (2024-05-13)Similar price
+0.6 pts vs Claude Opus 4.8 (predecessor) · Ahead of 3 of 4 similarly priced models
SimpleQA Verified
Best score per model · Higher is better
#19 of 87
- 75.6%
GPT-6 AstraSimilar price - 69.7%
GPT-5.6 SolSimilar price - 63.0%
GPT-5.5Similar price - 60.7%
GPT-6 SolNearby rank - 60.3%
Muse Spark 1.2Nearby rank - 59.9%
Claude Opus 5 - 57.8%
Muse Spark 1.1Nearby rank - 57.3%
GPT-6.1 TerraNearby rank - 53.0%
Claude Opus 4.8Predecessor - 45.1%
GPT-5.4Similar price
+6.9 pts vs Claude Opus 4.8 (predecessor) · Ahead of 1 of 4 similarly priced models
DeepSWE 1.1
Primary coding-agent benchmark · 113 tasks · Data generated 2026-09-22
- pass@4
- 88.5%
- Average cost / attempt
- $11.84
- Benchmark runs
- 4
Score vs cost by reasoning effort
Average cost per attempt (USD) against pass rate
All published configurations
| Harness / reasoning | pass@1 | pass@4 | Cost / attempt |
|---|---|---|---|
| mini-swe-agent · max | 73.6% | 88.5% | $11.84 |
| mini-swe-agent · xhigh | 73.2% | 85.8% | $9.07 |
| mini-swe-agent · high | 72.8% | 87.6% | $6.08 |
| mini-swe-agent · medium | 68.9% | 89.4% | $3.29 |
| mini-swe-agent · low | 58.1% | 85.0% | $1.66 |
pass@1 is the success rate across scored attempts; pass@4 is the share of tasks solved in at least one of four attempts. Costs are reported by DeepSWE for each configuration. These results are not directly comparable to evaluations using different harnesses. Official source JSON.
Reasoning effort
Arena Text
Score by reasoning effort · Higher is better
Arena WebDev
Score by reasoning effort · Higher is better
ARC-AGI-2
Score by reasoning effort · Higher is better
GPQA diamond
Score by reasoning effort · Higher is better
OSWorld 2.0
Score by reasoning effort · Higher is better
OTIS Mock AIME 2024-2025
Score by reasoning effort · Higher is better
All published scores
- Arena TextArena #151490
+8 vs Claude Opus 4.8 (predecessor)
- high149095% CI 1486 to 1493 · 73,697 votes
- max148995% CI 1484 to 1494 · 35,710 votes
- Arena WebDevArena #101691
+136 vs Claude Opus 4.8 (predecessor)
- high165795% CI 1651 to 1662 · 26,940 votes
- max169195% CI 1685 to 1697 · 19,561 votes
- Arena VisionArena #191285
Level with Claude Opus 4.8 (predecessor, overlapping 95% CIs)
- high128595% CI 1279 to 1292 · 19,345 votes
- Arena SearchArena #101213
+19 vs Claude Opus 4.8 (predecessor)
- high121395% CI 1206 to 1220 · 34,000 votes
- Arena DocumentArena #81486
Level with Claude Opus 4.8 (predecessor, overlapping 95% CIs)
- high148695% CI 1480 to 1492 · 16,189 votes
- Agent ArenaArena #12+0.072
- max+0.07295% CI +0.059 to +0.085 · 4,693,134 observations
- ARC-AGI-2#12 of 8588.3%
+16.3 pts vs Claude Opus 4.8 (predecessor)
- high88.3%
- max88.3%
- FrontierMath-Tier-4-v2-Private#15 of 7273.2%
+17.1 pts vs Claude Opus 4.8 (predecessor)
- max73.2%
Source:Epoch AI evaluation - FrontierMath-Tiers-1-3-v2-Private#15 of 8785.6%
+5.6 pts vs Claude Opus 4.8 (predecessor)
- max85.6%
Source:Epoch AI evaluation - GPQA diamond#14 of 16193.9%
+2.8 pts vs Claude Opus 4.8 (predecessor)
- low87.9%
- max93.9%
- default92.9%
Source:Epoch AI evaluation - OSWorld 2.0#1 of 931.4%
+12.9 pts vs Claude Opus 4.8 (predecessor)
- low22.3%
- medium25.3%
- high29.0%
- xhigh30.2%
- max31.4%
- OTIS Mock AIME 2024-2025#19 of 15898.9%
+0.6 pts vs Claude Opus 4.8 (predecessor)
- low93.3%
- max98.9%
- default97.8%
Source:Epoch AI evaluation - SimpleQA Verified#19 of 8759.9%
+6.9 pts vs Claude Opus 4.8 (predecessor)
- max59.9%
Source:Epoch AI evaluation
Benchmark compilation: Epoch AI, Capabilities & benchmarking (CC BY attribution). Scores are converted to percentages and model names matched to the Vertical Tabs catalog. Individual evaluations are credited to their original sources. External results may carry separate licences and evaluation configurations; do not interpret scores from different harnesses as directly comparable.
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.