- Context
- 1Mtokens
- Released
- 7mo ago
Cost/1M tokens
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
Not evaluated in DeepSWE 1.1
Arena Text
Best Arena rating per model · Higher is better
Arena #8
- 1511
Muse WonderSimilar price - 1503
Claude Opus 4.6Predecessor - 1503
Claude Fable 5.1Nearby rank - 1501
Claude Opus 4.7 - 1497
Gemini 3.8 FlashNearby rank - 1496
Muse Spark 1.3Nearby rank - 1485
GPT-5.6 SolSimilar price - 1481
GPT-5.5Similar price - 1346
GPT-4o (2024-05-13)Similar price
Level with Claude Opus 4.6 (predecessor, overlapping 95% CIs) · Ahead of 3 of 4 similarly priced models
Arena WebDev
Best Arena rating per model · Higher is better
Arena #48
- 1649
Muse WonderSimilar price - 1615
GPT-5.6 SolSimilar price - 1573
Step 5 PreviewNearby rank - 1564
Mistral Large 4Nearby rank - 1557
Claude Opus 4.7 - 1555
Claude Opus 4.8Nearby rank - 1552
Grok 4.5Nearby rank - 1547
Claude Opus 4.6Predecessor - 1512
GPT-5.5Similar price - 1462
GPT-5.4Similar price
Level with Claude Opus 4.6 (predecessor, overlapping 95% CIs) · Ahead of 2 of 4 similarly priced models
Arena Vision
Best Arena rating per model · Higher is better
Arena #2
- 1309
Claude Fable 5Nearby rank - 1300
Claude Opus 4.7 - 1299Qwen3.8 MaxNearby rank
- 1299
Claude Opus 4.6Predecessor - 1288
GPT-5.6 SolSimilar price - 1285
GPT-5.5Similar price - 1282
GPT-5.4Similar price - 1162
GPT-4o (2024-05-13)Similar price
Level with Claude Opus 4.6 (predecessor, overlapping 95% CIs) · Ahead of 3 of 4 similarly priced models
Arena Search
Best Arena rating per model · Higher is better
Arena #13
- 1255
GPT-5.6 SolSimilar price - 1245
GPT-6 AstraSimilar price - 1213
Claude Opus 5Similar price - 1213
Claude Opus 5.5Nearby rank - 1211
GPT-6 LunaNearby rank - 1210
Claude Opus 4.7 - 1205
Grok 4.5Nearby rank - 1201
Gemini 3 Pro PreviewNearby rank - 1142
Sonar ProSimilar price
Ahead of 1 of 4 similarly priced models
Arena Document
Best Arena rating per model · Higher is better
Arena #5
- 1498
Claude Fable 5.1Nearby rank - 1497
Claude Opus 4.6Predecessor - 1494
Claude Opus 4.7 - 1486
GPT-5.6 SolSimilar price - 1486
Claude Opus 5Nearby rank - 1481
GPT-5.5Similar price - 1474
GPT-6 AstraSimilar price - 1471
GPT-5.4Similar price
Level with Claude Opus 4.6 (predecessor, overlapping 95% CIs) · Ahead of 3 of 4 similarly priced models
Agent Arena
Best IPS score per model · Higher is better
Not ranked on Agent Arena yet
ARC-AGI-2
Best score per model · Higher is better
#22 of 85
- 95.0%
GPT-6 AstraSimilar price - 92.5%
GPT-5.6 SolSimilar price - 85.0%
GPT-5.5Similar price - 78.1%
DeepSeek V4.1 Flash 1017Nearby rank - 77.1%
Gemini 3.1 Pro PreviewNearby rank - 75.8%
Claude Opus 4.7 - 74.0%
GPT-5.4Similar price - 72.9%
DeepSeek V4.1 FlashNearby rank - 64.6%
Claude Opus 4.6Predecessor
+11.2 pts vs Claude Opus 4.6 (predecessor) · Ahead of 1 of 4 similarly priced models
FrontierMath-2025-02-28-Private
Best score per model · Higher is better
#4 of 58
- 47.6%
GPT-5.4Similar price - 47.2%
Claude Opus 4.8Similar price - 43.8%
Claude Opus 4.7 - 40.7%
Claude Opus 4.6Predecessor - 40.7%
GPT-5.2Nearby rank - 32.4%
Claude Sonnet 4.6Similar price - 20.7%
Claude Opus 4.5Similar price
+3.1 pts vs Claude Opus 4.6 (predecessor) · Ahead of 2 of 4 similarly priced models
FrontierMath-Tier-4-2025-07-01-Private
Best score per model · Higher is better
#3 of 46
- 31.3%
Claude Opus 4.8Similar price - 27.1%
GPT-5.4Similar price - 22.9%
Claude Opus 4.7 - 22.9%
Claude Opus 4.6Predecessor - 18.8%
GPT-5.2Nearby rank - 8.3%
Claude Sonnet 4.6Similar price - 4.2%
Claude Opus 4.5Similar price
Level with Claude Opus 4.6 (predecessor) · Ahead of 2 of 4 similarly priced models
FrontierMath-Tier-4-v2-Private
Best score per model · Higher is better
#32 of 72
- 97.6%
GPT-6 AstraSimilar price - 82.9%
GPT-5.6 SolSimilar price - 72.5%
GPT-5.5Similar price - 49.0%
GPT-5.4Similar price - 34.1%
Qwen3.7 MaxNearby rank - 34.1%
Qwen3.8 Max (0902)Nearby rank - 31.7%
Claude Opus 4.7 - 31.7%
Grok 4.6Nearby rank - 31.7%
GPT-5.2Nearby rank - 26.8%
Claude Opus 4.6Predecessor
+4.9 pts vs Claude Opus 4.6 (predecessor) · Ahead of 0 of 4 similarly priced models
FrontierMath-Tiers-1-3-v2-Private
Best score per model · Higher is better
#29 of 87
- 93.7%
GPT-6 AstraSimilar price - 89.1%
GPT-5.6 SolSimilar price - 85.3%
GPT-5.5Similar price - 78.6%
GPT-5.4Similar price - 71.6%
Gemini 3.7 FlashNearby rank - 71.2%
DeepSeek V4.1 Flash 1017Nearby rank - 70.2%
Claude Opus 4.7 - 68.8%
GLM 5.3Nearby rank - 68.4%
Gemini 3.8 FlashNearby rank - 66.0%
Claude Opus 4.6Predecessor
+4.2 pts vs Claude Opus 4.6 (predecessor) · Ahead of 0 of 4 similarly priced models
GPQA diamond
Best score per model · Higher is better
#43 of 161
- 93.5%
GPT-5.6 SolSimilar price - 93.3%
GPT-5.4Similar price - 90.7%
GPT-5.5Similar price - 90.5%
Claude Opus 4.6Predecessor - 90.5%
Claude Sonnet 5Nearby rank - 90.5%
GPT-6 LunaNearby rank - 90.2%
Claude Opus 4.7 - 90.2%
GLM 5.3 FlashNearby rank - 89.9%
GLM 5.1Nearby rank - 48.9%
GPT-4o (2024-05-13)Similar price
-0.4 pts vs Claude Opus 4.6 (predecessor) · Ahead of 1 of 4 similarly priced models
Humanity's Last Exam
Best score per model · Higher is better
#9 of 34
- 54.8%
GPT-6 AstraSimilar price - 37.5%
Gemini 3 Pro PreviewNearby rank - 36.2%
GPT-5.4Similar price - 36.2%
Claude Opus 4.7 - 34.4%
Claude Opus 4.6Predecessor - 31.6%
GPT-5 ProNearby rank - 14.2%
Claude Opus 4.5Similar price - 7.5%
Claude Sonnet 4.5Similar price
+1.8 pts vs Claude Opus 4.6 (predecessor) · Ahead of 2 of 4 similarly priced models
OSWorld 2.0
Best score per model · Higher is better
#4 of 9
- 31.4%
Claude Opus 5Similar price - 27.3%
GPT-5.6 SolSimilar price - 18.5%
Claude Opus 4.8Similar price - 13.9%
Claude Opus 4.7 - 13.0%
GPT-5.5Similar price - 9.3%
Claude Sonnet 4.6Earlier version
+4.6 pts vs Claude Sonnet 4.6 (earlier version) · Ahead of 1 of 4 similarly priced models
OTIS Mock AIME 2024-2025
Best score per model · Higher is better
#29 of 158
- 100.0%
GPT-5.6 SolSimilar price - 98.3%
GPT-5.6 LunaNearby rank - 98.1%
Grok 4.7Nearby rank - 97.8%
Claude Opus 4.7 - 97.8%
GPT-5.4Similar price - 97.8%
Grok 4.5Nearby rank - 94.4%
Claude Opus 4.6Predecessor - 84.4%
GPT-5.5Similar price - 6.3%
GPT-4o (2024-05-13)Similar price
+3.4 pts vs Claude Opus 4.6 (predecessor) · Ahead of 2 of 4 similarly priced models
SimpleQA Verified
Best score per model · Higher is better
#28 of 87
- 75.6%
GPT-6 AstraSimilar price - 69.7%
GPT-5.6 SolSimilar price - 63.0%
GPT-5.5Similar price - 52.9%
DeepSeek V4 Pro 0813Nearby rank - 52.0%Qwen3.6 Max PreviewNearby rank
- 51.7%
Claude Opus 4.7 - 50.6%
Kimi K3Nearby rank - 50.1%
GPT-5Nearby rank - 47.0%
Claude Opus 4.6Predecessor - 45.1%
GPT-5.4Similar price
+4.7 pts vs Claude Opus 4.6 (predecessor) · Ahead of 1 of 4 similarly priced models
SWE-bench Verified
Best score per model · Higher is better
#1 of 35
- 83.5%
Claude Opus 4.7 - 79.3%
Gemini 3.5 FlashNearby rank - 78.7%
GLM 5.2Nearby rank - 76.9%
GPT-5.4Similar price - 76.7%
Claude Opus 4.5Similar price - 75.6%
Claude Opus 4.6Predecessor - 75.2%
Claude Sonnet 4.6Similar price - 71.3%
Claude Sonnet 4.5Similar price
+7.9 pts vs Claude Opus 4.6 (predecessor) · Ahead of 4 of 4 similarly priced models
Terminal Bench
Best score per model · Higher is better
#3 of 49
- 84.7%
GPT-5.5Similar price - 81.8%
GPT-5.4Similar price - 80.2%
Claude Opus 4.7 - 80.2%
Gemini 3.1 Pro PreviewNearby rank - 79.8%
Claude Opus 4.6Predecessor - 63.1%
Claude Opus 4.5Similar price - 53.4%
Claude Sonnet 4.6Similar price
+0.4 pts vs Claude Opus 4.6 (predecessor) · Ahead of 2 of 4 similarly priced models
Not evaluated in DeepSWE 1.1 as of 2026-09-22. Results are only shown for exact model versions in the official dataset. Official leaderboard.
Reasoning effort
ARC-AGI-2
Score by reasoning effort · Higher is better
GPQA diamond
Score by reasoning effort · Higher is better
OTIS Mock AIME 2024-2025
Score by reasoning effort · Higher is better
All published scores
- Arena TextArena #81501
Level with Claude Opus 4.6 (predecessor, overlapping 95% CIs)
- high150195% CI 1497 to 1505 · 64,308 votes
- default149495% CI 1490 to 1498 · 65,369 votes
- Arena WebDevArena #481557
Level with Claude Opus 4.6 (predecessor, overlapping 95% CIs)
- high155795% CI 1551 to 1563 · 17,604 votes
- default155595% CI 1550 to 1561 · 17,929 votes
- Arena VisionArena #21300
Level with Claude Opus 4.6 (predecessor, overlapping 95% CIs)
- high129895% CI 1292 to 1305 · 22,609 votes
- default130095% CI 1294 to 1307 · 23,777 votes
- Arena SearchArena #131210
- default121095% CI 1205 to 1215 · 121,890 votes
- Arena DocumentArena #51494
Level with Claude Opus 4.6 (predecessor, overlapping 95% CIs)
- high149495% CI 1488 to 1501 · 23,327 votes
- default149495% CI 1488 to 1501 · 23,628 votes
- ARC-AGI-2#22 of 8575.8%
+11.2 pts vs Claude Opus 4.6 (predecessor)
- low62.1%
- medium67.5%
- high68.3%
- max75.8%
- FrontierMath-2025-02-28-Private#4 of 5843.8%
+3.1 pts vs Claude Opus 4.6 (predecessor)
- xhigh43.8%
Source:Epoch AI evaluation - FrontierMath-Tier-4-2025-07-01-Private#3 of 4622.9%
Level with Claude Opus 4.6 (predecessor)
- xhigh22.9%
Source:Epoch AI evaluation - FrontierMath-Tier-4-v2-Private#32 of 7231.7%
+4.9 pts vs Claude Opus 4.6 (predecessor)
- max31.7%
Source:Epoch AI evaluation - FrontierMath-Tiers-1-3-v2-Private#29 of 8770.2%
+4.2 pts vs Claude Opus 4.6 (predecessor)
- max70.2%
Source:Epoch AI evaluation - GPQA diamond#43 of 16190.2%
-0.4 pts vs Claude Opus 4.6 (predecessor)
- xhigh90.2%
- max86.4%
Source:Epoch AI evaluation - Humanity's Last Exam#9 of 3436.2%
+1.8 pts vs Claude Opus 4.6 (predecessor)
- unknown36.2%
- OSWorld 2.0#4 of 913.9%
+4.6 pts vs Claude Sonnet 4.6 (earlier version)
- max13.9%
- OTIS Mock AIME 2024-2025#29 of 15897.8%
+3.4 pts vs Claude Opus 4.6 (predecessor)
- xhigh97.8%
- max86.7%
Source:Epoch AI evaluation - SimpleQA Verified#28 of 8751.7%
+4.7 pts vs Claude Opus 4.6 (predecessor)
- xhigh51.7%
Source:Epoch AI evaluation - SWE-bench Verified#1 of 3583.5%
+7.9 pts vs Claude Opus 4.6 (predecessor)
- max83.5%
Source:Epoch AI evaluation - Terminal Bench#3 of 4980.2%
+0.4 pts vs Claude Opus 4.6 (predecessor)
- unknown80.2%WOZCODE
Benchmark compilation: Epoch AI, Capabilities & benchmarking (CC BY attribution). Scores are converted to percentages and model names matched to the Vertical Tabs catalog. Individual evaluations are credited to their original sources. External results may carry separate licences and evaluation configurations; do not interpret scores from different harnesses as directly comparable.
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.