- Context
- 1.1Mtokens
- Released
- 1mo ago
Cost/1M tokens
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
Not evaluated in DeepSWE 1.1
Arena Text
Best Arena rating per model · Higher is better
Arena #27
- 1526
Gemini 4 CarbonSimilar price - 1487
Kimi K3Similar price - 1483
Gemini 3.6 FlashNearby rank - 1483
GLM 5.3Nearby rank - 1482
GPT-6.1 Sol - 1482
Claude Opus 4.8Nearby rank - 1481
GPT-5.5Nearby rank - 1472
Claude Sonnet 5.5Similar price - 1457
GPT-6 SolPredecessor - 1354
Command ASimilar price
+25 vs GPT-6 Sol (predecessor) · Ahead of 1 of 4 similarly priced models
Arena WebDev
Best Arena rating per model · Higher is better
Arena #6
- 1773
Claude Sonnet 5.5Similar price - 1768
Gemini 4 CarbonSimilar price - 1764
GPT-6 AstraNearby rank - 1743
GPT-6.1 Sol - 1742
Claude Fable 5.1Nearby rank - 1723
Muse Wonder FullNearby rank - 1676
Qwen3.8 Max (0902)Similar price - 1673
GPT-6 SolPredecessor - 1658
Kimi K3Similar price
+70 vs GPT-6 Sol (predecessor) · Ahead of 2 of 4 similarly priced models
Arena Vision
Best Arena rating per model · Higher is better
Arena #12
- 1294
Muse Spark 1.2Nearby rank - 1293
Gemini 3.8 FlashNearby rank - 1293
GPT-6.1 Sol - 1291
Muse Spark 1.3Nearby rank - 1289
Gemini 3 Pro PreviewNearby rank - 1280
Grok 4.5Similar price - 1280
Gemini 3.1 Pro PreviewSimilar price - 1263
Claude Sonnet 5.5Similar price - 1258
GPT-6 SolPredecessor - 1199
Mistral Medium 3.5Similar price
+34 vs GPT-6 Sol (predecessor) · Ahead of 2 of 4 similarly priced models
Arena Search
Best Arena rating per model · Higher is better
Arena #2
- 1255
GPT-5.6 SolNearby rank - 1248
GPT-6.1 Sol - 1245
GPT-6 AstraNearby rank - 1229
GPT-6 SolPredecessor - 1213
Claude Opus 5.5Similar price - 1205
Grok 4.5Similar price - 1142
Sonar ProSimilar price - 1138
Sonar Reasoning ProSimilar price
+19 vs GPT-6 Sol (predecessor) · Ahead of 4 of 4 similarly priced models
Arena Document
Best Arena rating per model · Higher is better
Not ranked on Arena Document yet
Agent Arena
Best IPS score per model · Higher is better
Arena #7
- +0.170
Gemini 4 CarbonSimilar price - +0.160
Claude Sonnet 5.5Similar price - +0.148
Claude Fable 5.1Nearby rank - +0.145
GPT-6 AstraNearby rank - +0.125
GPT-6.1 Sol - +0.120
GPT-6.1 TerraNearby rank - +0.107
Gemini 4 ArgonNearby rank - +0.069
GPT-6 SolPredecessor - +0.026
Grok 4.7Similar price - +0.006
Kimi K3Similar price
+0.055 vs GPT-6 Sol (predecessor) · Ahead of 2 of 4 similarly priced models
ARC-AGI-2
Best score per model · Higher is better
#2 of 85
- 95.0%
GPT-6 AstraNearby rank - 94.2%
Gemini 4 CarbonSimilar price - 94.2%
GPT-6.1 Sol - 93.8%
Claude Fable 5.2Nearby rank - 93.3%
Claude Opus 5.5Nearby rank - 89.6%
GPT-6 SolPredecessor - 83.9%
Claude Sonnet 5.5Similar price - 67.1%
Grok 4.6Similar price - 60.4%
Kimi K3Similar price
+4.6 pts vs GPT-6 Sol (predecessor) · Ahead of 3 of 4 similarly priced models
FrontierMath-Tier-4-v2-Private
Best score per model · Higher is better
#1 of 72
- 100.0%
Claude Fable 5.2Nearby rank - 100.0%
GPT-6.1 Sol - 97.6%
GPT-6 AstraNearby rank - 95.1%
Gemini 4 CarbonSimilar price - 90.0%
GPT-6 SolPredecessor - 80.5%
Claude Sonnet 5.5Similar price - 39.0%
Kimi K3Similar price - 34.1%
Qwen3.8 Max (0902)Similar price
+10.0 pts vs GPT-6 Sol (predecessor) · Ahead of 4 of 4 similarly priced models
FrontierMath-Tiers-1-3-v2-Private
Best score per model · Higher is better
#2 of 87
- 95.1%
Claude Fable 5.2Nearby rank - 93.7%
GPT-6 AstraNearby rank - 93.7%
GPT-6.1 Sol - 91.2%
Claude Opus 5.5Nearby rank - 90.9%
GPT-6.1 TerraNearby rank - 90.5%
Gemini 4 CarbonSimilar price - 89.8%
GPT-6 SolPredecessor - 88.8%
Claude Sonnet 5.5Similar price - 72.2%
Kimi K3Similar price - 66.0%
Grok 4.6Similar price
+3.9 pts vs GPT-6 Sol (predecessor) · Ahead of 4 of 4 similarly priced models
GPQA diamond
Best score per model · Higher is better
#5 of 161
- 96.5%
Gemini 4 CarbonSimilar price - 95.6%
Claude Sonnet 5.5Similar price - 95.4%
Gemini 3.8 FlashNearby rank - 95.4%
GPT-6.1 Sol - 94.8%
Gemini 3.7 FlashNearby rank - 94.6%
GPT-5.4 ProNearby rank - 94.3%
GPT-6 SolPredecessor - 94.0%
Grok 4.6Similar price - 93.1%
Kimi K3Similar price
+1.1 pts vs GPT-6 Sol (predecessor) · Ahead of 2 of 4 similarly priced models
OTIS Mock AIME 2024-2025
Best score per model · Higher is better
#1 of 158
- 100.0%
Claude Sonnet 5.5Similar price - 100.0%
Gemini 4 ArgonSimilar price - 100.0%
GPT-6 AstraNearby rank - 100.0%
GPT-6 SolPredecessor - 100.0%
GPT-6.1 Sol - 100.0%
GPT-6.1 TerraNearby rank - 100.0%
Qwen3.8 Max (0902)Similar price - 97.2%
Kimi K3Similar price
Level with GPT-6 Sol (predecessor) · Ahead of 1 of 4 similarly priced models
SimpleQA Verified
Best score per model · Higher is better
#5 of 87
- 77.7%
Gemini 4 CarbonSimilar price - 76.0%
Gemini 4 ArgonNearby rank - 75.6%
GPT-6 AstraNearby rank - 73.9%
GPT-6.1 Sol - 73.5%
Gemini 3.1 Pro PreviewNearby rank - 72.2%
Claude Opus 5.5Nearby rank - 60.7%
GPT-6 SolPredecessor - 56.0%
Grok 4.7Similar price - 50.6%
Kimi K3Similar price - 46.5%
Claude Sonnet 5.5Similar price
+13.2 pts vs GPT-6 Sol (predecessor) · Ahead of 3 of 4 similarly priced models
Not evaluated in DeepSWE 1.1 as of 2026-09-22. Results are only shown for exact model versions in the official dataset. Official leaderboard.
Reasoning effort
ARC-AGI-2
Score by reasoning effort · Higher is better
All published scores
- Arena TextArena #271482
+25 vs GPT-6 Sol (predecessor)
- max148295% CI 1476 to 1488 · 12,952 votes
- Arena WebDevArena #61743
+70 vs GPT-6 Sol (predecessor)
- max174395% CI 1733 to 1753 · 5,539 votes
- Arena VisionArena #121293
+34 vs GPT-6 Sol (predecessor)
- max129395% CI 1282 to 1303 · 3,987 votes
- Arena SearchArena #21248
+19 vs GPT-6 Sol (predecessor)
- max124895% CI 1240 to 1256 · 26,556 votes
- Agent ArenaArena #7+0.125
+0.055 vs GPT-6 Sol (predecessor)
- max+0.12595% CI +0.113 to +0.136 · 3,093,307 observations
- ARC-AGI-2#2 of 8594.2%
+4.6 pts vs GPT-6 Sol (predecessor)
- low76.7%
- medium86.7%
- high91.7%
- xhigh91.7%
- max94.2%
- FrontierMath-Tier-4-v2-Private#1 of 72100.0%
+10.0 pts vs GPT-6 Sol (predecessor)
- max100.0%
Source:Epoch AI evaluation - FrontierMath-Tiers-1-3-v2-Private#2 of 8793.7%
+3.9 pts vs GPT-6 Sol (predecessor)
- max93.7%
Source:Epoch AI evaluation - GPQA diamond#5 of 16195.4%
+1.1 pts vs GPT-6 Sol (predecessor)
- max95.4%
Source:Epoch AI evaluation - OTIS Mock AIME 2024-2025#1 of 158100.0%
Level with GPT-6 Sol (predecessor)
- max100.0%
Source:Epoch AI evaluation - SimpleQA Verified#5 of 8773.9%
+13.2 pts vs GPT-6 Sol (predecessor)
- max73.9%
Source:Epoch AI evaluation
Benchmark compilation: Epoch AI, Capabilities & benchmarking (CC BY attribution). Scores are converted to percentages and model names matched to the Vertical Tabs catalog. Individual evaluations are credited to their original sources. External results may carry separate licences and evaluation configurations; do not interpret scores from different harnesses as directly comparable.
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.