- Context
- 1Mtokens
- Released
- 3mo ago
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
#16 of 28
- 74.1%
- 73.8%
- 73.6%
- 72.7%
- 69.9%
- 69.6%
- 69.0%
- 68.5%
- 67.5%
- 67.2%
- 57.5%qwen3-8-max
Arena Text
Best Arena rating per model · Higher is better
Arena #23
- 1493
Muse Spark 1.2Similar release date - 1486Gemini 3 Pro PreviewNearby rank
- 1485GPT-5.6 SolNearby rank
- 1484
Qwen3.8 Max - 1483Kimi K3.1 ProNearby rank
- 1483Gemini 3.6 FlashNearby rank
- 1456Grok 4.6Similar release date
- 1423Qwen3 MaxEarlier version
- 1405
Inkling SmallSimilar release date - 1387
Solar Pro 4Similar release date
+61 vs Qwen3 Max (earlier version) · Ahead of 3 of 4 models released around the same time
Arena WebDev
Best Arena rating per model · Higher is better
Arena #16
- 1676
Qwen3.8 Max (0902)Nearby rank - 1673GPT-6 SolNearby rank
- 1670Qwen3.8 Max
- 1664
DeepSeek V4.1 Flash 1017Nearby rank - 1658Kimi K3Nearby rank
- 1618Grok 4.6Similar release date
- 1534Muse Spark 1.2Similar release date
- 1515Qwen3.7 MaxPredecessor
- 1408Inkling SmallSimilar release date
- 1369Solar Pro 4Similar release date
+155 vs Qwen3.7 Max (predecessor) · Ahead of 4 of 4 models released around the same time
Arena Vision
Best Arena rating per model · Higher is better
Arena #3
- 1309Claude Fable 5Nearby rank
- 1300Claude Opus 4.7Nearby rank
- 1299Qwen3.8 Max
- 1299Claude Opus 4.6Nearby rank
- 1295Claude Opus 5.5Nearby rank
- 1294Gemini 3.7 FlashSimilar release date
- 1294Muse Spark 1.2Similar release date
- 1263Grok 4.6Similar release date
- 1262Qwen3.7 PlusEarlier version
- 1208Inkling SmallSimilar release date
+37 vs Qwen3.7 Plus (earlier version) · Ahead of 2 of 4 models released around the same time
Arena Search
Best Arena rating per model · Higher is better
Arena #6
- 1255GPT-5.6 SolSimilar release date
- 1229GPT-6 SolNearby rank
- 1229Claude Fable 5Nearby rank
- 1227Qwen3.8 Max
- 1227Claude Fable 5.1Nearby rank
- 1213Claude Opus 5Similar release date
- 1205Grok 4.5Similar release date
- 1192
ERNIE 5.1Similar release date
Ahead of 2 of 4 models released around the same time
Arena Document
Best Arena rating per model · Higher is better
Not ranked on Arena Document yet
Agent Arena
Best IPS score per model · Higher is better
Arena #32
- +0.005
Hy4 previewNearby rank - +0.002GLM 5.2Nearby rank
- +0.001Qwen3.8 Max
- -0.000
MiMo-V2.6-FlashNearby rank - -0.002Grok 4.6Similar release date
- -0.072Qwen3.7 MaxPredecessor
- -0.082Muse Spark 1.2Similar release date
- -0.181Inkling SmallSimilar release date
- -0.226Solar Pro 4Similar release date
+0.073 vs Qwen3.7 Max (predecessor) · Ahead of 3 of 4 models released around the same time
FrontierMath-Tier-4-v2-Private
Best score per model · Higher is better
#23 of 72
- 73.2%Claude Opus 5Similar release date
- 56.1%GPT-6 LunaNearby rank
- 49.0%GPT-5.4Nearby rank
- 46.3%Qwen3.8 Max
- 46.3%Claude Haiku 5.5Nearby rank
- 46.3%DeepSeek V4.1 Flash 1017Nearby rank
- 34.1%Qwen3.7 MaxPredecessor
- 31.7%Grok 4.6Similar release date
- 24.4%DeepSeek V4 Flash 0731Similar release date
- 17.1%Inkling SmallSimilar release date
+12.2 pts vs Qwen3.7 Max (predecessor) · Ahead of 3 of 4 models released around the same time
FrontierMath-Tiers-1-3-v2-Private
Best score per model · Higher is better
#23 of 87
- 85.6%Claude Opus 5Similar release date
- 78.6%GPT-5.4Nearby rank
- 75.1%Claude Haiku 5.5Nearby rank
- 74.7%Qwen3.8 Max
- 74.4%Muse Spark 1.3Nearby rank
- 74.0%GPT-5.2 ProNearby rank
- 66.0%Grok 4.6Similar release date
- 64.6%Qwen3.7 MaxPredecessor
- 57.5%DeepSeek V4 Flash 0731Similar release date
- 46.3%Inkling SmallSimilar release date
+10.2 pts vs Qwen3.7 Max (predecessor) · Ahead of 3 of 4 models released around the same time
GPQA diamond
Best score per model · Higher is better
#24 of 161
- 94.8%Gemini 3.7 FlashSimilar release date
- 94.0%Grok 4.6Similar release date
- 92.8%Gemini 3.5 FlashNearby rank
- 92.7%Grok 4.7Nearby rank
- 92.7%Qwen3.8 Max
- 92.6%Gemini 3 Pro PreviewNearby rank
- 92.3%Qwen3.8 Max (0902)Nearby rank
- 91.0%DeepSeek V4 Flash 0731Similar release date
- 90.9%Qwen3.7 MaxPredecessor
- 88.5%Inkling SmallSimilar release date
+1.8 pts vs Qwen3.7 Max (predecessor) · Ahead of 2 of 4 models released around the same time
OTIS Mock AIME 2024-2025
Best score per model · Higher is better
#15 of 158
- 99.7%Claude Fable 5.2Nearby rank
- 99.7%GPT-5.6 TerraNearby rank
- 99.4%Qwen3.8 Max
- 99.4%DeepSeek V4.1 Flash 1017Nearby rank
- 99.2%Muse Spark 1.3Nearby rank
- 99.2%Grok 4.6Similar release date
- 98.9%Claude Opus 5Similar release date
- 95.6%Qwen3.7 MaxPredecessor
- 94.4%DeepSeek V4 Flash 0731Similar release date
- 90.0%Inkling SmallSimilar release date
+3.9 pts vs Qwen3.7 Max (predecessor) · Ahead of 4 of 4 models released around the same time
SimpleQA Verified
Best score per model · Higher is better
#41 of 87
- 60.3%Muse Spark 1.2Similar release date
- 55.8%Qwen3.7 MaxPredecessor
- 49.3%Grok 4.6Similar release date
- 46.5%Claude Sonnet 5.5Nearby rank
- 46.3%GPT-5.4 ProNearby rank
- 45.8%Qwen3.8 Max
- 45.7%Claude Opus 4.5Nearby rank
- 45.1%GPT-5.4Nearby rank
- 33.6%DeepSeek V4 Flash 0731Similar release date
- 19.1%Inkling SmallSimilar release date
-10.0 pts vs Qwen3.7 Max (predecessor) · Ahead of 2 of 4 models released around the same time
DeepSWE 1.1
Primary coding-agent benchmark · 113 tasks · Data generated 2026-09-22
- pass@4
- 83.2%
- Average cost / attempt
- $3.73
- Benchmark runs
- 4
pass@1 is the success rate across scored attempts; pass@4 is the share of tasks solved in at least one of four attempts. Costs are reported by DeepSWE for each configuration. These results are not directly comparable to evaluations using different harnesses. Official source JSON.
All published scores
- Arena TextArena #231484
+61 vs Qwen3 Max (earlier version)
- default148495% CI 1479 to 1489 · 36,324 votes
- Arena WebDevArena #161670
+155 vs Qwen3.7 Max (predecessor)
- default167095% CI 1658 to 1682 · 3,371 votes
- Arena VisionArena #31299
+37 vs Qwen3.7 Plus (earlier version)
- default129995% CI 1292 to 1306 · 9,842 votes
- Arena SearchArena #61227
- default122795% CI 1214 to 1241 · 1,998 votes
- Agent ArenaArena #32+0.001
+0.073 vs Qwen3.7 Max (predecessor)
- default+0.00195% CI -0.006 to +0.008 · 8,609,159 observations
- FrontierMath-Tier-4-v2-Private#23 of 7246.3%
+12.2 pts vs Qwen3.7 Max (predecessor)
- xhigh46.3%
Source:Epoch AI evaluation - FrontierMath-Tiers-1-3-v2-Private#23 of 8774.7%
+10.2 pts vs Qwen3.7 Max (predecessor)
- xhigh74.7%
Source:Epoch AI evaluation - GPQA diamond#24 of 16192.7%
+1.8 pts vs Qwen3.7 Max (predecessor)
- xhigh92.7%
Source:Epoch AI evaluation - OTIS Mock AIME 2024-2025#15 of 15899.4%
+3.9 pts vs Qwen3.7 Max (predecessor)
- xhigh99.4%
Source:Epoch AI evaluation - SimpleQA Verified#41 of 8745.8%
-10.0 pts vs Qwen3.7 Max (predecessor)
- xhigh45.8%
Source:Epoch AI evaluation
Benchmark compilation: Epoch AI, Capabilities & benchmarking (CC BY attribution). Scores are converted to percentages and model names matched to the Vertical Tabs catalog. Individual evaluations are credited to their original sources. External results may carry separate licences and evaluation configurations; do not interpret scores from different harnesses as directly comparable.
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.