- Context
- 1Mtokens
- Released
- 5mo ago
Cost/1M tokens
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
#25 of 28
- 74.1%
- 73.8%
- 73.6%
- 72.7%
- 69.9%
- 69.6%
- 69.0%
- 68.5%
- 67.5%
- 67.2%
- 36.1%Gemini 3.5 Flash
Arena Text
Best Arena rating per model · Higher is better
Arena #31
- 1481GPT-5.5Nearby rank
- 1480
MiMo-V2.6-ProNearby rank - 1478Gemini 3.5 Flash
- 1476GPT-5.2 ChatNearby rank
- 1475GPT-5.4Nearby rank
- 1472Claude Sonnet 5.5Similar price
- 1466Grok 4.5Similar price
- 1455GPT-5.1Similar price
- 1427
Mistral Medium 3.5Similar price - 1409Gemini 2.5 FlashEarlier version
+69 vs Gemini 2.5 Flash (earlier version) · Ahead of 3 of 4 similarly priced models
Arena WebDev
Best Arena rating per model · Higher is better
Arena #69
- 1676
Qwen3.8 Max (0902)Similar price - 1641Grok 4.7Similar price
- 1508GLM 5.1Nearby rank
- 1506
Hy3Nearby rank - 1500Gemini 3.5 Flash
- 1493Claude Opus 4.5Nearby rank
- 1484
MiniMax M3Nearby rank - 1417GPT-5Similar price
- 1263Mistral Medium 3.5Similar price
- 1256Gemini 3.1 Flash Lite PreviewEarlier version
+244 vs Gemini 3.1 Flash Lite Preview (earlier version) · Ahead of 2 of 4 similarly priced models
Arena Vision
Best Arena rating per model · Higher is better
Arena #24
- 1283GPT-6 AstraNearby rank
- 1282GPT-5.4Nearby rank
- 1281Gemini 3.5 Flash
- 1280Grok 4.5Similar price
- 1280Gemini 3.1 Pro PreviewNearby rank
- 1263Claude Sonnet 5.5Similar price
- 1251GPT-5.1Similar price
- 1215Gemini 2.5 FlashEarlier version
- 1199Mistral Medium 3.5Similar price
+66 vs Gemini 2.5 Flash (earlier version) · Ahead of 3 of 4 similarly priced models
Arena Search
Best Arena rating per model · Higher is better
Not ranked on Arena Search yet
Arena Document
Best Arena rating per model · Higher is better
Arena #28
- 1480
Muse Spark 1.1Similar price - 1469Claude Sonnet 5Similar price
- 1466Grok 4.5Similar price
- 1460GPT-5.5 InstantNearby rank
- 1459Gemini 3.1 Pro PreviewNearby rank
- 1459Gemini 3.5 Flash
- 1454Gemini 3.6 FlashNearby rank
- 1452Gemma 4 31BNearby rank
- 1416GPT-5.1Similar price
Ahead of 1 of 4 similarly priced models
Agent Arena
Best IPS score per model · Higher is better
Not ranked on Agent Arena yet
ARC-AGI-2
Best score per model · Higher is better
#25 of 85
- 83.9%Claude Sonnet 5.5Similar price
- 72.9%
DeepSeek V4.1 FlashNearby rank - 72.1%Claude Opus 4.8Nearby rank
- 72.1%Gemini 3.5 Flash
- 67.1%Grok 4.6Similar price
- 65.8%GLM 5.3 FlashNearby rank
- 60.4%Kimi K3Similar price
- 17.6%GPT-5.1Similar price
- 1.3%Gemini 3 Flash PreviewPredecessor
+70.8 pts vs Gemini 3 Flash Preview (predecessor) · Ahead of 3 of 4 similarly priced models
FrontierMath-2025-02-28-Private
Best score per model · Higher is better
#7 of 58
- 40.7%Claude Opus 4.6Nearby rank
- 40.7%GPT-5.2Nearby rank
- 39.0%Gemini 3.5 Flash
- 39.0%Kimi K2.6Nearby rank
- 37.6%Gemini 3 Pro PreviewNearby rank
- 35.6%Gemini 3 Flash PreviewPredecessor
- 33.4%GLM 5.1Similar price
- 32.4%GPT-5Similar price
- 31.0%GPT-5.1Similar price
- 5.9%Claude Haiku 4.5Similar price
+3.3 pts vs Gemini 3 Flash Preview (predecessor) · Ahead of 4 of 4 similarly priced models
FrontierMath-Tier-4-2025-07-01-Private
Best score per model · Higher is better
#9 of 46
- 16.7%Gemini 3.1 Pro PreviewNearby rank
- 14.6%GPT-5 ProNearby rank
- 14.6%Gemini 3.5 Flash
- 14.6%Kimi K2.6Nearby rank
- 12.5%GPT-5Similar price
- 12.5%GPT-5.1Similar price
- 12.5%GLM 5.1Similar price
- 4.2%Gemini 3 Flash PreviewPredecessor
- 2.1%Claude Haiku 4.5Similar price
+10.4 pts vs Gemini 3 Flash Preview (predecessor) · Ahead of 4 of 4 similarly priced models
FrontierMath-Tier-4-v2-Private
Best score per model · Higher is better
#38 of 72
- 80.5%Claude Sonnet 5.5Similar price
- 34.1%Qwen3.8 Max (0902)Similar price
- 31.7%Grok 4.6Similar price
- 26.8%DeepSeek V4.1 FlashNearby rank
- 26.8%Gemini 3.1 Pro PreviewNearby rank
- 26.8%Gemini 3.5 Flash
- 26.8%Mistral Large 4Nearby rank
- 25.6%Kimi K2.6Nearby rank
- 22.0%GPT-5Similar price
- 17.1%Gemini 3 Flash PreviewPredecessor
+9.8 pts vs Gemini 3 Flash Preview (predecessor) · Ahead of 1 of 4 similarly priced models
FrontierMath-Tiers-1-3-v2-Private
Best score per model · Higher is better
#40 of 87
- 88.8%Claude Sonnet 5.5Similar price
- 66.0%Grok 4.6Similar price
- 65.6%Qwen3.8 Max (0902)Similar price
- 64.6%DeepSeek V4 Pro 0813Nearby rank
- 64.6%Qwen3.7 MaxNearby rank
- 62.8%Gemini 3.5 Flash
- 59.6%Gemini 3.1 Pro PreviewNearby rank
- 59.2%GLM 5.2Nearby rank
- 55.4%GPT-5Similar price
- 51.2%Gemini 3 Flash PreviewPredecessor
+11.6 pts vs Gemini 3 Flash Preview (predecessor) · Ahead of 1 of 4 similarly priced models
GPQA diamond
Best score per model · Higher is better
#22 of 161
- 94.0%Grok 4.6Similar price
- 93.2%DeepSeek V4.1 Flash 1017Nearby rank
- 93.1%Kimi K3Nearby rank
- 92.8%Gemini 3.5 Flash
- 92.7%Grok 4.7Nearby rank
- 92.7%
Qwen3.8 MaxNearby rank - 92.3%Qwen3.8 Max (0902)Similar price
- 89.4%Gemini 3 Flash PreviewPredecessor
- 87.6%GPT-5.1Similar price
- 49.0%Mistral Large 2407Similar price
+3.4 pts vs Gemini 3 Flash Preview (predecessor) · Ahead of 3 of 4 similarly priced models
OTIS Mock AIME 2024-2025
Best score per model · Higher is better
#38 of 158
- 100.0%Qwen3.8 Max (0902)Similar price
- 99.2%Grok 4.6Similar price
- 95.6%Gemini 3.1 Pro PreviewNearby rank
- 95.6%Gemini 3 Flash PreviewPredecessor
- 95.6%Gemini 3.5 Flash
- 95.6%Kimi K2.7 CodeNearby rank
- 95.6%Qwen3.7 MaxNearby rank
- 91.4%GPT-5Similar price
- 8.5%Mistral Large 2407Similar price
Level with Gemini 3 Flash Preview (predecessor) · Ahead of 2 of 4 similarly priced models
SimpleQA Verified
Best score per model · Higher is better
#14 of 87
- 69.2%Gemini 3.7 FlashNearby rank
- 66.8%Gemini 3 Flash PreviewPredecessor
- 66.2%Gemini 3.5 Flash
- 66.2%Gemini 3.6 FlashNearby rank
- 63.0%GPT-5.5Nearby rank
- 56.0%Grok 4.7Similar price
- 50.1%GPT-5Similar price
- 47.3%Qwen3.8 Max (0902)Similar price
- 46.5%Claude Sonnet 5.5Similar price
-0.6 pts vs Gemini 3 Flash Preview (predecessor) · Ahead of 4 of 4 similarly priced models
SWE-bench Verified
Best score per model · Higher is better
#2 of 35
- 83.5%Claude Opus 4.7Nearby rank
- 79.3%Gemini 3.5 Flash
- 78.7%GLM 5.2Nearby rank
- 77.6%DeepSeek V4 Pro 0423Nearby rank
- 77.3%Qwen3.7 MaxSimilar price
- 75.4%Gemini 3 Flash PreviewPredecessor
- 75.2%Claude Sonnet 4.6Similar price
- 74.2%GLM 5.1Similar price
- 73.6%GPT-5Similar price
+3.9 pts vs Gemini 3 Flash Preview (predecessor) · Ahead of 4 of 4 similarly priced models
DeepSWE 1.1
Primary coding-agent benchmark · 113 tasks · Data generated 2026-09-22
- pass@4
- 63.7%
- Average cost / attempt
- $3.45
- Benchmark runs
- 4
pass@1 is the success rate across scored attempts; pass@4 is the share of tasks solved in at least one of four attempts. Costs are reported by DeepSWE for each configuration. These results are not directly comparable to evaluations using different harnesses. Official source JSON.
Reasoning effort
Arena Text
Score by reasoning effort · Higher is better
Arena WebDev
Score by reasoning effort · Higher is better
Arena Vision
Score by reasoning effort · Higher is better
Arena Document
Score by reasoning effort · Higher is better
ARC-AGI-2
Score by reasoning effort · Higher is better
GPQA diamond
Score by reasoning effort · Higher is better
OTIS Mock AIME 2024-2025
Score by reasoning effort · Higher is better
All published scores
- Arena TextArena #311478
+69 vs Gemini 2.5 Flash (earlier version)
- medium147695% CI 1472 to 1480 · 46,377 votes
- high147895% CI 1475 to 1482 · 47,642 votes
- Arena WebDevArena #691500
+244 vs Gemini 3.1 Flash Lite Preview (earlier version)
- medium149095% CI 1483 to 1496 · 11,899 votes
- high150095% CI 1492 to 1507 · 9,160 votes
- Arena VisionArena #241281
+66 vs Gemini 2.5 Flash (earlier version)
- medium128195% CI 1275 to 1288 · 15,418 votes
- high128195% CI 1274 to 1288 · 15,325 votes
- Arena DocumentArena #281459
- medium145995% CI 1451 to 1467 · 6,835 votes
- high145595% CI 1446 to 1464 · 4,248 votes
- ARC-AGI-2#25 of 8572.1%
+70.8 pts vs Gemini 3 Flash Preview (predecessor)
- minimal8.9%
- high72.1%
- FrontierMath-2025-02-28-Private#7 of 5839.0%
+3.3 pts vs Gemini 3 Flash Preview (predecessor)
- high39.0%
Source:Epoch AI evaluation - FrontierMath-Tier-4-2025-07-01-Private#9 of 4614.6%
+10.4 pts vs Gemini 3 Flash Preview (predecessor)
- high14.6%
Source:Epoch AI evaluation - FrontierMath-Tier-4-v2-Private#38 of 7226.8%
+9.8 pts vs Gemini 3 Flash Preview (predecessor)
- high26.8%
Source:Epoch AI evaluation - FrontierMath-Tiers-1-3-v2-Private#40 of 8762.8%
+11.6 pts vs Gemini 3 Flash Preview (predecessor)
- high62.8%
Source:Epoch AI evaluation - GPQA diamond#22 of 16192.8%
+3.4 pts vs Gemini 3 Flash Preview (predecessor)
- minimal86.4%
- low88.9%
- high92.8%
Source:Epoch AI evaluation - OTIS Mock AIME 2024-2025#38 of 15895.6%
Level with Gemini 3 Flash Preview (predecessor)
- minimal80.0%
- low88.9%
- high95.6%
Source:Epoch AI evaluation - SimpleQA Verified#14 of 8766.2%
-0.6 pts vs Gemini 3 Flash Preview (predecessor)
- high66.2%
Source:Epoch AI evaluation - SWE-bench Verified#2 of 3579.3%
+3.9 pts vs Gemini 3 Flash Preview (predecessor)
- high79.3%
Source:Epoch AI evaluation
Benchmark compilation: Epoch AI, Capabilities & benchmarking (CC BY attribution). Scores are converted to percentages and model names matched to the Vertical Tabs catalog. Individual evaluations are credited to their original sources. External results may carry separate licences and evaluation configurations; do not interpret scores from different harnesses as directly comparable.
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.