- Context
- 1Mtokens
- Released
- 11mo ago
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
Not evaluated in DeepSWE 1.1
Arena Text
Best Arena rating per model · Higher is better
Arena #21
- 1487
Kimi K3Nearby rank - 1487
Gemini 3.1 Pro PreviewNearby rank - 1486Gemini 3 Pro Preview
- 1485
GPT-5.6 SolNearby rank - 1484
Qwen3.8 MaxNearby rank - 1474
Claude Opus 4.5Similar release date - 1455GPT-5.1Similar release date
- 1446Gemini 2.5 ProPredecessor
- 1431
Grok 4.1 Fast (Reasoning)Similar release date - 1306
Olmo 3 32B ThinkSimilar release date
+40 vs Gemini 2.5 Pro (predecessor) · Ahead of 4 of 4 models released around the same time
Arena WebDev
Best Arena rating per model · Higher is better
Arena #86
- 1493Claude Opus 4.5Similar release date
- 1448
DeepSeek V4 Pro 0423Nearby rank - 1446Gemini 3.1 Pro PreviewNearby rank
- 1441Gemini 3 Pro Preview
- 1438
MiMo-V2.5Nearby rank - 1437
Beam 1.0 501B A23BNearby rank - 1397GPT-5.1Similar release date
- 1322Kimi K2 Thinking TurboSimilar release date
- 1242Grok 4.1 Fast (Reasoning)Similar release date
- 1228Gemini 2.5 ProPredecessor
+213 vs Gemini 2.5 Pro (predecessor) · Ahead of 3 of 4 models released around the same time
Arena Vision
Best Arena rating per model · Higher is better
Arena #14
- 1293GPT-6.1 SolNearby rank
- 1291
Muse Spark 1.3Nearby rank - 1289Gemini 3 Pro Preview
- 1289Claude Fable 5.1Nearby rank
- 1288GPT-5.6 SolNearby rank
- 1251GPT-5.1Similar release date
- 1247Gemini 2.5 ProPredecessor
- 1200
Mistral Large 3 2512Similar release date - 1196Grok 4.1 Fast (Reasoning)Similar release date
- 1161
GLM 4.6VSimilar release date
+42 vs Gemini 2.5 Pro (predecessor) · Ahead of 4 of 4 models released around the same time
Arena Search
Best Arena rating per model · Higher is better
Arena #18
- 1210Claude Opus 4.7Similar release date
- 1205Grok 4.5Nearby rank
- 1201Gemini 3 Pro Preview
- 1194Claude Opus 4.8Nearby rank
- 1194Grok 4.3Similar release date
- 1192
ERNIE 5.1Similar release date
Ahead of 0 of 3 models released around the same time
Arena Document
Best Arena rating per model · Higher is better
Arena #32
- 1467Claude Opus 4.5Similar release date
- 1454Gemini 3.6 FlashNearby rank
- 1452Gemma 4 31BNearby rank
- 1451Gemini 3 Pro Preview
- 1450Kimi K2.6Nearby rank
- 1450Grok 4.6Nearby rank
- 1438Kimi K2.5Similar release date
- 1433Gemini 2.5 ProPredecessor
- 1416GPT-5.1Similar release date
- 1406GLM 5V TurboSimilar release date
+18 vs Gemini 2.5 Pro (predecessor) · Ahead of 2 of 4 models released around the same time
Agent Arena
Best IPS score per model · Higher is better
Not ranked on Agent Arena yet
ARC-AGI-2
Best score per model · Higher is better
#48 of 85
- 37.6%Claude Opus 4.5Similar release date
- 36.5%
InklingNearby rank - 31.1%Gemini 3 Pro Preview
- 22.8%GLM 5.2Nearby rank
- 18.9%GPT-5.4 MiniNearby rank
- 17.6%GPT-5.1Similar release date
- 11.8%Kimi K2.5Similar release date
- 4.9%Gemini 2.5 ProPredecessor
- 4.0%DeepSeek V3.2Similar release date
+26.3 pts vs Gemini 2.5 Pro (predecessor) · Ahead of 3 of 4 models released around the same time
FrontierMath-2025-02-28-Private
Best score per model · Higher is better
#9 of 58
- 39.0%Gemini 3.5 FlashNearby rank
- 39.0%Kimi K2.6Nearby rank
- 37.6%Gemini 3 Pro Preview
- 36.9%Gemini 3.1 Pro PreviewNearby rank
- 35.6%Gemini 3 Flash PreviewNearby rank
- 31.0%GPT-5.1Similar release date
- 21.4%Kimi K2 ThinkingSimilar release date
- 20.7%Claude Opus 4.5Similar release date
- 14.1%Gemini 2.5 ProPredecessor
- 1.7%DeepSeek V3Similar release date
+23.5 pts vs Gemini 2.5 Pro (predecessor) · Ahead of 4 of 4 models released around the same time
FrontierMath-Tier-4-2025-07-01-Private
Best score per model · Higher is better
#6 of 46
- 22.9%Claude Opus 4.6Nearby rank
- 18.8%GPT-5.2Nearby rank
- 18.8%Gemini 3 Pro Preview
- 16.7%Gemini 3.1 Pro PreviewNearby rank
- 14.6%GPT-5 ProNearby rank
- 12.5%GPT-5.1Similar release date
- 4.2%Claude Opus 4.5Similar release date
- 4.2%Gemini 2.5 ProPredecessor
- 0.0%Kimi K2 ThinkingSimilar release date
- 0.0%GLM 4.7Similar release date
+14.6 pts vs Gemini 2.5 Pro (predecessor) · Ahead of 4 of 4 models released around the same time
GPQA diamond
Best score per model · Higher is better
#25 of 161
- 92.7%Grok 4.7Nearby rank
- 92.7%Qwen3.8 MaxNearby rank
- 92.6%Gemini 3 Pro Preview
- 92.3%
Qwen3.8 Max (0902)Nearby rank - 91.9%GLM 5.2Nearby rank
- 87.6%GPT-5.1Similar release date
- 86.0%Claude Opus 4.5Similar release date
- 85.3%Gemini 2.5 ProPredecessor
- 84.2%Kimi K2 Thinking TurboSimilar release date
- 83.4%DeepSeek ReasonerSimilar release date
+7.3 pts vs Gemini 2.5 Pro (predecessor) · Ahead of 4 of 4 models released around the same time
Humanity's Last Exam
Best score per model · Higher is better
#7 of 34
- 44.5%Gemini 3.8 FlashNearby rank
- 44.3%GPT-5.4 ProNearby rank
- 37.5%Gemini 3 Pro Preview
- 36.2%GPT-5.4Nearby rank
- 36.2%Claude Opus 4.7Nearby rank
- 24.4%Kimi K2.5Similar release date
- 23.7%GPT-5.1Similar release date
- 21.6%Gemini 2.5 Pro Preview 06-05Earlier version
- 14.2%Claude Opus 4.5Similar release date
- 8.3%GLM 4.5Similar release date
+15.9 pts vs Gemini 2.5 Pro Preview 06-05 (earlier version) · Ahead of 4 of 4 models released around the same time
OTIS Mock AIME 2024-2025
Best score per model · Higher is better
#52 of 158
- 93.3%GLM 5.1Nearby rank
- 92.2%Grok 4.20 (Reasoning)Nearby rank
- 91.4%Gemini 3 Pro Preview
- 91.4%GPT-5Nearby rank
- 91.1%Qwen3.6 Max PreviewNearby rank
- 88.6%GPT-5.1Similar release date
- 87.8%DeepSeek ReasonerSimilar release date
- 86.1%Claude Opus 4.5Similar release date
- 84.7%Gemini 2.5 ProPredecessor
- 83.1%Kimi K2 Thinking TurboSimilar release date
+6.7 pts vs Gemini 2.5 Pro (predecessor) · Ahead of 4 of 4 models released around the same time
SWE-bench Verified
Best score per model · Higher is better
#23 of 35
- 76.7%Claude Opus 4.5Similar release date
- 73.8%Kimi K2.5Similar release date
- 73.6%GPT-5Nearby rank
- 73.3%Claude Opus 4.1Nearby rank
- 72.9%Gemini 3 Pro Preview
- 72.1%GLM 5Similar release date
- 71.3%Claude Sonnet 4.5Nearby rank
- 65.9%GPT-5.1Similar release date
- 57.6%Gemini 2.5 ProPredecessor
+15.4 pts vs Gemini 2.5 Pro (predecessor) · Ahead of 2 of 4 models released around the same time
Terminal Bench
Best score per model · Higher is better
#7 of 49
- 79.8%Claude Opus 4.6Nearby rank
- 78.4%GPT-5.3-CodexNearby rank
- 69.4%Gemini 3 Pro Preview
- 66.5%GPT-5.2-CodexNearby rank
- 64.9%GPT-5.2Nearby rank
- 63.1%Claude Opus 4.5Similar release date
- 61.6%GPT-5.1-Codex-MiniSimilar release date
- 39.6%DeepSeek V3.2Similar release date
- 35.7%Kimi K2 ThinkingSimilar release date
- 32.6%Gemini 2.5 ProPredecessor
+36.8 pts vs Gemini 2.5 Pro (predecessor) · Ahead of 4 of 4 models released around the same time
Not evaluated in DeepSWE 1.1 as of 2026-09-22. Results are only shown for exact model versions in the official dataset. Official leaderboard.
All published scores
- Arena TextArena #211486
+40 vs Gemini 2.5 Pro (predecessor)
- default148695% CI 1482 to 1489 · 41,829 votes
- Arena WebDevArena #861441
+213 vs Gemini 2.5 Pro (predecessor)
- default144195% CI 1432 to 1449 · 13,929 votes
- Arena VisionArena #141289
+42 vs Gemini 2.5 Pro (predecessor)
- default128995% CI 1282 to 1297 · 13,547 votes
- Arena SearchArena #181201
- default120195% CI 1195 to 1206 · 38,324 votes
- Arena DocumentArena #321451
+18 vs Gemini 2.5 Pro (predecessor)
- default145195% CI 1442 to 1460 · 11,338 votes
- ARC-AGI-2#48 of 8531.1%
+26.3 pts vs Gemini 2.5 Pro (predecessor)
- default31.1%
- FrontierMath-2025-02-28-Private#9 of 5837.6%
+23.5 pts vs Gemini 2.5 Pro (predecessor)
- default37.6%
Source:Epoch AI evaluation - FrontierMath-Tier-4-2025-07-01-Private#6 of 4618.8%
+14.6 pts vs Gemini 2.5 Pro (predecessor)
- default18.8%
Source:Epoch AI evaluation - GPQA diamond#25 of 16192.6%
+7.3 pts vs Gemini 2.5 Pro (predecessor)
- default92.6%
Source:Epoch AI evaluation - Humanity's Last Exam#7 of 3437.5%
+15.9 pts vs Gemini 2.5 Pro Preview 06-05 (earlier version)
- default37.5%
- OTIS Mock AIME 2024-2025#52 of 15891.4%
+6.7 pts vs Gemini 2.5 Pro (predecessor)
- default91.4%
Source:Epoch AI evaluation - SWE-bench Verified#23 of 3572.9%
+15.4 pts vs Gemini 2.5 Pro (predecessor)
- default72.9%
Source:Epoch AI evaluation - Terminal Bench#7 of 4969.4%
+36.8 pts vs Gemini 2.5 Pro (predecessor)
- default69.4%Antehttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- default65.2%SageAgenthttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- default62.2%CodeBrain-1.5https://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- default62.2%CodeBrain-1External evaluation via Epoch AIOriginal-source terms apply
- default61.8%II-Agenthttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- default61.1%Droidhttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- default56.9%Terminus 2https://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- default56.0%Letta Codehttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
Benchmark compilation: Epoch AI, Capabilities & benchmarking (CC BY attribution). Scores are converted to percentages and model names matched to the Vertical Tabs catalog. Individual evaluations are credited to their original sources. External results may carry separate licences and evaluation configurations; do not interpret scores from different harnesses as directly comparable.
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.