- Context
- 262Ktokens
- Released
- 6mo ago
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
Not evaluated in DeepSWE 1.1
Arena Text
Best Arena rating per model · Higher is better
Arena #62
- 1481
GPT-5.5Similar release date - 1467
MiMo-V2.5-ProSimilar release date - 1461
Claude Sonnet 5Nearby rank - 1461
Kimi K2.6Similar release date - 1460
Qwen3.6 Max Preview - 1458
DeepSeek V4 Pro 0423Nearby rank - 1457
GLM 5Nearby rank - 1434Qwen3 Max PreviewPredecessor
- 1409
Hy3 previewSimilar release date
+25 vs Qwen3 Max Preview (predecessor) · Ahead of 1 of 4 models released around the same time
Arena WebDev
Best Arena rating per model · Higher is better
Arena #75
- 1512GPT-5.5Similar release date
- 1509Kimi K2.6Similar release date
- 1493Claude Opus 4.5Nearby rank
- 1484
MiniMax M3Nearby rank - 1482Qwen3.6 Max Preview
- 1479MiMo-V2.5-ProSimilar release date
- 1474Kimi K2.7 CodeNearby rank
- 1358Hy3 previewSimilar release date
Ahead of 1 of 4 models released around the same time
Arena Search
Best Arena rating per model · Higher is better
Not ranked on Arena Search yet
Agent Arena
Best IPS score per model · Higher is better
Not ranked on Agent Arena yet
FrontierMath-2025-02-28-Private
Best score per model · Higher is better
#20 of 58
- 43.8%Claude Opus 4.7Similar release date
- 39.0%Kimi K2.6Similar release date
- 33.4%GLM 5.1Similar release date
- 26.2%
Qwen3.6 PlusNearby rank - 25.9%GPT-5.4 NanoNearby rank
- 23.1%Qwen3.6 Max Preview
- 21.4%Kimi K2 ThinkingNearby rank
- 21.0%Qwen3.5 PlusNearby rank
- 10.3%Qwen3.6 FlashSimilar release date
Ahead of 1 of 4 models released around the same time
FrontierMath-Tier-4-2025-07-01-Private
Best score per model · Higher is better
#18 of 46
- 22.9%Claude Opus 4.7Similar release date
- 14.6%Kimi K2.6Similar release date
- 12.5%GLM 5.1Similar release date
- 6.3%GPT-5 MiniNearby rank
- 6.3%GPT-5.4 NanoNearby rank
- 4.2%Qwen3.6 Max Preview
- 4.2%Claude Opus 4Nearby rank
- 4.2%Claude Opus 4.1Nearby rank
- 0.0%Qwen3.6 FlashSimilar release date
Ahead of 1 of 4 models released around the same time
GPQA diamond
Best score per model · Higher is better
#59 of 161
- 90.8%Kimi K2.6Similar release date
- 90.7%GPT-5.5Similar release date
- 88.8%
Grok 4.3Similar release date - 87.8%GLM 5Nearby rank
- 87.6%GPT-5.1Nearby rank
- 87.4%Qwen3.6 Max Preview
- 87.4%Claude Sonnet 4.6Nearby rank
- 86.9%GPT-5.4 MiniNearby rank
- 85.9%Qwen3.6 27BSimilar release date
Ahead of 1 of 4 models released around the same time
OTIS Mock AIME 2024-2025
Best score per model · Higher is better
#54 of 158
- 96.1%Kimi K2.6Similar release date
- 93.3%Grok 4.3Similar release date
- 91.4%
Gemini 3 Pro PreviewNearby rank - 91.4%GPT-5Nearby rank
- 91.1%Qwen3.6 Max Preview
- 91.1%Qwen3.6 27BSimilar release date
- 91.1%GLM 5.3Nearby rank
- 84.4%GPT-5.5Similar release date
Ahead of 1 of 4 models released around the same time
SimpleQA Verified
Best score per model · Higher is better
#26 of 87
- 63.0%GPT-5.5Similar release date
- 53.0%Claude Opus 4.8Nearby rank
- 52.9%DeepSeek V4 Pro 0813Nearby rank
- 52.0%Qwen3.6 Max Preview
- 51.7%Claude Opus 4.7Similar release date
- 50.6%Kimi K3Nearby rank
- 34.9%Kimi K2.6Similar release date
- 33.2%Grok 4.3Similar release date
Ahead of 3 of 4 models released around the same time
SWE-bench Verified
Best score per model · Higher is better
#7 of 35
- 83.5%Claude Opus 4.7Similar release date
- 77.6%DeepSeek V4 Pro 0423Similar release date
- 77.3%Qwen3.7 MaxNearby rank
- 76.9%GPT-5.4Nearby rank
- 76.7%Qwen3.6 Max Preview
- 76.7%Claude Opus 4.5Nearby rank
- 76.7%Kimi K2.6Similar release date
- 74.2%GLM 5.1Similar release date
Ahead of 1 of 4 models released around the same time
Not evaluated in DeepSWE 1.1 as of 2026-09-22. Results are only shown for exact model versions in the official dataset. Official leaderboard.
All published scores
- Arena TextArena #621460
+25 vs Qwen3 Max Preview (predecessor)
- default146095% CI 1451 to 1468 · 5,386 votes
- Arena WebDevArena #751482
- default148295% CI 1470 to 1495 · 2,674 votes
- FrontierMath-2025-02-28-Private#20 of 5823.1%
- default23.1%
Source:Epoch AI evaluation - FrontierMath-Tier-4-2025-07-01-Private#18 of 464.2%
- default4.2%
Source:Epoch AI evaluation - GPQA diamond#59 of 16187.4%
- default87.4%
Source:Epoch AI evaluation - OTIS Mock AIME 2024-2025#54 of 15891.1%
- default91.1%
Source:Epoch AI evaluation - SimpleQA Verified#26 of 8752.0%
- default52.0%
Source:Epoch AI evaluation - SWE-bench Verified#7 of 3576.7%
- default76.7%
Source:Epoch AI evaluation
Benchmark compilation: Epoch AI, Capabilities & benchmarking (CC BY attribution). Scores are converted to percentages and model names matched to the Vertical Tabs catalog. Individual evaluations are credited to their original sources. External results may carry separate licences and evaluation configurations; do not interpret scores from different harnesses as directly comparable.
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.