- Context
- 262Ktokens
- Released
- 1y ago
- Size
- —parameters
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
Not evaluated in DeepSWE 1.1
Arena Text
Best Arena rating per model · Higher is better
Arena #118
- 1486
Gemini 3 Pro PreviewSimilar release date - 1455
GPT-5.1Similar release date - 1431
Grok 4.1 Fast (Reasoning)Similar release date - 1430
MiMo-V2-OmniNearby rank - 1430
Kimi K2 Thinking Turbo - 1427
Mistral Medium 3.5Nearby rank - 1426
Nemotron 3 UltraNearby rank - 1344
MiniMax-M2Similar release date
Ahead of 1 of 4 models released around the same time
Arena WebDev
Best Arena rating per model · Higher is better
Arena #134
- 1441Gemini 3 Pro PreviewSimilar release date
- 1397GPT-5.1Similar release date
- 1331MiMo-V2-FlashNearby rank
- 1330
Claude Haiku 4.5Nearby rank - 1322Kimi K2 Thinking Turbo
- 1303
Laguna XS.2Nearby rank - 1299MiniMax-M2Similar release date
- 1242Grok 4.1 Fast (Reasoning)Similar release date
Ahead of 2 of 4 models released around the same time
Arena Search
Best Arena rating per model · Higher is better
Not ranked on Arena Search yet
Agent Arena
Best IPS score per model · Higher is better
Not ranked on Agent Arena yet
GPQA diamond
Best score per model · Higher is better
#74 of 161
- 92.6%Gemini 3 Pro PreviewSimilar release date
- 87.6%GPT-5.1Similar release date
- 86.0%Claude Opus 4.5Similar release date
- 84.8%Gemini 2.5 Pro Preview 06-05Nearby rank
- 84.8%
Qwen3.6 35B A3BNearby rank - 84.2%Kimi K2 Thinking Turbo
- 83.5%Qwen3.5-35B-A3BNearby rank
- 83.4%
DeepSeek ReasonerNearby rank - 77.6%
Seed 1.6Similar release date
Ahead of 1 of 4 models released around the same time
OTIS Mock AIME 2024-2025
Best score per model · Higher is better
#82 of 158
- 91.4%Gemini 3 Pro PreviewSimilar release date
- 88.6%GPT-5.1Similar release date
- 86.1%Claude Opus 4.5Similar release date
- 84.4%Qwen3.6 FlashNearby rank
- 83.3%
GLM 4.7Nearby rank - 83.1%Kimi K2 Thinking Turbo
- 82.2%Gemma 4 26B A4BNearby rank
- 81.1%GPT-5 NanoNearby rank
- 69.4%Seed 1.6Similar release date
Ahead of 1 of 4 models released around the same time
Not evaluated in DeepSWE 1.1 as of 2026-09-22. Results are only shown for exact model versions in the official dataset. Official leaderboard.
All published scores
- Arena TextArena #1181430
- default143095% CI 1427 to 1433 · 62,955 votes
- Arena WebDevArena #1341322
- default132295% CI 1314 to 1329 · 12,957 votes
- GPQA diamond#74 of 16184.2%
- default84.2%
Source:Epoch AI evaluation - OTIS Mock AIME 2024-2025#82 of 15883.1%
- default83.1%
Source:Epoch AI evaluation
Benchmark compilation: Epoch AI, Capabilities & benchmarking (CC BY attribution). Scores are converted to percentages and model names matched to the Vertical Tabs catalog. Individual evaluations are credited to their original sources. External results may carry separate licences and evaluation configurations; do not interpret scores from different harnesses as directly comparable.
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.