- Context
- 400Ktokens
- Released
- 6mo ago
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
Not evaluated in DeepSWE 1.1
Arena Text
Best Arena rating per model · Higher is better
Arena #44
- 1474
GPT-6 AstraNearby rank - 1474
Claude Opus 4.5Nearby rank - 1473GPT-5.5 Instant
- 1472Claude Sonnet 5.5Nearby rank
- 1472Claude Sonnet 4.6Nearby rank
- 1467
ERNIE 5.1Similar release date - 1467
MiMo-V2.5-ProSimilar release date - 1458
DeepSeek V4 Pro 0423Similar release date - 1427
Mistral Medium 3.5Similar release date
Ahead of 2 of 4 models released around the same time
Arena WebDev
Best Arena rating per model · Higher is better
Not ranked on Arena WebDev yet
Arena Vision
Best Arena rating per model · Higher is better
Arena #33
- 1281
Gemini 3.5 FlashSimilar release date - 1279
Muse Spark 1.1Nearby rank - 1278GPT-5.2 ChatNearby rank
- 1277GPT-5.5 Instant
- 1275Claude Sonnet 4.6Nearby rank
- 1272
GLM 5.3 FlashNearby rank - 1265
Kimi K2.6Similar release date - 1234MiMo-V2.5Similar release date
- 1199Mistral Medium 3.5Similar release date
Ahead of 2 of 4 models released around the same time
Arena Search
Best Arena rating per model · Higher is better
Not ranked on Arena Search yet
Arena Document
Best Arena rating per model · Higher is better
Arena #26
- 1494Claude Opus 4.7Similar release date
- 1480Muse Spark 1.1Similar release date
- 1466
Grok 4.5Nearby rank - 1460GPT-5.6 LunaNearby rank
- 1460GPT-5.5 Instant
- 1459Gemini 3.1 Pro PreviewNearby rank
- 1459Gemini 3.5 FlashSimilar release date
- 1450Kimi K2.6Similar release date
Ahead of 0 of 4 models released around the same time
Agent Arena
Best IPS score per model · Higher is better
Not ranked on Agent Arena yet
FrontierMath-Tier-4-v2-Private
Best score per model · Higher is better
#64 of 72
- 34.1%
Qwen3.7 MaxSimilar release date - 26.8%Gemini 3.5 FlashSimilar release date
- 25.6%Kimi K2.6Similar release date
- 2.4%DeepSeek V4 Pro 0423Similar release date
- 2.4%GPT-5 NanoNearby rank
- 2.4%GPT-5.5 Instant
- 0.0%Gemini 2.5 ProNearby rank
- 0.0%Gemini 3.5 Flash LiteNearby rank
Ahead of 0 of 4 models released around the same time
FrontierMath-Tiers-1-3-v2-Private
Best score per model · Higher is better
#71 of 87
- 57.2%Kimi K2.6Similar release date
- 45.3%DeepSeek V4 Pro 0423Similar release date
- 31.2%Qwen3.5 397B A17BNearby rank
- 27.7%Gemini 3.1 Flash LiteSimilar release date
- 26.3%GPT-5.5 Instant
- 26.0%Gemini 3.5 Flash LiteNearby rank
- 24.6%Gemini 2.5 ProNearby rank
- 22.5%Qwen3.6 FlashSimilar release date
Ahead of 1 of 4 models released around the same time
GPQA diamond
Best score per model · Higher is better
#80 of 161
- 90.9%DeepSeek V4 Pro 0423Similar release date
- 90.8%Kimi K2.6Similar release date
- 83.3%Qwen3.6 FlashSimilar release date
- 83.3%GLM 4.7Nearby rank
- 82.5%GPT-5.5 Instant
- 82.3%Claude Sonnet 4.5Nearby rank
- 82.3%Qwen3.5-FlashNearby rank
- 81.8%Gemini 3.1 Flash LiteSimilar release date
Ahead of 1 of 4 models released around the same time
OTIS Mock AIME 2024-2025
Best score per model · Higher is better
#101 of 158
- 96.7%DeepSeek V4 Pro 0423Similar release date
- 96.1%Kimi K2.6Similar release date
- 84.4%Qwen3.6 FlashSimilar release date
- 80.0%Gemini 3.1 Flash LiteSimilar release date
- 69.4%
Seed 1.6Nearby rank - 68.9%Claude Opus 4.1Nearby rank
- 68.1%GPT-5.5 Instant
- 67.5%Seed-OSS-36B-InstructNearby rank
- 66.9%Qwen3 32BNearby rank
Ahead of 0 of 4 models released around the same time
Not evaluated in DeepSWE 1.1 as of 2026-09-22. Results are only shown for exact model versions in the official dataset. Official leaderboard.
All published scores
- Arena TextArena #441473
- default147395% CI 1468 to 1478 · 26,865 votes
- Arena VisionArena #331277
- default127795% CI 1269 to 1286 · 7,567 votes
- Arena DocumentArena #261460
- default146095% CI 1452 to 1469 · 8,834 votes
- FrontierMath-Tier-4-v2-Private#64 of 722.4%
- default2.4%
Source:Epoch AI evaluation - FrontierMath-Tiers-1-3-v2-Private#71 of 8726.3%
- default26.3%
Source:Epoch AI evaluation - GPQA diamond#80 of 16182.5%
- default82.5%
Source:Epoch AI evaluation - OTIS Mock AIME 2024-2025#101 of 15868.1%
- default68.1%
Source:Epoch AI evaluation
Benchmark compilation: Epoch AI, Capabilities & benchmarking (CC BY attribution). Scores are converted to percentages and model names matched to the Vertical Tabs catalog. Individual evaluations are credited to their original sources. External results may carry separate licences and evaluation configurations; do not interpret scores from different harnesses as directly comparable.
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.