- input
- $0.09
- cached
- $0.018
- output
- $0.18
- Context
- 1Mtokens
- Released
- 6mo ago
- Size
- 291Bparameters
Cost/1M tokens
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
#20 of 28
- 74.1%
- 73.8%
- 73.6%
- 72.7%
- 69.9%
- 69.6%
- 69.0%
- 68.5%
- 67.5%
- 67.2%
- 53.3%
DeepSeek V4 Flash 0423
Arena Text
Best Arena rating per model · Higher is better
Arena #105
- 1437GPT-5.2Nearby rank
- 1436
MiniMax M3Nearby rank - 1436DeepSeek V4 Flash 0423
- 1435GPT-5Nearby rank
- 1435
Beam 1.0 501B A23BNearby rank - 1396
Qwen3.5-FlashSimilar price - 1314
Nemotron 3 Nano 30B A3BSimilar price - 1260
Nova Lite 1.0Similar price - 1167
Llama 3.2 3B InstructSimilar price
Ahead of 4 of 4 similarly priced models
Arena WebDev
Best Arena rating per model · Higher is better
Arena #43
- 1585MiniMax M3.1 FlashSimilar price
- 1583Gemini 3.8 FlashNearby rank
- 1582DeepSeek V4 Flash 0423
- 1581DeepSeek V4 Pro 0813Nearby rank
- 1577GPT-6 LunaNearby rank
- 1506
Hy3Similar price - 1358Gemma 4 26B A4BSimilar price
- 1244Qwen3.5-FlashSimilar price
Ahead of 3 of 4 similarly priced models
Arena Search
Best Arena rating per model · Higher is better
Not ranked on Arena Search yet
Agent Arena
Best IPS score per model · Higher is better
Not ranked on Agent Arena yet
DeepSWE 1.1
Primary coding-agent benchmark · 113 tasks · Data generated 2026-09-22
- pass@4
- 80.5%
- Average cost / attempt
- $0.10
- Benchmark runs
- 4
pass@1 is the success rate across scored attempts; pass@4 is the share of tasks solved in at least one of four attempts. Costs are reported by DeepSWE for each configuration. These results are not directly comparable to evaluations using different harnesses. Official source JSON.
All published scores
- Arena TextArena #1051436
- default143695% CI 1432 to 1440 · 51,368 votes
- Arena WebDevArena #431582
- high158295% CI 1572 to 1591 · 4,830 votes
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.