- input
- $1.00
- cached
- $0.10
- output
- $2.70
- Context
- 1Mtokens
- Released
- 1mo ago
- Size
- 604Bparameters
Cost/1M tokens
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
Not evaluated in DeepSWE 1.1
Arena Text
Best Arena rating per model · Higher is better
Arena #83
- 1489
Gemini 3.7 FlashSimilar price - 1461
Kimi K2.6Similar price - 1450
Claude Opus 4.1Nearby rank - 1450Kimi K2.5Nearby rank
- 1448
Step 5 Preview - 1448
MiMo-V2-ProNearby rank - 1448MiMo-V2.6-FlashNearby rank
- 1441
Qwen3.5 397B A17BSimilar price - 1290
Nova Pro 1.0Similar price
Ahead of 1 of 4 similarly priced models
Arena WebDev
Best Arena rating per model · Higher is better
Arena #46
- 1588Gemini 3.7 FlashSimilar price
- 1581
DeepSeek V4 Pro 0813Nearby rank - 1577
GPT-6 LunaNearby rank - 1573Step 5 Preview
- 1564
Mistral Large 4Nearby rank - 1557Claude Opus 4.7Nearby rank
- 1509Kimi K2.6Similar price
- 1435
GLM 5Similar price - 1400Qwen3.5 397B A17BSimilar price
Ahead of 3 of 4 similarly priced models
Arena Vision
Best Arena rating per model · Higher is better
Arena #60
- 1294Gemini 3.7 FlashSimilar price
- 1265Kimi K2.6Similar price
- 1247MiMo-V2.6-ProNearby rank
- 1246Qwen3.5 397B A17BSimilar price
- 1245Step 5 Preview
- 1243Qwen3.8 27BNearby rank
- 1242GPT-5.2Nearby rank
- 1019Nova Pro 1.0Similar price
Ahead of 1 of 4 similarly priced models
Arena Search
Best Arena rating per model · Higher is better
Not ranked on Arena Search yet
Agent Arena
Best IPS score per model · Higher is better
Arena #40
- +0.005
Hy4 previewSimilar price - +0.002GLM 5.2Similar price
- -0.022Gemini 3.7 FlashSimilar price
- -0.027
MiniMax M3.1 FlashNearby rank - -0.029GLM 5.3 FlashNearby rank
- -0.037Step 5 Preview
- -0.045Gemini 3.1 Pro PreviewNearby rank
- -0.046
Qwen3.8 Flash NextNearby rank - -0.173
InklingSimilar price
Ahead of 1 of 4 similarly priced models
Not evaluated in DeepSWE 1.1 as of 2026-09-22. Results are only shown for exact model versions in the official dataset. Official leaderboard.
All published scores
- Arena TextArena #831448
- default144895% CI 1442 to 1455 · 12,362 votes
- Arena WebDevArena #461573
- high157395% CI 1565 to 1581 · 7,583 votes
- Arena VisionArena #601245
- default124595% CI 1234 to 1256 · 3,523 votes
- Agent ArenaArena #40-0.037
- default-0.03795% CI -0.051 to -0.023 · 2,908,794 observations
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.