- input
- $0.30
- cached
- $0.04
- output
- $1.00
- Context
- 1Mtokens
- Listed
- 3w ago
- Size
- 501Bparameters
Cost/1M tokens
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
Not evaluated in DeepSWE 1.1
Arena Text
Best Arena rating per model · Higher is better
Arena #108
- 1436
DeepSeek V4 Flash 0423Nearby rank - 1435
GPT-5Nearby rank - 1435
Beam 1.0 501B A23B - 1434
Qwen3 Max PreviewNearby rank - 1434
GLM 5V TurboNearby rank - 1401GPT-5.4 NanoSimilar price
- 1391
MiniMax M2.5Similar price - 1378GLM 4.6VSimilar price
- 1358DeepSeek V3Similar price
Ahead of 4 of 4 similarly priced models
Arena WebDev
Best Arena rating per model · Higher is better
Arena #89
- 1624
MiMo-V2.6-ProSimilar price - 1621DeepSeek V4.1 FlashSimilar price
- 1519GPT-5.6 LunaSimilar price
- 1441
Gemini 3 Pro PreviewNearby rank - 1438MiMo-V2.5Nearby rank
- 1437Beam 1.0 501B A23B
- 1436
Kimi K2.5Nearby rank - 1435Gemini 3.5 Flash LiteNearby rank
- 1386MiniMax M2.5Similar price
Ahead of 1 of 4 similarly priced models
Arena Search
Best Arena rating per model · Higher is better
Not ranked on Arena Search yet
Agent Arena
Best IPS score per model · Higher is better
Arena #45
- +0.024MiMo-V2.6-ProSimilar price
- +0.010GLM 5.3Similar price
- -0.048
Qwen3.8 27BNearby rank - -0.054
Mistral Large 4Nearby rank - -0.059Beam 1.0 501B A23B
- -0.072Qwen3.7 MaxNearby rank
- -0.076
Hy3Nearby rank - -0.080Qwen3.7 PlusSimilar price
- -0.105MiniMax M3Similar price
Ahead of 1 of 4 similarly priced models
Not evaluated in DeepSWE 1.1 as of 2026-09-22. Results are only shown for exact model versions in the official dataset. Official leaderboard.
All published scores
- Arena TextArena #1081435
- default143595% CI 1425 to 1444 · 4,006 votes
- Arena WebDevArena #891437
- default143795% CI 1423 to 1450 · 2,515 votes
- Agent ArenaArena #45-0.059
- default-0.05995% CI -0.085 to -0.033 · 696,272 observations
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.