- input
- $1.50
- output
- $7.50
- Context
- 262Ktokens
- Released
- 6mo ago
- Size
- 128Bparameters
Cost/1M tokens
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
Not evaluated in DeepSWE 1.1
Arena Text
Best Arena rating per model · Higher is better
Arena #121
- 1483
Kimi K3.1 ProSimilar price - 1478
Gemini 3.5 FlashSimilar price - 1466
Grok 4.5Similar price - 1455
GPT-5.1Similar price - 1430
MiMo-V2-OmniNearby rank - 1430Kimi K2 Thinking TurboNearby rank
- 1427
Mistral Medium 3.5 - 1426
Nemotron 3 UltraNearby rank - 1426
Claude Opus 4Nearby rank - 1408Mistral Medium 3.1Predecessor
+18 vs Mistral Medium 3.1 (predecessor) · Ahead of 0 of 4 similarly priced models
Arena WebDev
Best Arena rating per model · Higher is better
Arena #140
- 1676
Qwen3.8 Max (0902)Similar price - 1641Grok 4.7Similar price
- 1500Gemini 3.5 FlashSimilar price
- 1417GPT-5Similar price
- 1299
MiniMax-M2Nearby rank - 1270
DeepSeek V3.2 ExpNearby rank - 1263Mistral Medium 3.5
- 1256Gemini 3.1 Flash Lite PreviewNearby rank
- 1253Qwen3.5-35B-A3BNearby rank
Ahead of 0 of 4 similarly priced models
Arena Vision
Best Arena rating per model · Higher is better
Arena #94
- 1281Gemini 3.5 FlashSimilar price
- 1280Grok 4.5Similar price
- 1263Claude Sonnet 5.5Similar price
- 1251GPT-5.1Similar price
- 1201GPT-4.1 MiniNearby rank
- 1200Mistral Large 3 2512Nearby rank
- 1199Mistral Medium 3.5
- 1199GPT-5.4 NanoNearby rank
- 1196Grok 4.1 Fast (Reasoning)Nearby rank
- 1160Mistral Medium 3.1Predecessor
+40 vs Mistral Medium 3.1 (predecessor) · Ahead of 0 of 4 similarly priced models
Arena Search
Best Arena rating per model · Higher is better
Not ranked on Arena Search yet
Arena Document
Best Arena rating per model · Higher is better
Not ranked on Arena Document yet
Agent Arena
Best IPS score per model · Higher is better
Arena #54
- +0.170Gemini 4 CarbonSimilar price
- +0.160Claude Sonnet 5.5Similar price
- +0.049Kimi K3.1 ProSimilar price
- +0.026Grok 4.7Similar price
- -0.082
Muse Spark 1.2Nearby rank - -0.105MiniMax M3Nearby rank
- -0.137Mistral Medium 3.5
- -0.156Nemotron 3 UltraNearby rank
- -0.173
InklingNearby rank
Ahead of 0 of 4 similarly priced models
Not evaluated in DeepSWE 1.1 as of 2026-09-22. Results are only shown for exact model versions in the official dataset. Official leaderboard.
All published scores
- Arena TextArena #1211427
+18 vs Mistral Medium 3.1 (predecessor)
- default142795% CI 1420 to 1433 · 11,656 votes
- Arena WebDevArena #1401263
- default126395% CI 1248 to 1278 · 2,280 votes
- Arena VisionArena #941199
+40 vs Mistral Medium 3.1 (predecessor)
- default119995% CI 1190 to 1209 · 5,330 votes
- Agent ArenaArena #54-0.137
- default-0.13795% CI -0.155 to -0.119 · 670,183 observations
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.