- input
- $0.30
- cached
- $0.06
- output
- $1.20
- Context
- 1Mtokens
- Released
- 5mo ago
- Size
- 0parameters
Cost/1M tokens
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
Not evaluated in DeepSWE 1.1
Arena Text
Best Arena rating per model · Higher is better
Arena #104
- 1480
MiMo-V2.6-ProSimilar price - 1475
DeepSeek V4.1 FlashSimilar price - 1455
Qwen3.7 PlusSimilar price - 1437
Gemma 4 26B A4BNearby rank - 1437
GPT-5.2Nearby rank - 1436
MiniMax M3 - 1436DeepSeek V4 Flash 0423Nearby rank
- 1435GPT-5Nearby rank
- 1415MiniMax M2.7Predecessor
- 1327
Llama 4 MaverickSimilar price
+21 vs MiniMax M2.7 (predecessor) · Ahead of 1 of 4 similarly priced models
Arena WebDev
Best Arena rating per model · Higher is better
Arena #74
- 1624MiMo-V2.6-ProSimilar price
- 1621DeepSeek V4.1 FlashSimilar price
- 1500Gemini 3.5 FlashNearby rank
- 1493
Claude Opus 4.5Nearby rank - 1484MiniMax M3
- 1482
Qwen3.6 Max PreviewNearby rank - 1479MiMo-V2.5-ProNearby rank
- 1437
Beam 1.0 501B A23BSimilar price - 1398MiniMax M2.7Predecessor
- 1256Gemini 3.1 Flash Lite PreviewSimilar price
+86 vs MiniMax M2.7 (predecessor) · Ahead of 2 of 4 similarly priced models
Arena Vision
Best Arena rating per model · Higher is better
Arena #71
- 1263DeepSeek V4.1 FlashSimilar price
- 1262Qwen3.7 PlusSimilar price
- 1247MiMo-V2.6-ProSimilar price
- 1235Gemini 3.1 Flash Lite PreviewNearby rank
- 1234MiMo-V2.5Nearby rank
- 1232MiniMax M3
- 1232
GLM 5V TurboNearby rank - 1231
Mistral Large 4Nearby rank - 1145Llama 4 MaverickSimilar price
Ahead of 1 of 4 similarly priced models
Arena Search
Best Arena rating per model · Higher is better
Not ranked on Arena Search yet
Arena Document
Best Arena rating per model · Higher is better
Arena #40
- 1460GPT-5.6 LunaSimilar price
- 1446Qwen3.7 PlusSimilar price
- 1438
Kimi K2.5Similar price - 1435MiniMax M3
- 1433Gemini 2.5 ProNearby rank
- 1425GPT-5.2Nearby rank
Ahead of 0 of 3 similarly priced models
Agent Arena
Best IPS score per model · Higher is better
Arena #52
- +0.053
Hy4Similar price - +0.024MiMo-V2.6-ProSimilar price
- -0.059Beam 1.0 501B A23BSimilar price
- -0.080Qwen3.7 PlusSimilar price
- -0.080Muse Spark 1.1Nearby rank
- -0.082Muse Spark 1.2Nearby rank
- -0.105MiniMax M3
- -0.137Mistral Medium 3.5Nearby rank
- -0.156
Nemotron 3 UltraNearby rank
Ahead of 0 of 4 similarly priced models
GPQA diamond
Best score per model · Higher is better
#33 of 161
- 91.0%DeepSeek V4 Flash 0731Nearby rank
- 90.9%DeepSeek V4 Pro 0423Nearby rank
- 90.9%MiniMax M3
- 90.9%Qwen3.7 MaxNearby rank
- 90.9%GLM 5.3Nearby rank
- 89.8%DeepSeek V4.1 FlashSimilar price
- 87.9%Qwen3.7 PlusSimilar price
- 81.8%Gemini 3.1 Flash LiteSimilar price
- 78.5%GPT-5.4 NanoSimilar price
Ahead of 4 of 4 similarly priced models
OSWorld 2.0
Best score per model · Higher is better
#7 of 9
- 13.0%GPT-5.5Nearby rank
- 9.3%Claude Sonnet 4.6Nearby rank
- 4.6%MiniMax M3
- 4.6%Kimi K2.6Nearby rank
- 2.8%Qwen3.7 PlusSimilar price
Ahead of 1 of 1 similarly priced model
OTIS Mock AIME 2024-2025
Best score per model · Higher is better
#91 of 158
- 98.3%DeepSeek V4.1 FlashSimilar price
- 93.3%Qwen3.7 PlusSimilar price
- 87.8%GPT-5.4 NanoSimilar price
- 80.0%Gemini 3.1 Flash LiteSimilar price
- 71.1%Claude Sonnet 4Nearby rank
- 71.1%Gemini 3.5 Flash LiteNearby rank
- 71.1%MiniMax M3
- 70.8%Gemini 2.5 Flash Preview 05-20Nearby rank
- 70.0%Qwen3.5-35B-A3BNearby rank
Ahead of 0 of 4 similarly priced models
Not evaluated in DeepSWE 1.1 as of 2026-09-22. Results are only shown for exact model versions in the official dataset. Official leaderboard.
All published scores
- Arena TextArena #1041436
+21 vs MiniMax M2.7 (predecessor)
- default143695% CI 1432 to 1440 · 72,963 votes
- Arena WebDevArena #741484
+86 vs MiniMax M2.7 (predecessor)
- default148495% CI 1478 to 1489 · 21,048 votes
- Arena VisionArena #711232
- default123295% CI 1226 to 1238 · 21,159 votes
- Arena DocumentArena #401435
- default143595% CI 1427 to 1443 · 6,617 votes
- Agent ArenaArena #52-0.105
- default-0.10595% CI -0.116 to -0.093 · 3,757,219 observations
- GPQA diamond#33 of 16190.9%
- none81.3%
- default90.9%
Source:Epoch AI evaluation - OSWorld 2.0#7 of 94.6%
- default4.6%
- OTIS Mock AIME 2024-2025#91 of 15871.1%
- none26.7%
- default71.1%
Source:Epoch AI evaluation
Benchmark compilation: Epoch AI, Capabilities & benchmarking (CC BY attribution). Scores are converted to percentages and model names matched to the Vertical Tabs catalog. Individual evaluations are credited to their original sources. External results may carry separate licences and evaluation configurations; do not interpret scores from different harnesses as directly comparable.
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.