- input
- $1.00
- cached
- $0.20
- output
- $3.20
- Context
- 205Ktokens
- Released
- 9mo ago
- Size
- 754Bparameters
Cost/1M tokens
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
Not evaluated in DeepSWE 1.1
Arena Text
Best Arena rating per model · Higher is better
Arena #66
- 1489
Gemini 3.7 FlashSimilar price - 1460
Qwen3.6 Max PreviewNearby rank - 1458
DeepSeek V4 Pro 0423Nearby rank - 1457
GLM 5 - 1457
GPT-6 SolNearby rank - 1457
Seed 2.0 ProNearby rank - 1448
Step 5 PreviewSimilar price - 1447GPT-5.4 MiniSimilar price
- 1442
Grok 4.3Similar price
Ahead of 2 of 4 similarly priced models
Arena WebDev
Best Arena rating per model · Higher is better
Arena #92
- 1588Gemini 3.7 FlashSimilar price
- 1573Step 5 PreviewSimilar price
- 1436
Kimi K2.5Nearby rank - 1435Gemini 3.5 Flash LiteNearby rank
- 1435GLM 5
- 1435GLM 4.7Nearby rank
- 1434
MiMo-V2-ProNearby rank - 1398GPT-5.4 MiniSimilar price
- 1357Grok 4.3Similar price
Ahead of 2 of 4 similarly priced models
Arena Search
Best Arena rating per model · Higher is better
Not ranked on Arena Search yet
Agent Arena
Best IPS score per model · Higher is better
Not ranked on Agent Arena yet
ARC-AGI-2
Best score per model · Higher is better
#62 of 85
- 84.6%Gemini 3.7 FlashSimilar price
- 65.1%Grok 4.20Similar price
- 36.5%
InklingSimilar price - 18.9%GPT-5.4 MiniSimilar price
- 4.9%
MiniMax M2.5Nearby rank - 4.9%o3 ProNearby rank
- 4.9%GLM 5
- 4.4%GPT-5 MiniNearby rank
- 4.0%
Claude Haiku 4.5Nearby rank
Ahead of 0 of 4 similarly priced models
FrontierMath-2025-02-28-Private
Best score per model · Higher is better
#27 of 58
- 39.0%Kimi K2.6Similar price
- 28.3%GPT-5.4 MiniSimilar price
- 20.7%Claude Opus 4.5Nearby rank
- 18.7%o3Nearby rank
- 16.4%GLM 5
- 15.2%Claude Sonnet 4.5Nearby rank
- 14.1%Gemini 2.5 ProNearby rank
- 8.5%
Qwen3 235B A22B Thinking 2507Similar price - 5.9%Claude Haiku 4.5Similar price
Ahead of 2 of 4 similarly priced models
FrontierMath-Tier-4-2025-07-01-Private
Best score per model · Higher is better
#31 of 46
- 14.6%Kimi K2.6Similar price
- 4.2%Gemini 3 Flash PreviewNearby rank
- 2.1%GLM 4.6Nearby rank
- 2.1%GLM 5
- 2.1%Qwen3.5 PlusNearby rank
- 2.1%Claude Haiku 4.5Similar price
- 2.1%GPT-5.4 MiniSimilar price
- 0.0%Qwen3 235B A22B Thinking 2507Similar price
Ahead of 1 of 4 similarly priced models
GPQA diamond
Best score per model · Higher is better
#57 of 161
- 94.8%Gemini 3.7 FlashSimilar price
- 88.8%Grok 4.3Similar price
- 88.3%InklingSimilar price
- 87.9%Kimi K2.7 CodeNearby rank
- 87.9%Qwen3.7 PlusNearby rank
- 87.8%GLM 5
- 87.6%GPT-5.1Nearby rank
- 87.4%Qwen3.6 Max PreviewNearby rank
- 86.9%GPT-5.4 MiniSimilar price
Ahead of 1 of 4 similarly priced models
OTIS Mock AIME 2024-2025
Best score per model · Higher is better
#85 of 158
- 97.2%Gemini 3.7 FlashSimilar price
- 93.3%Grok 4.3Similar price
- 88.9%GPT-5.4 MiniSimilar price
- 88.9%InklingSimilar price
- 81.1%GPT-5 NanoNearby rank
- 80.0%Gemini 3.1 Flash LiteNearby rank
- 80.0%GLM 5
- 77.8%Claude Sonnet 4.5Nearby rank
- 73.3%Qwen3 MaxNearby rank
Ahead of 0 of 4 similarly priced models
SWE-bench Verified
Best score per model · Higher is better
#24 of 35
- 77.6%DeepSeek V4 Pro 0423Similar price
- 77.3%Qwen3.7 MaxSimilar price
- 76.7%Kimi K2.6Similar price
- 75.4%Gemini 3 Flash PreviewSimilar price
- 73.3%Claude Opus 4.1Nearby rank
- 72.9%Gemini 3 Pro PreviewNearby rank
- 72.1%GLM 5
- 71.3%Claude Sonnet 4.5Nearby rank
- 70.7%Claude Opus 4Nearby rank
Ahead of 0 of 4 similarly priced models
Terminal Bench
Best score per model · Higher is better
#18 of 49
- 64.3%Gemini 3 Flash PreviewSimilar price
- 57.3%Grok 4.20Similar price
- 53.4%Claude Sonnet 4.6Nearby rank
- 52.4%GLM 5
- 49.6%GPT-5Nearby rank
- 47.6%GPT-5.1Nearby rank
- 35.7%Kimi K2 ThinkingSimilar price
- 35.5%Claude Haiku 4.5Similar price
Ahead of 2 of 4 similarly priced models
Not evaluated in DeepSWE 1.1 as of 2026-09-22. Results are only shown for exact model versions in the official dataset. Official leaderboard.
All published scores
- Arena TextArena #661457
- default145795% CI 1453 to 1462 · 29,020 votes
- Arena WebDevArena #921435
- default143595% CI 1427 to 1443 · 7,417 votes
- ARC-AGI-2#62 of 854.9%
- default4.9%
- FrontierMath-2025-02-28-Private#27 of 5816.4%
- default16.4%
Source:Epoch AI evaluation - FrontierMath-Tier-4-2025-07-01-Private#31 of 462.1%
- default2.1%
Source:Epoch AI evaluation - GPQA diamond#57 of 16187.8%
- default87.8%
Source:Epoch AI evaluation - OTIS Mock AIME 2024-2025#85 of 15880.0%
- default80.0%
Source:Epoch AI evaluation - SWE-bench Verified#24 of 3572.1%
- default72.1%
Source:Epoch AI evaluation - Terminal Bench#18 of 4952.4%
- default52.4%Terminus 2
Benchmark compilation: Epoch AI, Capabilities & benchmarking (CC BY attribution). Scores are converted to percentages and model names matched to the Vertical Tabs catalog. Individual evaluations are credited to their original sources. External results may carry separate licences and evaluation configurations; do not interpret scores from different harnesses as directly comparable.
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.