- Context
- 200Ktokens
- Released
- 1y ago
Cost/1M tokens
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
Not evaluated in DeepSWE 1.1
Arena Text
Best Arena rating per model · Higher is better
Arena #80
- 1504
Claude Fable 5Similar price - 1503
Claude Fable 5.1Similar price - 1474
GPT-6 AstraSimilar price - 1453
GPT-5.6 LunaNearby rank - 1452
Gemma 4 31BNearby rank - 1450
Claude Opus 4.1 - 1450
Kimi K2.5Nearby rank - 1448
Step 5 PreviewNearby rank - 1426
Claude Opus 4Predecessor
+24 vs Claude Opus 4 (predecessor) · Ahead of 0 of 3 similarly priced models
Arena WebDev
Best Arena rating per model · Higher is better
Arena #110
- 1836
Claude Fable 5.2Similar price - 1764
GPT-6 AstraSimilar price - 1742
Claude Fable 5.1Similar price - 1625
Claude Fable 5Similar price - 1397
GPT-5.1Nearby rank - 1393
Claude Sonnet 4.5Nearby rank - 1389
Claude Opus 4.1 - 1386MiniMax M2.5Nearby rank
- 1370
Gemma 4 31BNearby rank
Ahead of 0 of 4 similarly priced models
Arena Vision
Best Arena rating per model · Higher is better
Not ranked on Arena Vision yet
Arena Search
Best Arena rating per model · Higher is better
Not ranked on Arena Search yet
Arena Document
Best Arena rating per model · Higher is better
Not ranked on Arena Document yet
Agent Arena
Best IPS score per model · Higher is better
Not ranked on Agent Arena yet
FrontierMath-2025-02-28-Private
Best score per model · Higher is better
#35 of 58
- 8.5%
Qwen3 235B A22B Thinking 2507Nearby rank - 8.3%
GPT-5 NanoNearby rank - 7.2%
Claude Opus 4.1 - 6.2%
Qwen3.5-FlashNearby rank - 5.9%
Claude Haiku 4.5Nearby rank - 4.5%
Claude Opus 4Predecessor
+2.8 pts vs Claude Opus 4 (predecessor)
FrontierMath-Tier-4-2025-07-01-Private
Best score per model · Higher is better
#18 of 46
- 14.6%
GPT-5 ProSimilar price - 4.2%Qwen3.6 Max PreviewNearby rank
- 4.2%
Claude Opus 4Predecessor - 4.2%
Claude Opus 4.1 - 4.2%
Claude Opus 4.5Nearby rank - 4.2%
Claude Sonnet 4.5Nearby rank
Level with Claude Opus 4 (predecessor) · Ahead of 0 of 1 similarly priced model
FrontierMath-Tier-4-v2-Private
Best score per model · Higher is better
#64 of 72
- 97.6%
GPT-6 AstraSimilar price - 90.2%
Claude Fable 5Similar price - 46.0%
GPT-5.2 ProSimilar price - 19.5%
GPT-5 ProSimilar price - 4.9%
Claude Opus 4.5Nearby rank - 4.9%
InklingNearby rank - 2.4%
Claude Opus 4.1 - 2.4%
Claude Sonnet 4.5Nearby rank - 2.4%
DeepSeek V4 Pro 0423Nearby rank
Ahead of 0 of 4 similarly priced models
FrontierMath-Tiers-1-3-v2-Private
Best score per model · Higher is better
#82 of 87
- 93.7%
GPT-6 AstraSimilar price - 90.2%
Claude Fable 5.1Similar price - 74.0%
GPT-5.2 ProSimilar price - 55.8%
GPT-5 ProSimilar price - 18.9%Qwen3 MaxNearby rank
- 18.2%
Qwen3.5-FlashNearby rank - 12.6%
Claude Opus 4.1 - 6.7%
GPT-4.1 MiniNearby rank - 6.0%
GPT-4.1Nearby rank
Ahead of 0 of 4 similarly priced models
GPQA diamond
Best score per model · Higher is better
#94 of 161
- 95.8%
GPT-6 AstraSimilar price - 93.9%
Claude Fable 5.2Similar price - 85.9%
Claude Fable 5Similar price - 78.5%
GPT-5.4 NanoNearby rank - 77.6%
Seed 1.6Nearby rank - 77.3%
Claude Opus 4.1 - 76.3%
R1 0528Nearby rank - 76.3%
Claude Opus 4Predecessor
+1.0 pts vs Claude Opus 4 (predecessor) · Ahead of 0 of 3 similarly priced models
Humanity's Last Exam
Best score per model · Higher is better
#26 of 34
- 54.8%
GPT-6 AstraSimilar price - 46.5%
Claude Fable 5.1Similar price - 31.6%
GPT-5 ProSimilar price - 8.1%
GLM 4.5 AirNearby rank - 8.0%
Claude Sonnet 3.7Nearby rank - 7.9%
Claude Opus 4.1 - 7.5%
Claude Sonnet 4.5Nearby rank - 6.7%
Claude Opus 4Predecessor
+1.2 pts vs Claude Opus 4 (predecessor) · Ahead of 0 of 3 similarly priced models
OTIS Mock AIME 2024-2025
Best score per model · Higher is better
#99 of 158
- 100.0%
Claude Fable 5Similar price - 100.0%
Claude Fable 5.1Similar price - 100.0%
GPT-6 AstraSimilar price - 99.7%
Claude Fable 5.2Similar price - 70.0%
Qwen3.5-35B-A3BNearby rank - 69.4%
Seed 1.6Nearby rank - 68.9%
Claude Opus 4.1 - 68.1%
GPT-5.5 InstantNearby rank - 67.5%
Seed-OSS-36B-InstructNearby rank - 64.4%
Claude Opus 4Predecessor
+4.4 pts vs Claude Opus 4 (predecessor) · Ahead of 0 of 4 similarly priced models
SWE-bench Verified
Best score per model · Higher is better
#21 of 35
- 73.8%
GPT-5.2Nearby rank - 73.6%
GPT-5Nearby rank - 73.3%
Claude Opus 4.1 - 72.9%
Gemini 3 Pro PreviewNearby rank - 72.1%
GLM 5Nearby rank - 70.7%
Claude Opus 4Predecessor
+2.7 pts vs Claude Opus 4 (predecessor)
Terminal Bench
Best score per model · Higher is better
#28 of 49
- 42.7%MiniMax M2.5Nearby rank
- 39.6%
DeepSeek V3.2Nearby rank - 38.0%
Claude Opus 4.1 - 36.6%MiniMax-M2.1Nearby rank
- 35.7%
Kimi K2 ThinkingNearby rank
Not evaluated in DeepSWE 1.1 as of 2026-09-22. Results are only shown for exact model versions in the official dataset. Official leaderboard.
All published scores
- Arena TextArena #801450
+24 vs Claude Opus 4 (predecessor)
- default145016K, thinking95% CI 1446 to 1453 · 49,374 votes
- default144895% CI 1445 to 1451 · 76,931 votes
- Arena WebDevArena #1101389
- default138995% CI 1379 to 1400 · 6,818 votes
- FrontierMath-2025-02-28-Private#35 of 587.2%
+2.8 pts vs Claude Opus 4 (predecessor)
- default7.2%27K
- default5.9%
Source:Epoch AI evaluation - FrontierMath-Tier-4-2025-07-01-Private#18 of 464.2%
Level with Claude Opus 4 (predecessor)
- default4.2%27K
Source:Epoch AI evaluation - FrontierMath-Tier-4-v2-Private#64 of 722.4%
- default2.4%32K
Source:Epoch AI evaluation - FrontierMath-Tiers-1-3-v2-Private#82 of 8712.6%
- default12.6%32K
Source:Epoch AI evaluation - GPQA diamond#94 of 16177.3%
+1.0 pts vs Claude Opus 4 (predecessor)
- default77.3%16K
- default76.8%27K
- default73.2%
Source:Epoch AI evaluation - Humanity's Last Exam#26 of 347.9%
+1.2 pts vs Claude Opus 4 (predecessor)
- unknown7.9%
- OTIS Mock AIME 2024-2025#99 of 15868.9%
+4.4 pts vs Claude Opus 4 (predecessor)
- default68.9%27K
- default64.4%16K
- default40.0%
Source:Epoch AI evaluation - SWE-bench Verified#21 of 3573.3%
+2.7 pts vs Claude Opus 4 (predecessor)
- default73.3%
Source:Epoch AI evaluation - Terminal Bench#28 of 4938.0%
- default38.0%Terminus 2Terminal-Bench v2 LeaderboardOriginal-source terms apply
- unknown38.0%Terminus 2https://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- default36.9%OpenHandsTerminal-Bench v2 LeaderboardOriginal-source terms apply
- unknown36.9%OpenHandshttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- default35.1%Mini-SWE-AgentTerminal-Bench v2 LeaderboardOriginal-source terms apply
- unknown35.1%Mini-SWE-Agenthttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- unknown34.8%Claude Codehttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- default34.8%Claude CodeTerminal-Bench v2 LeaderboardOriginal-source terms apply
Benchmark compilation: Epoch AI, Capabilities & benchmarking (CC BY attribution). Scores are converted to percentages and model names matched to the Vertical Tabs catalog. Individual evaluations are credited to their original sources. External results may carry separate licences and evaluation configurations; do not interpret scores from different harnesses as directly comparable.
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.