- Context
- 200Ktokens
- Released
- 11mo ago
Cost/1M tokens
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
Not evaluated in DeepSWE 1.1
Arena Text
Best Arena rating per model · Higher is better
Arena #43
- 1511
Muse WonderSimilar price - 1485
GPT-5.6 SolSimilar price - 1481
GPT-5.5Similar price - 1474
GLM 5.3 FlashNearby rank - 1474
GPT-6 AstraNearby rank - 1474
Claude Opus 4.5 - 1473
GPT-5.5 InstantNearby rank - 1472
Claude Sonnet 5.5Nearby rank - 1450
Claude Opus 4.1Predecessor - 1346
GPT-4o (2024-05-13)Similar price
+24 vs Claude Opus 4.1 (predecessor) · Ahead of 1 of 4 similarly priced models
Arena WebDev
Best Arena rating per model · Higher is better
Arena #70
- 1649
Muse WonderSimilar price - 1615
GPT-5.6 SolSimilar price - 1512
GPT-5.5Similar price - 1506
Hy3Nearby rank - 1500
Gemini 3.5 FlashNearby rank - 1493
Claude Opus 4.5 - 1484MiniMax M3Nearby rank
- 1482Qwen3.6 Max PreviewNearby rank
- 1462
GPT-5.4Similar price - 1389
Claude Opus 4.1Predecessor
+104 vs Claude Opus 4.1 (predecessor) · Ahead of 1 of 4 similarly priced models
Arena Vision
Best Arena rating per model · Higher is better
Not ranked on Arena Vision yet
Arena Search
Best Arena rating per model · Higher is better
Not ranked on Arena Search yet
Arena Document
Best Arena rating per model · Higher is better
Arena #22
- 1486
GPT-5.6 SolSimilar price - 1481
GPT-5.5Similar price - 1474
GPT-6 AstraSimilar price - 1471
GPT-5.4Similar price - 1469
Claude Sonnet 5Nearby rank - 1468
Gemini 3.7 FlashNearby rank - 1467
Claude Opus 4.5 - 1466
Grok 4.5Nearby rank - 1460
GPT-5.6 LunaNearby rank
Ahead of 0 of 4 similarly priced models
Agent Arena
Best IPS score per model · Higher is better
Not ranked on Agent Arena yet
ARC-AGI-2
Best score per model · Higher is better
#45 of 85
- 95.0%
GPT-6 AstraSimilar price - 92.5%
GPT-5.6 SolSimilar price - 85.0%
GPT-5.5Similar price - 74.0%
GPT-5.4Similar price - 42.4%
Qwen3.8 27BNearby rank - 40.1%
Inkling SmallNearby rank - 37.6%
Claude Opus 4.5 - 36.5%
InklingNearby rank - 31.1%
Gemini 3 Pro PreviewNearby rank - 8.6%
Claude Opus 4Earlier version
+29.0 pts vs Claude Opus 4 (earlier version) · Ahead of 0 of 4 similarly priced models
FrontierMath-2025-02-28-Private
Best score per model · Higher is better
#24 of 58
- 47.6%
GPT-5.4Similar price - 47.2%
Claude Opus 4.8Similar price - 43.8%
Claude Opus 4.7Similar price - 40.7%
Claude Opus 4.6Similar price - 21.4%
Kimi K2 ThinkingNearby rank - 21.0%Qwen3.5 PlusNearby rank
- 20.7%
Claude Opus 4.5 - 18.7%
o3Nearby rank - 16.4%
GLM 5Nearby rank - 7.2%
Claude Opus 4.1Predecessor
+13.4 pts vs Claude Opus 4.1 (predecessor) · Ahead of 0 of 4 similarly priced models
FrontierMath-Tier-4-2025-07-01-Private
Best score per model · Higher is better
#18 of 46
- 31.3%
Claude Opus 4.8Similar price - 27.1%
GPT-5.4Similar price - 22.9%
Claude Opus 4.7Similar price - 22.9%
Claude Opus 4.6Similar price - 4.2%
Claude Opus 4Nearby rank - 4.2%
Claude Opus 4.1Predecessor - 4.2%
Claude Opus 4.5 - 4.2%
Claude Sonnet 4.5Nearby rank - 4.2%
Gemini 2.5 FlashNearby rank
Level with Claude Opus 4.1 (predecessor) · Ahead of 0 of 4 similarly priced models
FrontierMath-Tier-4-v2-Private
Best score per model · Higher is better
#61 of 72
- 97.6%
GPT-6 AstraSimilar price - 82.9%
GPT-5.6 SolSimilar price - 72.5%
GPT-5.5Similar price - 49.0%
GPT-5.4Similar price - 12.2%
GPT-5.4 NanoNearby rank - 9.8%
GPT-5.4 MiniNearby rank - 4.9%
Claude Opus 4.5 - 4.9%
InklingNearby rank - 2.4%
Claude Opus 4.1Predecessor
+2.4 pts vs Claude Opus 4.1 (predecessor) · Ahead of 0 of 4 similarly priced models
FrontierMath-Tiers-1-3-v2-Private
Best score per model · Higher is better
#64 of 87
- 93.7%
GPT-6 AstraSimilar price - 89.1%
GPT-5.6 SolSimilar price - 85.3%
GPT-5.5Similar price - 78.6%
GPT-5.4Similar price - 36.8%
GLM 5.1Nearby rank - 35.1%
Qwen3.6 27BNearby rank - 34.4%
Claude Opus 4.5 - 34.4%
Qwen3.7 PlusNearby rank - 33.3%
o3Nearby rank - 12.6%
Claude Opus 4.1Predecessor
+21.8 pts vs Claude Opus 4.1 (predecessor) · Ahead of 0 of 4 similarly priced models
GPQA diamond
Best score per model · Higher is better
#65 of 161
- 93.5%
GPT-5.6 SolSimilar price - 93.3%
GPT-5.4Similar price - 90.7%
GPT-5.5Similar price - 86.4%
Qwen3.5 397B A17BNearby rank - 86.2%
GPT-5Nearby rank - 86.0%
Claude Opus 4.5 - 85.9%
Claude Fable 5Nearby rank - 85.9%
Qwen3.6 27BNearby rank - 77.3%
Claude Opus 4.1Predecessor - 48.9%
GPT-4o (2024-05-13)Similar price
+8.8 pts vs Claude Opus 4.1 (predecessor) · Ahead of 1 of 4 similarly priced models
Humanity's Last Exam
Best score per model · Higher is better
#20 of 34
- 54.8%
GPT-6 AstraSimilar price - 36.2%
GPT-5.4Similar price - 36.2%
Claude Opus 4.7Similar price - 34.4%
Claude Opus 4.6Similar price - 20.3%
o3Nearby rank - 19.4%
GPT-5 MiniNearby rank - 14.2%
Claude Opus 4.5 - 11.0%
Gemini 2.5 Flash Preview 05-20Nearby rank - 8.6%
Gemini 3.1 Flash LiteNearby rank - 7.9%
Claude Opus 4.1Predecessor
+6.2 pts vs Claude Opus 4.1 (predecessor) · Ahead of 0 of 4 similarly priced models
OSWorld
Best score per model · Higher is better
#2 of 10
- 72.1%
Claude Sonnet 4.6Similar price - 66.3%
Claude Opus 4.5 - 63.3%
Kimi K2.5Nearby rank - 62.9%
Claude Sonnet 4.5Similar price - 43.9%
Claude Sonnet 4Earlier version
+22.4 pts vs Claude Sonnet 4 (earlier version) · Ahead of 1 of 2 similarly priced models
OTIS Mock AIME 2024-2025
Best score per model · Higher is better
#73 of 158
- 100.0%
GPT-5.6 SolSimilar price - 97.8%
GPT-5.4Similar price - 86.7%
Qwen3.7 FlashNearby rank - 86.4%
GLM 5.2Nearby rank - 86.1%
Claude Opus 4.5 - 85.8%
Claude Sonnet 4.6Nearby rank - 84.7%
Gemini 2.5 ProNearby rank - 84.4%
GPT-5.5Similar price - 68.9%
Claude Opus 4.1Predecessor - 6.3%
GPT-4o (2024-05-13)Similar price
+17.2 pts vs Claude Opus 4.1 (predecessor) · Ahead of 2 of 4 similarly priced models
SimpleQA Verified
Best score per model · Higher is better
#42 of 87
- 75.6%
GPT-6 AstraSimilar price - 69.7%
GPT-5.6 SolSimilar price - 63.0%
GPT-5.5Similar price - 46.3%
GPT-5.4 ProNearby rank - 45.8%Qwen3.8 MaxNearby rank
- 45.7%
Claude Opus 4.5 - 45.1%
GPT-5.4Similar price - 44.1%
Qwen3.6 PlusNearby rank
Ahead of 1 of 4 similarly priced models
SWE-bench Verified
Best score per model · Higher is better
#7 of 35
- 83.5%
Claude Opus 4.7Similar price - 76.9%
GPT-5.4Similar price - 76.7%Qwen3.6 Max PreviewNearby rank
- 76.7%
Claude Opus 4.5 - 76.7%
Kimi K2.6Nearby rank - 75.6%
Claude Opus 4.6Similar price - 75.2%
Claude Sonnet 4.6Similar price - 73.3%
Claude Opus 4.1Predecessor
+3.3 pts vs Claude Opus 4.1 (predecessor) · Ahead of 2 of 4 similarly priced models
Terminal Bench
Best score per model · Higher is better
#11 of 49
- 84.7%
GPT-5.5Similar price - 81.8%
GPT-5.4Similar price - 80.2%
Claude Opus 4.7Similar price - 79.8%
Claude Opus 4.6Similar price - 64.9%
GPT-5.2Nearby rank - 64.3%
Gemini 3 Flash PreviewNearby rank - 63.1%
Claude Opus 4.5 - 61.6%
GPT-5.1-Codex-MiniNearby rank - 60.4%
GPT-5.1-Codex-MaxNearby rank - 38.0%
Claude Opus 4.1Predecessor
+25.1 pts vs Claude Opus 4.1 (predecessor) · Ahead of 0 of 4 similarly priced models
Not evaluated in DeepSWE 1.1 as of 2026-09-22. Results are only shown for exact model versions in the official dataset. Official leaderboard.
All published scores
- Arena TextArena #431474
+24 vs Claude Opus 4.1 (predecessor)
- high147432K95% CI 1470 to 1478 · 37,253 votes
- default147095% CI 1467 to 1473 · 72,313 votes
- Arena WebDevArena #701493
+104 vs Claude Opus 4.1 (predecessor)
- high149332K95% CI 1485 to 1501 · 10,925 votes
- default146995% CI 1461 to 1476 · 13,303 votes
- Arena DocumentArena #221467
- default146795% CI 1457 to 1478 · 8,304 votes
- ARC-AGI-2#45 of 8537.6%
+29.0 pts vs Claude Opus 4 (earlier version)
- default37.6%64K
- default22.8%16K
- default13.9%8K
- default7.8%
- FrontierMath-2025-02-28-Private#24 of 5820.7%
+13.4 pts vs Claude Opus 4.1 (predecessor)
- default20.7%
- default20.7%32K
- default20.3%16K
Source:Epoch AI evaluation - FrontierMath-Tier-4-2025-07-01-Private#18 of 464.2%
Level with Claude Opus 4.1 (predecessor)
- default4.2%
- default4.2%32K
- default2.1%16K
Source:Epoch AI evaluation - FrontierMath-Tier-4-v2-Private#61 of 724.9%
+2.4 pts vs Claude Opus 4.1 (predecessor)
- default4.9%32K
Source:Epoch AI evaluation - FrontierMath-Tiers-1-3-v2-Private#64 of 8734.4%
+21.8 pts vs Claude Opus 4.1 (predecessor)
- default34.4%32K
Source:Epoch AI evaluation - GPQA diamond#65 of 16186.0%
+8.8 pts vs Claude Opus 4.1 (predecessor)
- default86.0%32K
- default85.5%16K
- default80.7%
Source:Epoch AI evaluation - Humanity's Last Exam#20 of 3414.2%
+6.2 pts vs Claude Opus 4.1 (predecessor)
- unknown14.2%
- OSWorld#2 of 1066.3%
+22.4 pts vs Claude Sonnet 4 (earlier version)
- default66.3%Claude Opus 4.5
- OTIS Mock AIME 2024-2025#73 of 15886.1%
+17.2 pts vs Claude Opus 4.1 (predecessor)
- default86.1%32K
- default81.7%16K
- default48.1%
Source:Epoch AI evaluation - SimpleQA Verified#42 of 8745.7%
- default45.7%32K
Source:Epoch AI evaluation - SWE-bench Verified#7 of 3576.7%
+3.3 pts vs Claude Opus 4.1 (predecessor)
- default76.7%
Source:Epoch AI evaluation - Terminal Bench#11 of 4963.1%
+25.1 pts vs Claude Opus 4.1 (predecessor)
- unknown63.1%Droidhttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- default63.1%DroidTerminal-Bench v2 LeaderboardOriginal-source terms apply
- unknown59.1%Letta Codehttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- default59.1%Letta Code, 128KTerminal-Bench v2 LeaderboardOriginal-source terms apply
- unknown58.4%Muxhttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- unknown57.8%Terminus 2https://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- unknown54.3%Goosehttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- unknown52.1%Claude Codehttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- unknown51.9%OpenHandshttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- unknown51.7%OpenCodehttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
Benchmark compilation: Epoch AI, Capabilities & benchmarking (CC BY attribution). Scores are converted to percentages and model names matched to the Vertical Tabs catalog. Individual evaluations are credited to their original sources. External results may carry separate licences and evaluation configurations; do not interpret scores from different harnesses as directly comparable.
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.