- input
- $2.50
- cached
- $0.25
- output
- $15.00
- Context
- 1.1Mtokens
- Released
- 8mo ago
Cost/1M tokens
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
#22 of 28
- 74.1%
- 73.8%
- 73.6%
- 72.7%
- 69.9%
- 69.6%
- 69.0%
- 68.5%
- 67.5%
- 67.2%
- 51.8%GPT-5.4
Arena Text
Best Arena rating per model · Higher is better
Arena #35
- 1487Kimi K3Similar price
- 1487Gemini 3.1 Pro PreviewSimilar price
- 1478Gemini 3.5 FlashNearby rank
- 1476GPT-5.2 ChatNearby rank
- 1475GPT-5.4
- 1475GLM 5.2Nearby rank
- 1475
DeepSeek V4.1 FlashNearby rank - 1472Claude Sonnet 4.6Similar price
- 1437GPT-5.2Predecessor
- 1354
Command ASimilar price
+38 vs GPT-5.2 (predecessor) · Ahead of 1 of 4 similarly priced models
Arena WebDev
Best Arena rating per model · Higher is better
Arena #80
- 1658Kimi K3Similar price
- 1649
Muse WonderSimilar price - 1522Claude Sonnet 4.6Similar price
- 1479
MiMo-V2.5-ProNearby rank - 1474Kimi K2.7 CodeNearby rank
- 1462GPT-5.4
- 1461
Qwen3.6 PlusNearby rank - 1448DeepSeek V4 Pro 0423Nearby rank
- 1446Gemini 3.1 Pro PreviewSimilar price
- 1420GPT-5.2Predecessor
+42 vs GPT-5.2 (predecessor) · Ahead of 0 of 4 similarly priced models
Arena Vision
Best Arena rating per model · Higher is better
Arena #23
- 1285GPT-5.5Nearby rank
- 1283GPT-6 AstraNearby rank
- 1282GPT-5.4
- 1281Gemini 3.5 FlashNearby rank
- 1280Grok 4.5Similar price
- 1280Gemini 3.1 Pro PreviewSimilar price
- 1275Claude Sonnet 4.6Similar price
- 1242GPT-5.2Predecessor
- 1199
Mistral Medium 3.5Similar price
+40 vs GPT-5.2 (predecessor) · Ahead of 1 of 4 similarly priced models
Arena Search
Best Arena rating per model · Higher is better
Not ranked on Arena Search yet
Arena Document
Best Arena rating per model · Higher is better
Arena #18
- 1478Claude Sonnet 4.6Similar price
- 1474Muse Spark 1.3Nearby rank
- 1474GPT-6 AstraNearby rank
- 1471GPT-5.4
- 1469Gemini 3.8 FlashNearby rank
- 1469Claude Sonnet 5Nearby rank
- 1466Grok 4.5Similar price
- 1459Gemini 3.1 Pro PreviewSimilar price
- 1448Claude Sonnet 4.5Similar price
- 1425GPT-5.2Predecessor
+46 vs GPT-5.2 (predecessor) · Ahead of 2 of 4 similarly priced models
Agent Arena
Best IPS score per model · Higher is better
Not ranked on Agent Arena yet
ARC-AGI-2
Best score per model · Higher is better
#23 of 85
- 77.1%Gemini 3.1 Pro PreviewSimilar price
- 75.8%Claude Opus 4.7Nearby rank
- 74.0%GPT-5.4
- 72.9%DeepSeek V4.1 FlashNearby rank
- 72.1%Claude Opus 4.8Nearby rank
- 67.1%Grok 4.6Similar price
- 60.4%Claude Sonnet 4.6Similar price
- 60.4%Kimi K3Similar price
- 52.9%GPT-5.2Predecessor
+21.0 pts vs GPT-5.2 (predecessor) · Ahead of 3 of 4 similarly priced models
FrontierMath-2025-02-28-Private
Best score per model · Higher is better
#2 of 58
- 50.0%GPT-5.4 ProNearby rank
- 47.6%GPT-5.4
- 47.2%Claude Opus 4.8Nearby rank
- 43.8%Claude Opus 4.7Nearby rank
- 40.7%GPT-5.2Predecessor
- 36.9%Gemini 3.1 Pro PreviewSimilar price
- 32.4%Claude Sonnet 4.6Similar price
- 15.2%Claude Sonnet 4.5Similar price
- 14.1%Gemini 2.5 ProSimilar price
+6.9 pts vs GPT-5.2 (predecessor) · Ahead of 4 of 4 similarly priced models
FrontierMath-Tier-4-2025-07-01-Private
Best score per model · Higher is better
#2 of 46
- 31.3%Claude Opus 4.8Nearby rank
- 27.1%GPT-5.4
- 22.9%Claude Opus 4.7Nearby rank
- 22.9%Claude Opus 4.6Nearby rank
- 18.8%GPT-5.2Predecessor
- 16.7%Gemini 3.1 Pro PreviewSimilar price
- 8.3%Claude Sonnet 4.6Similar price
- 4.2%Claude Sonnet 4.5Similar price
- 4.2%Gemini 2.5 ProSimilar price
+8.3 pts vs GPT-5.2 (predecessor) · Ahead of 4 of 4 similarly priced models
FrontierMath-Tier-4-v2-Private
Best score per model · Higher is better
#22 of 72
- 56.1%Claude Opus 4.8Nearby rank
- 56.1%GPT-6 LunaNearby rank
- 49.0%GPT-5.4
- 46.3%
Qwen3.8 MaxNearby rank - 46.3%Claude Haiku 5.5Nearby rank
- 39.0%Kimi K3Similar price
- 34.1%Qwen3.8 Max (0902)Similar price
- 31.7%GPT-5.2Predecessor
- 26.8%Gemini 3.1 Pro PreviewSimilar price
- 2.4%Claude Sonnet 4.5Similar price
+17.3 pts vs GPT-5.2 (predecessor) · Ahead of 4 of 4 similarly priced models
FrontierMath-Tiers-1-3-v2-Private
Best score per model · Higher is better
#21 of 87
- 80.0%Claude Opus 4.8Nearby rank
- 78.9%GPT-6 LunaNearby rank
- 78.6%GPT-5.4
- 75.1%Claude Haiku 5.5Nearby rank
- 74.7%Qwen3.8 MaxNearby rank
- 72.2%Kimi K3Similar price
- 67.4%GPT-5.2Predecessor
- 66.0%Grok 4.6Similar price
- 59.6%Gemini 3.1 Pro PreviewSimilar price
- 23.9%Claude Sonnet 4.5Similar price
+11.2 pts vs GPT-5.2 (predecessor) · Ahead of 4 of 4 similarly priced models
GPQA diamond
Best score per model · Higher is better
#19 of 161
- 94.4%Gemini 3.1 Pro PreviewSimilar price
- 94.0%Grok 4.6Similar price
- 93.4%Grok 4.5Nearby rank
- 93.3%GPT-5.6 TerraNearby rank
- 93.3%GPT-5.4
- 93.2%DeepSeek V4.1 Flash 1017Nearby rank
- 93.1%Kimi K3Similar price
- 91.4%GPT-5.2Predecessor
- 87.4%Claude Sonnet 4.6Similar price
+1.9 pts vs GPT-5.2 (predecessor) · Ahead of 2 of 4 similarly priced models
Humanity's Last Exam
Best score per model · Higher is better
#8 of 34
- 46.4%Gemini 3.1 Pro PreviewSimilar price
- 46.1%Claude Sonnet 5.5Similar price
- 44.3%GPT-5.4 ProNearby rank
- 37.5%Gemini 3 Pro PreviewNearby rank
- 36.2%GPT-5.4
- 36.2%Claude Opus 4.7Nearby rank
- 34.4%Claude Opus 4.6Nearby rank
- 27.8%GPT-5.2Predecessor
- 21.6%Gemini 2.5 Pro Preview 06-05Similar price
- 7.5%Claude Sonnet 4.5Similar price
+8.4 pts vs GPT-5.2 (predecessor) · Ahead of 2 of 4 similarly priced models
OTIS Mock AIME 2024-2025
Best score per model · Higher is better
#30 of 158
- 100.0%Qwen3.8 Max (0902)Similar price
- 98.1%Grok 4.7Nearby rank
- 97.8%Claude Opus 4.7Nearby rank
- 97.8%GPT-5.4
- 97.8%Grok 4.5Nearby rank
- 97.2%Gemini 3.7 FlashNearby rank
- 97.2%Kimi K3Similar price
- 96.1%GPT-5.2Predecessor
- 95.6%Gemini 3.1 Pro PreviewSimilar price
- 85.8%Claude Sonnet 4.6Similar price
+1.7 pts vs GPT-5.2 (predecessor) · Ahead of 3 of 4 similarly priced models
SimpleQA Verified
Best score per model · Higher is better
#44 of 87
- 73.5%Gemini 3.1 Pro PreviewSimilar price
- 56.0%Grok 4.7Similar price
- 50.6%Kimi K3Similar price
- 45.8%Qwen3.8 MaxNearby rank
- 45.7%Claude Opus 4.5Nearby rank
- 45.1%GPT-5.4
- 44.1%Qwen3.6 PlusNearby rank
- 43.2%GPT-5.6 TerraNearby rank
- 37.1%GPT-5.2Predecessor
- 35.5%Claude Sonnet 4.6Similar price
+8.0 pts vs GPT-5.2 (predecessor) · Ahead of 1 of 4 similarly priced models
SWE-bench Verified
Best score per model · Higher is better
#6 of 35
- 77.6%DeepSeek V4 Pro 0423Nearby rank
- 77.3%Qwen3.7 MaxNearby rank
- 76.9%GPT-5.4
- 76.7%Qwen3.6 Max PreviewNearby rank
- 76.7%Claude Opus 4.5Nearby rank
- 75.6%Gemini 3.1 Pro Preview Custom ToolsSimilar price
- 75.2%Claude Sonnet 4.6Similar price
- 73.8%GPT-5.2Predecessor
- 71.3%Claude Sonnet 4.5Similar price
- 57.6%Gemini 2.5 ProSimilar price
+3.1 pts vs GPT-5.2 (predecessor) · Ahead of 4 of 4 similarly priced models
Terminal Bench
Best score per model · Higher is better
#2 of 49
- 84.7%GPT-5.5Nearby rank
- 81.8%GPT-5.4
- 80.2%Claude Opus 4.7Nearby rank
- 80.2%Gemini 3.1 Pro PreviewSimilar price
- 64.9%GPT-5.2Predecessor
- 53.4%Claude Sonnet 4.6Similar price
- 46.5%Claude Sonnet 4.5Similar price
- 32.6%Gemini 2.5 ProSimilar price
+16.9 pts vs GPT-5.2 (predecessor) · Ahead of 4 of 4 similarly priced models
DeepSWE 1.1
Primary coding-agent benchmark · 113 tasks · Data generated 2026-09-22
- pass@4
- 77.9%
- Average cost / attempt
- $5.65
- Benchmark runs
- 4
pass@1 is the success rate across scored attempts; pass@4 is the share of tasks solved in at least one of four attempts. Costs are reported by DeepSWE for each configuration. These results are not directly comparable to evaluations using different harnesses. Official source JSON.
Reasoning effort
Arena WebDev
Score by reasoning effort · Higher is better
ARC-AGI-2
Score by reasoning effort · Higher is better
GPQA diamond
Score by reasoning effort · Higher is better
OTIS Mock AIME 2024-2025
Score by reasoning effort · Higher is better
All published scores
- Arena TextArena #351475
+38 vs GPT-5.2 (predecessor)
- high147595% CI 1471 to 1479 · 64,103 votes
- default146595% CI 1461 to 1469 · 67,105 votes
- Arena WebDevArena #801462
+42 vs GPT-5.2 (predecessor)
- medium1442Codex Harness95% CI 1424 to 1460 · 1,363 votes
- high1462Codex Harness95% CI 1443 to 1481 · 1,338 votes
- default139995% CI 1390 to 1409 · 4,659 votes
- Arena VisionArena #231282
+40 vs GPT-5.2 (predecessor)
- high128295% CI 1277 to 1288 · 32,045 votes
- default127895% CI 1272 to 1285 · 22,608 votes
- Arena DocumentArena #181471
+46 vs GPT-5.2 (predecessor)
- default147195% CI 1465 to 1478 · 35,183 votes
- ARC-AGI-2#23 of 8574.0%
+21.0 pts vs GPT-5.2 (predecessor)
- low29.2%
- medium55.4%
- high67.5%
- xhigh74.0%
- FrontierMath-2025-02-28-Private#2 of 5847.6%
+6.9 pts vs GPT-5.2 (predecessor)
- xhigh47.6%
Source:Epoch AI evaluation - FrontierMath-Tier-4-2025-07-01-Private#2 of 4627.1%
+8.3 pts vs GPT-5.2 (predecessor)
- xhigh27.1%
Source:Epoch AI evaluation - FrontierMath-Tier-4-v2-Private#22 of 7249.0%
+17.3 pts vs GPT-5.2 (predecessor)
- xhigh49.0%
Source:Epoch AI evaluation - FrontierMath-Tiers-1-3-v2-Private#21 of 8778.6%
+11.2 pts vs GPT-5.2 (predecessor)
- xhigh78.6%
Source:Epoch AI evaluation - GPQA diamond#19 of 16193.3%
+1.9 pts vs GPT-5.2 (predecessor)
- none74.7%
- low84.8%
- medium88.9%
- high89.9%
- xhigh93.3%
Source:Epoch AI evaluation - Humanity's Last Exam#8 of 3436.2%
+8.4 pts vs GPT-5.2 (predecessor)
- xhigh36.2%
- OTIS Mock AIME 2024-2025#30 of 15897.8%
+1.7 pts vs GPT-5.2 (predecessor)
- none57.8%
- low84.4%
- medium95.6%
- high97.8%
- xhigh95.3%
Source:Epoch AI evaluation - SimpleQA Verified#44 of 8745.1%
+8.0 pts vs GPT-5.2 (predecessor)
- xhigh45.1%
Source:Epoch AI evaluation - SWE-bench Verified#6 of 3576.9%
+3.1 pts vs GPT-5.2 (predecessor)
- high76.9%
Source:Epoch AI evaluation - Terminal Bench#2 of 4981.8%
+16.9 pts vs GPT-5.2 (predecessor)
- unknown81.8%ForgeCode
Benchmark compilation: Epoch AI, Capabilities & benchmarking (CC BY attribution). Scores are converted to percentages and model names matched to the Vertical Tabs catalog. Individual evaluations are credited to their original sources. External results may carry separate licences and evaluation configurations; do not interpret scores from different harnesses as directly comparable.
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.