- Context
- 1Mtokens
- Released
- 8mo ago
Cost/1M tokens
Benchmarks
DeepSWE is the primary coding benchmark where an exact model evaluation exists. Arena leaderboards and other sourced evaluations appear below; missing scores are not zero.
How it ranks
Against its predecessor, similarly priced models and the models ranked around it.
DeepSWE 1.1
Best pass@1 per model · Higher is better
#28 of 28
- 74.1%
- 73.8%
- 73.6%
- 72.7%
- 69.9%
- 69.6%
- 69.0%
- 68.5%
- 67.5%
- 67.2%
- 11.7%Gemini 3.1 Pro Preview
Arena Text
Best Arena rating per model · Higher is better
Arena #20
- 1489Gemini 3.7 FlashNearby rank
- 1487Kimi K3Similar price
- 1487Gemini 3.1 Pro Preview
- 1486Gemini 3 Pro PreviewPredecessor
- 1485GPT-5.6 SolNearby rank
- 1472Claude Sonnet 5.5Similar price
- 1468GPT-5.6 TerraSimilar price
- 1354
Command ASimilar price
Level with Gemini 3 Pro Preview (predecessor, overlapping 95% CIs) · Ahead of 3 of 4 similarly priced models
Arena WebDev
Best Arena rating per model · Higher is better
Arena #84
- 1773Claude Sonnet 5.5Similar price
- 1676
Qwen3.8 Max (0902)Similar price - 1658Kimi K3Similar price
- 1520GPT-5.6 TerraSimilar price
- 1461Qwen3.6 PlusNearby rank
- 1448
DeepSeek V4 Pro 0423Nearby rank - 1446Gemini 3.1 Pro Preview
- 1441Gemini 3 Pro PreviewPredecessor
- 1438
MiMo-V2.5Nearby rank
Level with Gemini 3 Pro Preview (predecessor, overlapping 95% CIs) · Ahead of 0 of 4 similarly priced models
Arena Vision
Best Arena rating per model · Higher is better
Arena #27
- 1289Gemini 3 Pro PreviewPredecessor
- 1281Gemini 3.5 FlashNearby rank
- 1280Grok 4.5Similar price
- 1280Gemini 3.1 Pro Preview
- 1280Gemini 3.6 FlashNearby rank
- 1279
Muse Spark 1.1Nearby rank - 1266GPT-5.6 TerraSimilar price
- 1263Claude Sonnet 5.5Similar price
- 1199
Mistral Medium 3.5Similar price
Level with Gemini 3 Pro Preview (predecessor, overlapping 95% CIs) · Ahead of 3 of 4 similarly priced models
Arena Search
Best Arena rating per model · Higher is better
Not ranked on Arena Search yet
Arena Document
Best Arena rating per model · Higher is better
Arena #27
- 1474GPT-5.6 TerraSimilar price
- 1469Claude Sonnet 5Similar price
- 1466Grok 4.5Similar price
- 1460GPT-5.6 LunaNearby rank
- 1460GPT-5.5 InstantNearby rank
- 1459Gemini 3.1 Pro Preview
- 1459Gemini 3.5 FlashNearby rank
- 1454Gemini 3.6 FlashNearby rank
- 1451Gemini 3 Pro PreviewPredecessor
- 1425GPT-5.2Similar price
Level with Gemini 3 Pro Preview (predecessor, overlapping 95% CIs) · Ahead of 1 of 4 similarly priced models
Agent Arena
Best IPS score per model · Higher is better
Arena #41
- +0.160Claude Sonnet 5.5Similar price
- +0.026Grok 4.7Similar price
- +0.006Kimi K3Similar price
- -0.018GPT-5.6 TerraSimilar price
- -0.029GLM 5.3 FlashNearby rank
- -0.037
Step 5 PreviewNearby rank - -0.045Gemini 3.1 Pro Preview
- -0.046
Qwen3.8 Flash NextNearby rank - -0.048Qwen3.8 27BNearby rank
Ahead of 0 of 4 similarly priced models
ARC-AGI-2
Best score per model · Higher is better
#21 of 85
- 83.9%GPT-5.6 TerraSimilar price
- 83.9%Claude Sonnet 5.5Similar price
- 81.7%
Inkling 2 SmallNearby rank - 78.1%DeepSeek V4.1 Flash 1017Nearby rank
- 77.1%Gemini 3.1 Pro Preview
- 75.8%Claude Opus 4.7Nearby rank
- 74.0%GPT-5.4Nearby rank
- 67.1%Grok 4.6Similar price
- 60.4%Kimi K3Similar price
- 31.1%Gemini 3 Pro PreviewPredecessor
+46.0 pts vs Gemini 3 Pro Preview (predecessor) · Ahead of 2 of 4 similarly priced models
FrontierMath-2025-02-28-Private
Best score per model · Higher is better
#10 of 58
- 40.7%GPT-5.2Similar price
- 39.0%Kimi K2.6Nearby rank
- 37.6%Gemini 3 Pro PreviewPredecessor
- 36.9%Gemini 3.1 Pro Preview
- 35.6%Gemini 3 Flash PreviewNearby rank
- 33.4%GLM 5.1Nearby rank
- 32.4%Claude Sonnet 4.6Similar price
- 0.3%GPT-4o (2024-08-06)Similar price
- 0.3%GPT-4o (2024-11-20)Similar price
-0.7 pts vs Gemini 3 Pro Preview (predecessor) · Ahead of 3 of 4 similarly priced models
FrontierMath-Tier-4-2025-07-01-Private
Best score per model · Higher is better
#7 of 46
- 27.1%GPT-5.4Similar price
- 18.8%GPT-5.2Similar price
- 18.8%Gemini 3 Pro PreviewPredecessor
- 16.7%Gemini 3.1 Pro Preview
- 14.6%GPT-5 ProNearby rank
- 14.6%Gemini 3.5 FlashNearby rank
- 8.3%Claude Sonnet 4.6Similar price
- 2.1%o3Similar price
-2.1 pts vs Gemini 3 Pro Preview (predecessor) · Ahead of 2 of 4 similarly priced models
FrontierMath-Tier-4-v2-Private
Best score per model · Higher is better
#38 of 72
- 80.5%Claude Sonnet 5.5Similar price
- 70.7%GPT-5.6 TerraSimilar price
- 39.0%Kimi K3Similar price
- 34.1%Qwen3.8 Max (0902)Similar price
- 26.8%DeepSeek V4 Pro 0813Nearby rank
- 26.8%DeepSeek V4.1 FlashNearby rank
- 26.8%Gemini 3.1 Pro Preview
- 26.8%Gemini 3.5 FlashNearby rank
- 26.8%Mistral Large 4Nearby rank
- 0.0%Gemini 2.5 ProEarlier version
+26.8 pts vs Gemini 2.5 Pro (earlier version) · Ahead of 0 of 4 similarly priced models
FrontierMath-Tiers-1-3-v2-Private
Best score per model · Higher is better
#41 of 87
- 88.8%Claude Sonnet 5.5Similar price
- 86.0%GPT-5.6 TerraSimilar price
- 72.2%Kimi K3Similar price
- 66.0%Grok 4.6Similar price
- 64.6%Qwen3.7 MaxNearby rank
- 62.8%Gemini 3.5 FlashNearby rank
- 59.6%Gemini 3.1 Pro Preview
- 59.2%GLM 5.2Nearby rank
- 58.9%Gemini 3.6 FlashNearby rank
- 24.6%Gemini 2.5 ProEarlier version
+35.1 pts vs Gemini 2.5 Pro (earlier version) · Ahead of 0 of 4 similarly priced models
GPQA diamond
Best score per model · Higher is better
#9 of 161
- 95.6%Claude Sonnet 5.5Similar price
- 94.8%Gemini 3.7 FlashNearby rank
- 94.6%GPT-5.4 ProNearby rank
- 94.4%Gemini 3.1 Pro Preview
- 94.3%GPT-6 SolNearby rank
- 94.3%GPT-6.1 TerraNearby rank
- 94.0%Grok 4.6Similar price
- 93.3%GPT-5.6 TerraSimilar price
- 93.1%Kimi K3Similar price
- 92.6%Gemini 3 Pro PreviewPredecessor
+1.8 pts vs Gemini 3 Pro Preview (predecessor) · Ahead of 3 of 4 similarly priced models
Humanity's Last Exam
Best score per model · Higher is better
#3 of 34
- 54.8%GPT-6 AstraNearby rank
- 46.5%Claude Fable 5.1Nearby rank
- 46.4%Gemini 3.1 Pro Preview
- 46.1%Claude Sonnet 5.5Similar price
- 44.5%Gemini 3.8 FlashNearby rank
- 37.5%Gemini 3 Pro PreviewPredecessor
- 36.2%GPT-5.4Similar price
- 27.8%GPT-5.2Similar price
- 2.7%GPT-4o (2024-11-20)Similar price
+8.9 pts vs Gemini 3 Pro Preview (predecessor) · Ahead of 4 of 4 similarly priced models
OTIS Mock AIME 2024-2025
Best score per model · Higher is better
#37 of 158
- 100.0%Claude Sonnet 5.5Similar price
- 100.0%Qwen3.8 Max (0902)Similar price
- 99.7%GPT-5.6 TerraSimilar price
- 97.2%Kimi K3Similar price
- 96.1%Kimi K2.6Nearby rank
- 96.1%GPT-5.2Nearby rank
- 95.6%Gemini 3.1 Pro Preview
- 95.6%Gemini 3 Flash PreviewNearby rank
- 95.6%Gemini 3.5 FlashNearby rank
- 91.4%Gemini 3 Pro PreviewPredecessor
+4.2 pts vs Gemini 3 Pro Preview (predecessor) · Ahead of 0 of 4 similarly priced models
SimpleQA Verified
Best score per model · Higher is better
#6 of 87
- 75.6%GPT-6 AstraNearby rank
- 73.9%GPT-6.1 SolNearby rank
- 73.5%Gemini 3.1 Pro Preview
- 72.2%Claude Opus 5.5Nearby rank
- 70.8%Claude Fable 5.1Nearby rank
- 66.8%Gemini 3 Flash PreviewEarlier version
- 56.0%Grok 4.7Similar price
- 50.6%Kimi K3Similar price
- 46.5%Claude Sonnet 5.5Similar price
- 43.2%GPT-5.6 TerraSimilar price
+6.7 pts vs Gemini 3 Flash Preview (earlier version) · Ahead of 4 of 4 similarly priced models
Terminal Bench
Best score per model · Higher is better
#3 of 49
- 81.8%GPT-5.4Nearby rank
- 80.2%Claude Opus 4.7Nearby rank
- 80.2%Gemini 3.1 Pro Preview
- 79.8%Claude Opus 4.6Nearby rank
- 78.4%GPT-5.3-CodexSimilar price
- 69.4%Gemini 3 Pro PreviewPredecessor
- 66.5%GPT-5.2-CodexSimilar price
- 64.9%GPT-5.2Similar price
- 53.4%Claude Sonnet 4.6Similar price
+10.8 pts vs Gemini 3 Pro Preview (predecessor) · Ahead of 4 of 4 similarly priced models
DeepSWE 1.1
Primary coding-agent benchmark · 113 tasks · Data generated 2026-09-22
- pass@4
- 28.3%
- Average cost / attempt
- $2.14
- Benchmark runs
- 4
pass@1 is the success rate across scored attempts; pass@4 is the share of tasks solved in at least one of four attempts. Costs are reported by DeepSWE for each configuration. These results are not directly comparable to evaluations using different harnesses. Official source JSON.
All published scores
- Arena TextArena #201487
Level with Gemini 3 Pro Preview (predecessor, overlapping 95% CIs)
- default148795% CI 1484 to 1490 · 132,296 votes
- Arena WebDevArena #841446
Level with Gemini 3 Pro Preview (predecessor, overlapping 95% CIs)
- default144695% CI 1441 to 1452 · 23,846 votes
- Arena VisionArena #271280
Level with Gemini 3 Pro Preview (predecessor, overlapping 95% CIs)
- default128095% CI 1275 to 1285 · 48,242 votes
- Arena DocumentArena #271459
Level with Gemini 3 Pro Preview (predecessor, overlapping 95% CIs)
- default145995% CI 1454 to 1464 · 52,244 votes
- Agent ArenaArena #41-0.045
- default-0.04595% CI -0.060 to -0.030 · 2,967,695 observations
- ARC-AGI-2#21 of 8577.1%
+46.0 pts vs Gemini 3 Pro Preview (predecessor)
- default77.1%
- FrontierMath-2025-02-28-Private#10 of 5836.9%
-0.7 pts vs Gemini 3 Pro Preview (predecessor)
- default36.9%
Source:Epoch AI evaluation - FrontierMath-Tier-4-2025-07-01-Private#7 of 4616.7%
-2.1 pts vs Gemini 3 Pro Preview (predecessor)
- default16.7%
Source:Epoch AI evaluation - FrontierMath-Tier-4-v2-Private#38 of 7226.8%
+26.8 pts vs Gemini 2.5 Pro (earlier version)
- default26.8%
Source:Epoch AI evaluation - FrontierMath-Tiers-1-3-v2-Private#41 of 8759.6%
+35.1 pts vs Gemini 2.5 Pro (earlier version)
- default59.6%
Source:Epoch AI evaluation - GPQA diamond#9 of 16194.4%
+1.8 pts vs Gemini 3 Pro Preview (predecessor)
- high94.4%
- default94.1%
Source:Epoch AI evaluation - Humanity's Last Exam#3 of 3446.4%
+8.9 pts vs Gemini 3 Pro Preview (predecessor)
- default46.4%
- OTIS Mock AIME 2024-2025#37 of 15895.6%
+4.2 pts vs Gemini 3 Pro Preview (predecessor)
- high95.6%
- default95.6%
Source:Epoch AI evaluation - SimpleQA Verified#6 of 8773.5%
+6.7 pts vs Gemini 3 Flash Preview (earlier version)
- high73.5%
Source:Epoch AI evaluation - Terminal Bench#3 of 4980.2%
+10.8 pts vs Gemini 3 Pro Preview (predecessor)
- default80.2%TongAgentshttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- default78.4%Forge CodeTerminal-Bench v2 LeaderboardOriginal-source terms apply
- default78.4%ForgeCodehttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- default74.8%Terminus-KIRAhttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
- default61.4%Gemini CLIhttps://www.tbench.ai/leaderboard/terminal-bench/2.0Original-source terms apply
Benchmark compilation: Epoch AI, Capabilities & benchmarking (CC BY attribution). Scores are converted to percentages and model names matched to the Vertical Tabs catalog. Individual evaluations are credited to their original sources. External results may carry separate licences and evaluation configurations; do not interpret scores from different harnesses as directly comparable.
Arena ratings: Arena (lmarena-ai/leaderboard-dataset), licensed CC BY 4.0. Ratings come from pairwise human votes and are only comparable within one leaderboard; the interval is Arena's 95% confidence interval. Model names are matched to the Vertical Tabs catalog.