A new benchmark revision dropped this week with real, reproducible numbers on how frontier LLMs perform on actual engineering tasks — and the lessons are sharper than ever.
DataCurve released DeepSWE v1.1, an update to their coding agent benchmark that grades models on 113 long-horizon engineering tasks with commits verified in isolated containers. The methodology changed — agents now commit diffs that get tested in clean environments, the same approach SWE-bench pioneered — but the top-line story is clear: performance and cost are diverging fast, and the cheapest option is often the best.
The Leaderboard: Performance and Price
Here are the top 14 model configurations, ranked by PASS@1 (pass rate on committed code), with average cost per task in parentheses:
| Rank | Model | PASS@1 | Avg Cost |
|---|---|---|---|
| 1 | gemini-3.8-flash [high] | 74% ±1% | $2.36 |
| 2 | claude-opus-5 [max] | 74% ±4% | $11.84 |
| 3 | gpt-5.6-sol [max] | 73% ±3% | $6.46 |
| 4 | claude-fable-5 [max] | 70% ±4% | $21.63 |
| 5 | glm-5.3 [max] | 69% ±3% | $3.99 |
| 6 | kimi-k3 [max] | 69% ±5% | $4.65 |
| 7 | gpt-5.6-luna [max] | 67% ±4% | $0.61 |
| 8 | gpt-5.5 [xhigh] | 67% ±6% | $7.23 |
| 9 | grok-4.6 [xhigh] | 67% ±2% | $5.50 |
| 10 | gemini-3.7-flash [high] | 65% ±2% | $2.18 |
| 11 | glm-5.3-flash [max] | 63% ±4% | $0.24 |
| 12 | deepseek-v4-pro [max] | 63% ±6% | $1.67 |
| 13 | claude-opus-4.8 [max] | 59% ±2% | $13.22 |
| 14 | qwen3.8-max [xhigh] | 57% ±3% | $3.73 |
The full leaderboard includes 27 configurations. I’m focusing on the 14 that made the “Best” effort level view — the settings each model vendor selected as optimal.
Three Lessons from the Numbers
1. Gemini leads on both axes — performance AND cost
This is the headline. gemini-3.8-flash doesn’t just tie for the highest pass rate at 74% ±1% — it does so at $2.36 average cost, well below the field. Google’s model is the new reference point: the one everyone else has to beat on price or performance.
The error margin (±1%) is also the tightest in the top tier, which means the 74% isn’t a lucky spike. It’s consistent.
2. The cost-performance frontier is collapsing
A year ago, the conversation was “pay more, get better.” Today, the data shows you can pay less and still land in the top tier:
- gemini-3.8-flash: 74% at $2.36
- gemini-3.7-flash: 65% at $2.18 (just $0.18 cheaper, 9 points lower)
- gpt-5.6-luna: 67% at $0.61 (a fraction of the cost, only 7 points below the leader)
- glm-5.3-flash: 63% at $0.24 (the cheapest model in the top 14, competitive at 63%)
The old model of “Claude Opus is the best, pay the premium” no longer holds. Claude-opus-5 matches Gemini’s 74% pass rate but costs 5x more ($11.84 vs $2.36). Claude-fable-5 scores 70% but at $21.63 — nearly 9x Gemini’s cost for 4 fewer points.
3. Anthropic’s pricing is becoming a competitive liability
Two Anthropic models appear in the top tier, and both suffer the same problem: they’re expensive relative to their output. Claude-opus-5 ($11.84 for 74%) and claude-fable-5 ($21.63 for 70%) are both in the top 5 by performance but rank poorly on cost efficiency.
This isn’t a quality problem. It’s a pricing strategy problem. As open-weight alternatives (DeepSeek, Qwen, Kimi) and Google’s flash models improve and price aggressively, Anthropic’s premium positioning is getting tested.
Meanwhile, DeepSeek and Zhipu AI are punching above their weight class — glm-5.3 at 69% ($3.99) and deepseek-v4-pro at 63% ($1.67) show that non-US labs can compete on value.
What’s Different in v1.1
DataCurve made three key changes to the benchmark methodology:
Isolated verification — Agents commit their code diffs, and a separate clean container applies the patch and runs tests. This follows the SWE-bench approach and prevents agents from gaming their own environment. You grade only the committed patch.
Structured test reports — Tests now emit CTRF (Common Test Retry Framework) reports, recording each task-defining test by name and status. This gives per-test visibility into partial progress.
Natural Git environment — Instead of detached HEAD, the
mainbranch is set to the task’s starting commit with no future commits visible. Agents work more naturally, and the risk of finding upstream solutions through git history is eliminated (DataCurve swept all task repos for similar implementations as of June 5th).
The bottom line: scores stay close to v1. The model ordering at the top is unchanged, and most configurations land within a few points of their v1 result. The v1.1 changes make results harder to game, not fundamentally different.
There’s one notable exception: Claude Fable 5 had ~73 of its 2,260 trials not complete due to a US government directive suspending access partway through the sweep. Pass rates are computed over completed trials only.
My Take: The new equation for engineering teams
If you’re evaluating which LLM to deploy for coding assistance or agentic workflows, this benchmark gives you a clear framework:
| Need the absolute best, cost no object? | Claude-opus-5 or gemini-3.8-flash, both at 74% |
| Need the best value? | gemini-3.8-flash — same top score, 5x cheaper than Claude |
| Need the cheapest option that still works? | glm-5.3-flash at $0.24 for 63% |
| Need a budget-conscious middle ground? | gpt-5.6-luna at $0.61 for 67% |
The era where you pick the most expensive model and call it a day is over. Today’s engineering leaders are optimizing for pass rate per dollar, and the data shows that hitting 65%+ accuracy doesn’t require a $20+ price tag anymore.
One caveat: these are benchmark tasks in a controlled environment. Real-world performance on your specific codebase, with your specific workflows, may differ. But as a baseline for vendor selection, DeepSWE v1.1 gives you the clearest picture yet of where the frontier actually sits.
Want to track these changes? DataCurve lets you sign up for leaderboard updates when new models are added. The benchmark and trajectories are also available on GitHub.
