Skip to content

Benchmark

SWE-bench Pro

Scale AI's benchmark of AI coding agents on harder, longer-horizon real-world repository tasks than SWE-bench Verified, scored by tests passing.

Known aliases

  • SWE-bench Pro v1

Current stories

buildOne report1 publisher

Reflection's Beam splits two coding benchmarks with GLM 5.2 before its weights ship

Reflection AI's 501-billion-parameter Beam beats GLM 5.2 on SWE-bench Pro and trails it on Terminal-Bench, by the company's own scores. Its claimed compute saving covers only part of a serving bill, so planning around it as a US-built open model has to wait for outside tests.

Publishers:runtimewire.com

Reality

Evidence35
Adoption8
Hype gap+30
Incentives75
Confidence40
buildConfirmed3 publishers

Four passing runs out of 80 separate first from second on Specific's private-code benchmark

Specific Labs scores coding agents on licensed production codebases. The best setup clears 38.8%. The analysis covers ten tasks at eight runs each, so every published score is a count of passing rollouts out of 80.

Perspective Coverage

3 publishers
Builder
Builder 52%
Operator
Operator 33%
Investor
Investor 15%

Reality

Evidence48
Adoption
Not scored
Hype gap+30
Incentives70
Confidence58
buildOne report1 publisher

Anthropic prices its newer Sonnet a third below Sonnet 4.5

Sonnet 4.5 still leads GPT-5 on the coding leaderboards, and GPT-5 lists about 46 percent below it on a 5:1 token mix. Anthropic's current Sonnet undercuts both of Sonnet 4.5's list prices, and that complicates a routing plan built on the older pair.

Publishers:dev.to

Reality

Evidence40
Adoption20
Hype gap+15
Incentives55
Confidence45
buildOne report1 publisher

Firing 4.8% of the weights per token still leaves 125GB to keep resident

Alibaba's Qwen3.8-Flash-Next preview activates 6B of its 125B parameters per token. Per-token compute drops to under a quarter of the dense 27B's, and about 125GB of weights still has to stay on device.

Publishers:dev.to

Reality

Evidence34
Adoption18
Hype gap+15
Incentives60
Confidence42
buildOne report1 publisher

Discovery Loop bets the research bottleneck sits in taste and evaluation

Oriol Vinyals has left Google DeepMind to build Discovery Loop with Jeff Dean, Sanjay Ghemawat and Quoc Le, on the argument that AI already writes the code and runs the experiments while idea generation and judging results lag.

Publishers:the-decoder.com

Reality

Evidence45
Adoption15
Hype gap+20
Incentives75
Confidence45
buildOne report1 publisher

NVFP4 squeezes Qwen3.8's 2.4 trillion weights onto eight B300s at 150 GB a GPU

Alibaba's largest open-weight release fits on a single eight-GPU node only because a community four-bit build takes it down to 1.2 TB. AWS publishes the vLLM config for it and skips the price.

Publishers:aws.amazon.com

Reality

Evidence52
Adoption22
Hype gap+32
Incentives78
Confidence48
buildConfirmed5 publishers

Alibaba ships the Qwen4 architecture as open weights before the flagship exists

Qwen3.8-Flash-Next puts 36 Gated DeltaNet layers and 12 sparse-attention layers on Hugging Face, which means the retrieval budget Qwen4 will inherit is something you can measure against your own traces now.

Perspective Coverage

5 publishers
Builder
Builder 52%
Operator
Operator 28%
Investor
Investor 20%

Reality

Evidence58
Adoption52
Hype gap+32
Incentives76
Confidence71
buildOne report1 publisher

Cached prefixes push 91% of a ReAct agent's LLM time into decode

An arXiv tracing study of Claude Code agents on Gemma and Qwen measured prefix-cache hit rates between 84.6 and 99.5 percent, which moves the serving bottleneck to how long you can keep KV blocks resident between tool calls.

Publishers:arxiv.org

Reality

Evidence58
Adoption25
Hype gap+12
Incentives
Not scored
Confidence55
productOne report1 publisher

Four leaderboards, four denominators: what you buy when you standardize on a coding agent

Token traffic, survey reach and production-model ledgers rank different vendors because they count different things. The autonomy figures say the hard part is still unbought.

Publishers:devops.com

Reality

Evidence58
Adoption64
Hype gap+32
Incentives68
Confidence52
productOne report1 publisher

A 27B laptop model scores like a rented one, and thinks three times as hard to do it

Alibaba's Apache-2.0 Qwen3.8-27B fits in about 17GB and matched near-frontier scores, per Artificial Analysis. It also burned 3.7x the median output tokens getting there.

Publishers:thenextweb.com

Reality

Evidence62
Adoption64
Hype gap+18
Incentives60
Confidence55
buildOne report1 publisher

Microsoft's MAI-Thinking-1 lands in Foundry, and .NET teams get a reasoning model without Python

MAI-Thinking-1 is in public preview in Microsoft Foundry at $2 per million input tokens. For C# shops the consequence is an interface swap, not a new runtime to operate.

Publishers:dev.to

Reality

Evidence34
Adoption20
Hype gap+32
Incentives46
Confidence41
buildOne report1 publisher

A 27B Apache-2.0 model in 17GB makes local inference a wiring decision, not a demo

Qwen 3.8 27B ran on a MacBook Pro from a 17GB GGUF and spent 21 minutes on one SVG. Licensing and access stopped being the blocker; latency and KV cache budgeting became the job.

Publishers:dev.to

Reality

Evidence48
Adoption58
Hype gap+18
Incentives62
Confidence52