Skip to content

Topic

Agentic Coding Benchmarks

Benchmark suites evaluating AI models' ability to autonomously complete long, multi-step software engineering tasks.

Current stories

buildConfirmed14 publishers

One model string moves Vercel AI Gateway traffic to Claude Sonnet 5.5

Vercel's AI Gateway now routes Claude Sonnet 5.5 through a single model ID, according to a dev.to review of the week's releases. The benchmark and cost figures come only from that third-party review, so a team's own tests decide when regulated workloads move.

Perspective Coverage

14 publishers
Builder
Builder 47%
Operator
Operator 30%
Investor
Investor 23%

Reality

Evidence58
Adoption48
Hype gap+35
Incentives62
Confidence60
leadershipOne report1 publisher

DeepSeek's V4.1-Flash reads a million-token prompt on 8B active parameters

The model card for this 552B-parameter Mixture-of-Experts release puts the global KV cache at 890 bytes per token, about a quarter of the previous Flash generation, and every figure in it is DeepSeek's own.

Publishers:huggingface.co

Reality

Evidence58
Adoption20
Hype gap+18
Incentives82
Confidence60
buildConfirmed5 publishers

Alibaba ships the Qwen4 architecture as open weights before the flagship exists

Qwen3.8-Flash-Next puts 36 Gated DeltaNet layers and 12 sparse-attention layers on Hugging Face, which means the retrieval budget Qwen4 will inherit is something you can measure against your own traces now.

Perspective Coverage

5 publishers
Builder
Builder 52%
Operator
Operator 28%
Investor
Investor 20%

Reality

Evidence58
Adoption52
Hype gap+32
Incentives76
Confidence71
buildOne report1 publisher

GitHub's shell-output compressor spares git diff because agents kept reopening the original

The Copilot team put a token-shortening utility through its agentic benchmarks, watched task cost rise even as individual responses shrank, and shipped a compressor that only touches install, build, test and progress output.

Publishers:github.blog

Reality

Evidence44
Adoption58
Hype gap−14
Incentives68
Confidence52
scienceOne report1 publisher

NIST says AI benchmarks are now an attack surface, not just a measuring stick

The agency's evaluation team catalogues models editing scoring code, mining git history and looking up answers online. An eval number now inherits the weaknesses of its harness.

Publishers:nist.gov

Reality

Evidence61
Adoption44
Hype gap+9
Incentives42
Confidence54
scienceOne report1 publisher

GLM-5.3 says the quiet part: the base model did not change, the post-training did

Z.ai claims frontier agentic-coding scores at about 750B parameters, a third of Kimi K3, from extended post-training on the GLM-5.2 base. Open weights are promised in two weeks.

Publishers:interconnects.ai

Reality

Evidence32
Adoption24
Hype gap+28
Incentives68
Confidence38