Skip to content

Benchmark

HumanEval

A benchmark dataset of Python programming problems used to evaluate code-generation ability of language models, introduced by OpenAI in 2021.

Current stories

buildConfirmed3 publishers

Instinct raises $1 billion at a $10 billion valuation for an agent that operates users' phones and computers

Instinct raised $1 billion at a $10 billion valuation, four times the price TechCrunch reported for its August round. Its agent works through users' own screens and accounts, so permission and recovery are Instinct's to engineer before any app maker is asked.

Perspective Coverage

3 publishers
Builder
Builder 23%
Operator
Operator 27%
Investor
Investor 50%

Reality

Evidence55
Adoption20
Hype gap+45
Incentives60
Confidence60
productOne report1 publisher

Twenty model calls turn a two-second step into a 45-second wait

Red Hat clocks the same 20-call agent task at roughly 45 seconds on a slow backend and about 13 on a fast one. Model choice for agents is turning into a per-call latency budget, with capability as one input.

Publishers:redhat.com

Reality

Evidence45
Adoption
Not scored
Hype gap+10
Incentives50
Confidence55
productOne report1 publisher

Extra memory made post-training agents better at the plan they already picked

A Tsinghua-led study handed model training to coding agents and got real gains out of them, then found the agents almost never revised the method they chose in the first few minutes, which is the exact part that successor-building roadmaps assume.

Publishers:turingpost.com

Reality

Evidence44
Adoption
Not scored
Hype gap+18
Incentives30
Confidence38
buildOne report1 publisher

Post-training agents pick their strategy before they run a single experiment

An empirical study of publicly released PostTrainBench trajectories finds the training strategy is fixed at step one and the whole remaining budget goes on local tweaks, with scaffolding and human hints improving execution and leaving that pattern intact.

Publishers:arxiv.org

Reality

Evidence58
Adoption20
Hype gap+12
Incentives35
Confidence55
buildOne report1 publisher

DeepSeek paid for Engram's lookup table by deleting 17 of its 72 routed experts

The Engram explainer sells a second scaling axis beyond mixture-of-experts. The published comparison holds parameters and per-token compute fixed, which makes it a swap.

Publishers:runtimewire.com

Reality

Evidence52
Adoption14
Hype gap+32
Incentives62
Confidence55
buildOne report1 publisher

Re-graded on post-cutoff problems, the cheap model loses 30 points and routing triples its value

OmnisBench's author rebuilt his split from date-stamped LiveCodeBench problems. The cheap tier fell from 90% to 60%, and a 4,096-token cap had been marking the frontier model absent.

Publishers:dev.to

Reality

Evidence42
Adoption
Not scored
Hype gap+12
Incentives78
Confidence48