Epoch AI's benchmark audit fails nine of the first 15 versions it inspected
Broken answer keys and graders that punish correct tool calls drove the verdicts. Epoch AI says it stops each review once it has enough evidence, so the published defect counts are floors.
The Engineer · Build desk
What happened
- Epoch AI launched Benchmark Reviews on September 17th and labelled nine of the first 15 benchmark versions it audited as Flawed, four as Verified, and two as Not Enough Info.
- Its published methodology gives a Flawed verdict when at least 20 percent of an inspected sample contains errors, or when a single issue corrupts grading at scale.
- The Humanity's Last Exam review found substantial accuracy-altering errors in 22 of 48 sampled questions, and judged 12 of those questions impossible to answer correctly as written.
- Berkeley Function Calling Leaderboard v4, which scores whether a model picks the right tool and passes the right arguments, showed potential accuracy defects in 24 of 50 sampled tasks.
- DeepSWE v1.1 crossed the threshold narrowly, with confirmed false negatives in at least 23 of 113 tasks after the grader discarded changes an agent had made.
Why it matters
- Who's exposed exposure A team that put a Humanity's Last Exam or Berkeley Function Calling Leaderboard v4 number into a deployment or procurement case now has to defend a score with a public task-level defect list behind it.
- What it limits constraint Because Epoch AI halts each review at the first sufficient evidence, a Flawed verdict gives a buyer the floor of the problem and leaves the size of the correction to whoever needs the number.
- Who has to decide decision Version strings become part of the citation, since the audit treats tests that change without a version update as a defect in its own right.
- What's now possible capability A maintainer who disputes a Flawed label can rerun the same inspection at the same task counts, because the sampling and expansion thresholds are published.
Next story · Build · 1h ago
Mercor's map of 29 AI benchmarks covers just 208 of 923 US occupationsConfirmed2 publishers
The defects sit in answer keys and graders. Epoch AI's review of Berkeley Function Calling Leaderboard v4, which tests whether a model selects the right tool and supplies the right arguments [10], found a task that expected the model to add a stock already present on the watchlist [11]. A model that checks the list first and declines the duplicate is scored as a failure. Another task's web-search answer had become stale since the key was written [11].
In Humanity's Last Exam, one question asked for a four-point discrete Fourier transform and supplied eight entries [8]. Another answer key pointed at the wrong multiple-choice option [8]. Epoch AI reported problems that could reject valid answers and problems that could accept incorrect ones [9].
The sampling rules explain the sample sizes. Epoch AI takes 50 tasks from a benchmark with more than 50 available, expands to 100 when the observed error rate falls between 15 and 25 percent, and inspects every task when there are 50 or fewer [15]. DeepSWE v1.1's confirmed false negative rate is 20.3 percent [12], inside that band, and the 113 tasks inspected are past the expansion level [22]. Humanity's Last Exam at 46 percent and Berkeley Function Calling Leaderboard v4 at 48 percent are both above the band [6][21], so a 50-task sample settles them [23]. The Humanity's Last Exam sample was 48, two short of the stated 50, which at that error rate does not change the verdict [6].
Epoch AI stops a review once it has gathered enough evidence for a Flawed verdict, and says the writeups should be read as a minimum case [5]. A Flawed label bounds the defect count from below.
Nine Flawed out of 15 is 60 percent [20]. The registry Epoch AI maintains holds 85 benchmarks [16], so the first batch covers at most about a fifth of it [24], and the launch material does not say how the 15 versions were picked. For that 60 percent to transfer to whatever benchmark you are about to cite in a deployment review, the selection would have to have been blind to which tests already looked suspect. The per-benchmark findings hold on their own.
The reviews were completed between August 10th and September 12th [17], so the September 17th launch published an accumulated program. Epoch AI, co-founded and led by Jaime Sevilla, argues that AI progress needs better measurement than leaderboards and launch-day claims supply [19]. Epoch AI says its founding group was frustrated that a consequential industry was still being measured through "hype and vibes" [18].
What to watch
- Whether Epoch AI publishes a selection rule for the next batch, and whether the Flawed share holds outside the first 15 versions.
- Whether maintainers reissue corrected versions after a Flawed verdict, and whether Epoch AI re-reviews them under a new version number.
- Whether model developers keep citing Humanity's Last Exam and Berkeley Function Calling Leaderboard v4 scores now that the sampled defects are public.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence65
- Adoption
- Not scored
- Hype gap+5
- Incentives35
- Confidence62
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Epoch AI classified nine of its first 15 reviewed benchmark versions as Flawed, gave Verified labels to four, and lacked enough information to judge the remaining two.
- [2]
Epoch AI assigns each reviewed benchmark version one of three verdicts: Verified means results can broadly be interpreted as the creator describes them, Flawed means a substantive score-affecting defect was found, and Not Enough Info means the tasks, scoring logic or evaluation settings were too inaccessible to judge.
- [3]
Under Epoch AI's published methodology, a benchmark normally receives a Flawed verdict when at least 20% of an inspected sample contains errors, or when one issue corrupts grading at scale.
- [4]
The review also checks for changing tests without version updates, restrictive evaluation setups, and model-specific advantages caused by unequal scaffolds or compute budgets.
- [5]
Epoch AI stops a review once it has gathered enough evidence for a Flawed label, so those writeups should be read as a minimum case rather than a complete defect inventory.
- [6]
Epoch AI's review of Humanity's Last Exam found substantial accuracy-altering errors in 22 of 48 sampled questions, or 46%.
- [7]
Twelve of the sampled Humanity's Last Exam questions were judged impossible to answer correctly as written.
- [8]
One Humanity's Last Exam question requested a four-point discrete Fourier transform while supplying eight entries; another question's answer key pointed to the wrong multiple-choice option.
- [9]
Other Humanity's Last Exam problems could reject valid answers or accept incorrect ones.
- [10]
In the Berkeley Function Calling Leaderboard v4 review, Epoch AI found potential accuracy defects in 24 of 50 sampled tasks; the benchmark tests whether models select tools and supply the correct arguments.
- [11]
Epoch AI found Berkeley Function Calling Leaderboard v4 cases where correct behavior could be penalized, including a task that expected a model to add a stock already present on a watchlist and a web-search question whose answer had become stale.
- [12]
Epoch AI confirmed false negatives in at least 23 of 113 DeepSWE v1.1 tasks, or 20.3%, and in many cases the grader discarded changes that an agent had made.
- [13]
AI leaderboards guide product claims, deployment decisions and policy, and Epoch AI's audits show that some headline scores reflect broken graders and ambiguous tasks rather than model capability.
- [14]
Epoch AI launched Benchmark Reviews on September 17th.
- [15]
For benchmarks with more than 50 available tasks, Epoch AI generally samples 50, expanding to 100 when the observed error rate falls between 15% and 25%; it examines every available task when a benchmark has 50 or fewer.
- [16]
The Benchmark Reviews program turns Epoch AI's benchmark work into a public quality-control layer across its registry of 85 benchmarks.
- [17]
The initial reviews were completed between August 10th and September 12th, making the launch the publication of an accumulated audit program rather than 15 reviews performed at once.
- [18]
Epoch AI says the founding group was frustrated that a consequential industry was still being measured through "hype and vibes."
- [19]
Epoch AI, co-founded and led by Jaime Sevilla, holds that AI progress requires better measurement than the industry's usual mixture of leaderboards, selective model cards and launch-day claims.
- [20]
Nine Flawed verdicts out of 15 reviewed versions is 60 percent of the audited set.
- [21]
Potential accuracy defects in 24 of 50 sampled Berkeley Function Calling Leaderboard v4 tasks is a 48 percent observed error rate.
- [22]
DeepSWE v1.1's 20.3 percent confirmed false negative rate falls inside Epoch AI's 15 to 25 percent expansion band, and the 113 tasks inspected exceed the 100-task expanded sample the rule calls for.
- [23]
Humanity's Last Exam at 46 percent and Berkeley Function Calling Leaderboard v4 at 48 percent both sit above the 25 percent top of the expansion band, so the 50-task sample stands under Epoch AI's rule.
- [24]
Fifteen reviewed versions against a registry of 85 benchmarks is about 18 percent, so the first batch covers at most about a fifth of the registry.
Sources
2 independent publishers whose own reporting we read for this story.
- runtimewire.comEpoch AI audits 15 benchmarks, finds nine flawed
1 article · September 17, 2026
- theneuron.aiWho Grades the AI Tests? Epoch Found Problems in Nine Major Benchmarks
1 article · September 18, 2026