GitHub opens ReviewBench to grade AI code reviewers on 219 real pull requests
GitHub's ReviewBench grades AI code reviewers on 219 pull requests from 187 open-source repositories, counting real issues found and false alarms raised. Security teams get a shared, repeatable check before letting a bot gate a release, on a test GitHub also uses to tune its own Copilot reviewer.
The Watch · Security desk
What happened
- GitHub built its golden set of known issues from human reviewer comments, follow-up code changes, analysis tools and AI model output, then merged duplicates and graded them on a shared rubric.
- GitHub ranks systems on grounded recall, the share of known issues found, and uses separate augmented scores to assess each system on its own.
- GitHub said combining several model runs in Copilot code review's lite tier raised production recall 13.6% and cut cost per review 8% against its control.
- Results stay private until a maintainer approves them and are published only for an agent's first leaderboard entry or a score above its last.
Why it matters
- Who has to decide decision A team weighing a bot as a merge gate can rank agents on security findings with the beta weight tilted toward fewer false alarms, and need not adopt GitHub's recall-first default as its ranking.
- What's now possible capability Any team can package its own reviewer as a container image with its configuration and model key and run it through the same 219 pull requests the listed agents faced.
- What it limits constraint The public board keeps each agent's first entry and its highs, so a vendor's later regression shows up only in a buyer's own reruns.
- Who has to decide decision Gating releases on a leaderboard score trusts the benchmark further than GitHub does, since GitHub uses it to pick changes for user tests and treats those tests as final.
Next story · Build · Oct 5
GitHub's ReviewBench grades AI code reviewers against 219 public pull requestsConfirmed2 publishers
A release gate depends on two numbers. Grounded precision is the share of an agent's findings that match known issues, and grounded recall is the share of known issues it detects; F1 weights the two equally [6]. Augmented precision, recall and F1 add credit for valid findings that are missing from the golden set [7].
Security is one of the categories the leaderboard breaks out [3]. The published description does not say how many golden-set issues fall in that category.
Models enter the scoring twice [20]. AI models were among the sources of candidate findings for the golden set [5]. The same AI judge then scores every agent [15], and an AI judge decides whether findings outside the golden set are valid [7].
The test set is weighted by design. "We make one deliberate adjustment to this distribution: while language and repository size mirror GitHub directly, pull request size is weighted toward the reviewable middle and tail. This reduces the overrepresentation of tiny, single-file changes while preserving more substantive, multi-file pull requests where review quality matters most," Michelle Zhou, a data scientist at Microsoft, and Alejandro Carderera de Diego, a senior applied researcher at GitHub, wrote [4][17].
The production figures come from GitHub testing its own reviewer [10]. The lite-tier change combined several independent model runs into a single review [10]. According to GitHub, the comment metric, up 8%, counts review comments that an AI model judged to have prompted code changes [11]. Production recall is inferred from how much additional human review was still needed [12]. GitHub also reported that feedback moved toward critical and moderate issues, with fewer minor suggestions [12].
A scored run is three rounds over all 219 pull requests, or 657 reviews, after a 25-pull-request set used to assess and refine the agent [14][18]. Those reviews run on the model access key the submitter registers [14].
ReviewBench is in research preview [1]. Zhou and Carderera de Diego asked researchers and practitioners to run their own systems against it, probe the assumptions behind it and contribute to refining its methods [16].
What to watch
- Whether GitHub discloses how many golden-set issues are security issues, or publishes security-filtered scores for listed agents.
- Whether the rule that publishes only first entries and new highs survives when ReviewBench leaves research preview.
- Whether outside submitters running the full 219-pull-request set get results consistent with GitHub's Copilot lite-tier figures.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence60
- Adoption20
- Hype gap+10
- Incentives70
- Confidence60
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
ReviewBench is available in research preview through its website and lets users compare review agents and evaluate their own tools.
- [2]
ReviewBench measures AI reviewers' ability to identify issues and avoid false alarms; users can inspect the test data, reproduce results and track improvements.
- [3]
ReviewBench tests agents on the same 219 pull requests from 187 public repositories with open-source licenses across 19 programming languages, measuring how many known issues each agent finds, how many findings are valid, and how performance varies by severity and category, including security issues.
- [4]
We make one deliberate adjustment to this distribution: while language and repository size mirror GitHub directly, pull request size is weighted toward the reviewable middle and tail. This reduces the overrepresentation of tiny, single-file changes while preserving more substantive, multi-file pull requests where review quality matters most,
- [5]
GitHub built the golden set of validated findings in three stages: it gathered candidate findings from human reviewers, follow-up code changes, analysis tools and AI models, merged findings describing the same issue, and assessed them using a shared evaluation rubric.
- [6]
The benchmark reports six metrics in two groups; grounded precision is the share of an agent's findings that match known issues, grounded recall is the proportion of known issues it detects, and F1 gives both equal weight.
- [7]
Augmented precision, recall and F1 account for findings outside the golden set; an AI judge assesses whether these additional findings are valid.
- [8]
GitHub uses grounded recall as the main measure for comparing systems and augmented metrics to assess each system individually.
- [9]
Users can filter results by severity and category and adjust beta in the F-beta score to give more weight to issue detection or to fewer false alarms; the leaderboard updates rankings based on those preferences.
- [10]
GitHub uses ReviewBench to evaluate changes to Copilot code review before testing them with users; in one experiment with the lite tier, it combined several independent model runs into a single review.
- [11]
In the production test, the share of review comments judged by an AI model to have prompted code changes rose by 8%, recall rose by 13.6% and cost per review fell by 8%, relative to the production control.
- [12]
GitHub measures production recall through how much additional human review is still needed; feedback moved toward critical and moderate issues, with fewer minor suggestions.
- [13]
GitHub says the benchmark helps identify updates for production testing, and tests with users remain its final measure of impact.
- [14]
Users sign in with GitHub and register an agent by providing its container image, configuration and model access key; a 25-pull-request test set lets them assess and refine performance before running three rounds on the full set of 219 pull requests.
- [15]
ReviewBench scores every agent with the same AI judge; scores remain private until a maintainer approves the submission, and are published for an agent's first leaderboard entry or when it exceeds its previous leaderboard score.
- [16]
Michelle Zhou and Alejandro Carderera de Diego invited researchers and practitioners to evaluate their systems, examine the benchmark's assumptions and help improve its methodology.
- [17]
Michelle Zhou is a Data Scientist at Microsoft and Alejandro Carderera de Diego is a Senior Applied Researcher at GitHub.
- [18]
A full scored ReviewBench run is 657 pull request reviews.
- [19]
A resubmission that does not exceed an agent's previous leaderboard score is not published on the public leaderboard.
- [20]
AI models enter ReviewBench scoring at two points: as one source of golden-set candidate findings and as the judge that scores agents.
Sources
2 independent publishers whose own reporting we read for this story.
- dev.toAI review benchmarks score the bot. Nobody scores the reader
1 article · October 9, 2026
- helpnetsecurity.comGitHub’s ReviewBench puts AI code reviewers to the test
1 article · October 6, 2026
Topics and entities
Entities
- GitHub Copilot code review
- Microsoft
- Alejandro Carderera de Diego
- GitHub