GitHub's ReviewBench grades AI code reviewers against 219 public pull requests

The best engineering in ReviewBench is how the answer key gets built. GitHub wrote: "No single reviewer, whether human or model, can identify everything worth finding in a pull request." ^[12] Candidate findings come from four sources: human reviewers, issues inferred from the author's follow-up commits, deterministic analysis tools, and several frontier LLMs from different model families ^[5]. Findings that describe the same underlying issue are then merged ^[6]. That step matters. Without it, a bug every model can spot would be counted several times in the golden set, and a reviewer that thinks like the consensus would look stronger than it is.

Scoring is where I would spend review time. One model, Claude Sonnet 5, grades every submission, and a finding counts only if it is true, relevant and non-trivial ^[7]. The metrics are built to measure newly discovered issues as well as known ones ^[9]. That implies the grader also rules on findings outside the golden set, with no human label to check against. Relevant and non-trivial are judgment calls. A team whose senior engineers draw that line somewhere else will get a ranking that does not match its own. GitHub says senior engineers independently validated the benchmark ^[10]; the post does not say how many, or what share of the grader's calls they checked. An LLM grading LLM output against an answer key partly written by LLMs is a loop someone should check by hand. GitHub publishes both the rubric and the judge, so anyone can run that check ^[8].

For a score to transfer, a team's pull requests have to resemble the corpus. The 219 PRs come from public open source repositories, sampled so that language and repository size track the 103.9 million PRs GitHub analysed ^[2][3]. Spread over 19 languages, 219 PRs average about 11.5 per language ^[15]. The mix follows GitHub's, so the most-used languages take more than their share and the rest get less. Severity and category slices cut those cells smaller again. Every one of the 187 repositories contributes at least one PR, so at least 155 of them contribute exactly one ^[16].

PR size was deliberately weighted toward the middle and tail, away from tiny single-file changes ^[4]. I think the weighting is right for measuring review quality. It does mean a team whose traffic is mostly one-file fixes is scored on a harder mix than it ships.

The production claim is GitHub's, about its own product. GitHub says ReviewBench made its offline evaluation of Copilot code review better at anticipating the direction of production experiments ^[17]. Direction is a weaker claim than size of effect, and it is reported for one reviewer. Nothing in that result shows that a ranking of two other vendors' tools on these 219 PRs predicts which one a particular team will prefer. GitHub lists severity, category and precision-recall breakdowns among the things a good benchmark should support ^[13]. Its post describes the trade-offs between reviewers this way: "Some reviewers surface more issues, some produce less noise, and some are stronger at catching critical problems while others surface smaller improvements, too." ^[14]

I would start a reviewer bake-off here. ReviewBench is available now ^[1]. The dataset is public, and the post documents how to onboard a code review system and submit results ^[11]. Any vendor's number can be rerun by someone who does not work for that vendor, GitHub included. What ReviewBench cannot supply is a grade from a team's own engineers on a sample of that team's own PRs.