Paper2Agent converted 74 of 100 unfiltered computational biology papers into tested agents
Paper2Agent tests every function it extracts against the paper's own published output before shipping it. Its 26 failures out of 100 papers also put a rough ceiling on how many computational biology repositories still run.
The Engineer · Build desk
What happened
- Jiacheng Miao and James Zou published Paper2Agent in Nature on September 16th, after a preprint that first appeared on September 8th, 2025; Joe R. Davis, Yaohui Zhang and Jonathan K. Pritchard co-authored it.
- The framework's validated tools are assembled into a Model Context Protocol server that a coding agent can call through natural-language instructions.
- Run against 100 computational biology papers sampled without filtering for documentation or repository quality, the pipeline produced 74 validated agents and failed on the remaining 26.
- The MIT-licensed repository ships a skill for Claude Code, Codex and the Gemini command-line interface, and had about 2,500 GitHub stars as of September 17th.
Why it matters
- Who has to decide decision Method authors now face a machine-checkable target: whether an automated pipeline can execute their tutorial and match their published outputs decides whether the paper gets a callable agent.
- What it costs cost Conversion costs a coding-agent session and an isolated environment build per paper, and that work falls on whoever wants to reuse the method.
- What it limits constraint The conversion rate was measured on computational biology papers only, so anyone quoting 74 percent for another field is extrapolating from one discipline's repository habits.
- What's now possible capability Composing two or three papers' methods on one dataset becomes something a reader can attempt without reimplementing any of them.
Next story · Science · 5h ago
Baker lab's from-scratch zinc proteases cut their chosen target in 36% of 135 designsConfirmed2 publishers
The 26 comes out of the validation stage. Paper2Agent runs six stages in order: it locates the paper's repository, builds an isolated software environment, identifies tutorials, executes them, extracts reusable functions, then tests those functions against the paper's original outputs [4]. Functions that keep failing after repeated attempts are dropped from the finished server, and the pipeline keeps a record of what ran and what did not [6]. A reader sees only the tools that survived that test [18].
Tutorials are found and executed before any function is extracted [4], so a repository with working code and no tutorial gives the extractor nothing to run. Twenty-six failures out of 100 papers is roughly one paper in four [16]. Because each of the 74 successes required code that executed in an automatically built environment, 26 percent puts a ceiling on the share of these papers whose code does not run [22]. The paper does not give the full cause breakdown [9].
The 100 papers were drawn without filtering for documentation or repository quality [7], so 74 conversions [8] is a rate over papers as they were actually published. Four conditions have to hold for that rate to travel: a locatable repository, an environment the builder can construct, at least one tutorial that executes end to end, and published outputs precise enough to test a function against [4]. The evaluation covered computational biology [7]. runtimewire, which reported the work, calls the repository's star count a measure of developer interest and not evidence of scientific adoption [19].
"Papers are written for humans," Zou wrote in the announcement thread [10]. Zou is a Stanford associate professor of biomedical data science who co-conceived the framework with Miao and supervised the work [21]. They name ordinary friction: find the repository, install fragile dependencies, reconstruct undocumented assumptions, work out which inputs produce which outputs [15]. Their aim was an agent that runs the work inside a virtual lab and gains experience with the methods [11].
The demonstrations were built around AlphaGenome, Scanpy and TISSUE, and those agents reproduced published analyses and handled new queries [12]. A second demonstration put three paper agents on a psoriasis-associated locus, connecting AlphaGenome predictions with two independent perturbation datasets, and prioritized GPR137 as the probable causal gene across methods and datasets that had been published separately [14]. In a supplementary case, an AlphaGenome agent worked with an ADHD genome-wide association dataset agent to prioritize a variant that may affect the splicing and expression of MPHOSPH9 [13]. The paper treats that result as a candidate mechanism requiring experimental validation [13].
What to watch
- Publication of the full cause breakdown for the 26 failures would settle whether missing tutorials or unbuildable environments dominate.
- A run on an unfiltered sample outside computational biology would test whether 74 of 100 is a comp-bio number.
- Experimental work on the MPHOSPH9 splicing candidate or GPR137 would move the demonstrations from prioritisation to result.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence64
- Adoption12
- Hype gap+18
- Incentives40
- Confidence60
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Jiacheng Miao (@Jiacheng_Miao) and James Zou (@james_y_zou) published Paper2Agent in Nature on September 16th, giving researchers an automated way to turn a scientific paper, its code and its supporting data into an AI agent that can explain the work, rerun analyses and apply its methods to new questions.
- [2]
The peer-reviewed publication is new, though the underlying preprint first appeared on September 8th, 2025.
- [3]
Joe R. Davis, Yaohui Zhang and Jonathan K. Pritchard co-authored the work.
- [4]
Paper2Agent uses a six-stage, multi-agent pipeline: it locates the paper's repository, builds an isolated software environment, identifies tutorials, executes them, extracts reusable functions and tests those functions against the original outputs.
- [5]
The validated tools are assembled into a Model Context Protocol (MCP) server that a coding agent can call through natural-language instructions.
- [6]
Failed functions are excluded from the final MCP server after repeated test attempts, leaving a record of what ran and what did not.
- [7]
The researchers ran the pipeline on 100 computational biology papers sampled without filtering for documentation or repository quality.
- [8]
Paper2Agent successfully converted 74 of the 100 papers into validated agents.
- [9]
The other 26 papers failed; the reported account of the causes breaks off mid-list.
- [10]
"Papers are written for humans," Zou wrote in the announcement thread.
- [11]
The researchers wanted the agent to run the work inside a virtual lab, gaining experience with the methods instead of treating the PDF as another document to summarize.
- [12]
Miao and his collaborators demonstrated Paper2Agent with agents built around AlphaGenome, Scanpy and TISSUE, and the agents reproduced published analyses and handled new queries.
- [13]
In one supplementary case, an AlphaGenome agent worked with an ADHD genome-wide association dataset agent to prioritize a variant that may affect the splicing and expression of MPHOSPH9; the paper treats that result as a candidate mechanism requiring experimental validation.
- [14]
A separate demonstration combined three paper agents to analyze a psoriasis-associated genetic locus, connecting AlphaGenome predictions with two independent perturbation datasets and prioritizing GPR137 as the probable causal gene; the procedure crossed methods and datasets that had originally been published separately.
- [15]
A researcher reusing a published computational result still has to locate the right repository, install fragile dependencies, reconstruct undocumented assumptions and work out which inputs produce which outputs.
- [16]
Twenty-six failures out of 100 papers is a 26 percent failure rate, roughly one paper in four.
- [17]
The MIT-licensed code had about 2,500 GitHub stars as of September 17th.
- [18]
Because functions that fail repeated tests are excluded, the tools in a finished Paper2Agent server are a subset of the functions extracted from the paper.
- [19]
runtimewire describes the GitHub star count as a measure of developer interest rather than evidence of scientific adoption.
- [20]
The open-source repository provides a skill that coding agents including Claude Code, Codex and the Gemini command-line interface can use to convert a paper and its associated repository.
- [21]
Zou, a Stanford associate professor of biomedical data science with appointments in computer science and electrical engineering, co-conceived Paper2Agent with Miao and supervised the work.
- [22]
Since conversion required executing the paper's tutorials and matching its original outputs, at least 74 of the 100 papers had code that ran in an automatically built environment, so the 26 percent failure rate is an upper bound on the share of the sample whose code does not run.
Sources
2 independent publishers whose own reporting we read for this story.
- dev.toThe Paper That Answers Back: Stanford Turned 100 Research Papers into Agents
1 article · October 7, 2026
- runtimewire.comPaper2Agent turns research papers into agents that rerun the science
1 article · September 16, 2026
Topics and entities
Topics
- AI Agents for Scientific Research
- Computational reproducibility
Entities
- Model Context Protocol
- AlphaGenome
- Jonathan K. Pritchard
- Nature