Skip to content

Topic

Research Reproducibility

Whether analytical steps in published research can be inspected and repeated, and how unverifiable components affect confidence in findings.

Current stories

scienceOne report1 publisher

USC-led commentary traces Africa's water disputes to uncertainty about drought supply

Essam Heggy and co-authors argue in Nature Sustainability that uncertain drought supply, more than shortage, drives disputes over Africa's 63 shared basins. They want reproducible hydrological models to give negotiators common numbers, a remedy the commentary argues for but does not test.

Publishers:phys.org

Reality

Evidence35
Adoption
Not scored
Hype gap+30
Incentives40
Confidence40
buildOne report1 publisher

Charging 10 bps a side turns a 13.44 Sharpe into -40.13 on the same code revision

A cross-sectional reversal baseline on 15-minute Bybit perpetual bars won 67 of 76 out-of-sample windows before costs and none of them after. Charging the trading it requires puts breakeven near 2.5 bps per side.

Publishers:dev.to

Reality

Evidence58
Adoption
Not scored
Hype gap−20
Incentives20
Confidence45
scienceOne report1 publisher

Google certified a quantum attack on elliptic curve cryptography without disclosing the algorithm

The zero-knowledge proof in the March 2026 white paper turned out to certify false statements too, and Scientific American reports the algorithm Google was protecting stayed secret for only a few days.

Reality

Evidence55
Adoption30
Hype gap+30
Incentives50
Confidence58
buildOne report1 publisher

A pinned provider or a LoRA alpha can decide whether a lab's alignment result replicates

A LessWrong post argues that frontier labs publish alignment results with no code and thin methodology. Nobody outside the lab can tell which choices moved the number. It wants a dedicated effort to reproduce them.

Publishers:lesswrong.com

Reality

Evidence33
Adoption
Not scored
Hype gap+14
Incentives45
Confidence46
buildOne report1 publisher

Z-scoring the J-lens against base-model token frequency lifts hidden-word recovery to 0.805

The offset in a J-lens readout mostly tracks how often a token appears, and scaling it by variance, after plain subtraction failed, lifted hidden-word elicitation to 0.805 from 0.665 on Gemma-2-9B-it. The paired test over 20 words gives p of about 0.19.

Publishers:lesswrong.com

Reality

Evidence46
Adoption12
Hype gap+10
Incentives40
Confidence56
scienceOne report1 publisher

Rebuilt tag records suggest Atlantic menhaden natural mortality rate is half prior estimates

Jerald Ault and Jiangang Luo spent about three years rebuilding a 1960s federal tagging experiment and put Atlantic menhaden natural mortality at 0.50 a year, well below the 1.17 that entered the SEDAR 69 benchmark assessment.

Publishers:phys.org

Reality

Evidence62
Adoption
Not scored
Hype gap−10
Incentives50
Confidence55
securityOne report1 publisher

An 11-model probe puts the "this is impossible" signal 85 degrees from the refusal direction

Yucheng Du and Xiyang Hu report a single direction that separates answerable from unanswerable math and code prompts at 0.939 mean AUC across 11 models, and its mean cosine with the canonical safety-refusal direction is 0.087.

Publishers:github.com

Reality

Evidence60
Adoption
Not scored
Hype gap+12
Incentives60
Confidence50
buildOne report1 publisher

Raising the seed count from three to ten dropped most RL policies below random

A molecular optimisation experiment cleared significance on three seeds per policy and produced readable weight matrices. At ten seeds most learned policies trailed the random baseline. The first warning was a ranking that changed with the host.

Publishers:dev.to

Reality

Evidence45
Adoption
Not scored
Hype gap−15
Incentives20
Confidence55
scienceOne report1 publisher

PathSegmentor swaps the click for a typed description in pathology segmentation

Chen and colleagues report that a single text-prompted model scored highest overall across 16 pathology test sets, measured against baselines their code list names. The abstract publishes the counts; the margins are left out.

Publishers:nature.com

Reality

Evidence45
Adoption15
Hype gap+30
Incentives55
Confidence55
scienceOne report1 publisher

MatBrain splits crystal discovery between a 30B reasoner and a 14B tool-caller

Its authors report 30,000 candidate structures and 38 leads in 48 hours from models small enough to run locally, though the comparison with frontier systems arrives in the abstract without a named baseline.

Publishers:nature.com

Reality

Evidence46
Adoption14
Hype gap+34
Incentives62
Confidence54
scienceOne report1 publisher

Reasoning models spend extra tokens when a prompt cuts against a stereotype

A team adapting the implicit association test to reasoning traces found four of five models working harder on association-incompatible prompts, which puts a measurable bias signal in the process rather than only in the answer.

Publishers:nature.com

Reality

Evidence60
Adoption
Not scored
Hype gap+14
Incentives45
Confidence55
scienceOne report1 publisher

Leftover guide RNA inflates CRISPR knockdown scores whenever the primers cross the cut site

A catalytically dead Csm enzyme posted 78 to 90 percent apparent knockdown by RT-qPCR. That result points at guide RNA surviving extraction rather than at any cutting. The primer design that produced it is field convention.

Publishers:nature.com

Reality

Evidence78
Adoption30
Hype gap+8
Incentives35
Confidence65
buildOne report1 publisher

K-Veritas signs a digest of the run so a reviewer can check the number without the data

The wrapper hashes code and output, sends only a digest out to be signed with RSA-PSS, and hands the reviewer a PDF. The cost of adopting it is a command prefix. The cost of believing it is trusting whoever holds the signing key.

Publishers:runtimewire.com

Reality

Evidence60
Adoption10
Hype gap+14
Incentives55
Confidence45
scienceOne report1 publisher

The self-driving lab is out. Whether AI shows up in your filing is still open.

Stanford's Le Cong says the fully autonomous lab is a bad idea and that humans should keep the mission. The unsettled question is how AI involvement gets reported to regulators.

Publishers:genengnews.com

Reality

Evidence24
Adoption
Not scored
Hype gap+18
Incentives68
Confidence36
scienceOne report1 publisher

Selenium's metabolite zoo finally gets one naming system, two decades late

A Chiba University-led review sets a harmonized nomenclature for selenosugars plus a minimum analytical bar. It is the dull prerequisite for pooling twenty years of urinary selenium data.

Publishers:phys.org

Reality

Evidence48
Adoption
Not scored
Hype gap+24
Incentives66
Confidence57
scienceOne report1 publisher

One pot, 1,000 gene fragments: the DNA writing bottleneck starts to move

A Nature Biotechnology paper reports a hybridization-driven synthesis method that builds over 1,000 distinct gene fragments in a single reaction, then uses it to find PETase variants beating the standard enzyme.

Publishers:nature.com

Reality

Evidence57
Adoption
Not scored
Hype gap+22
Incentives62
Confidence50

Earlier coverage