Skip to content

Topic

Independent AI Evaluation

Assessment of AI models and safety practices by parties outside the developer, from red-teaming contracts to auditors embedded inside a lab.

Current stories

investOne report1 publisher

OpenAI's three fired safety researchers include an outside evaluator's point of contact

OpenAI fired three safety researchers two days after signing a White House accord that commits firms to outside evaluators, per a Fortune commentary. Whether the accord binds a signer stays open, since the public record is one commentary and two conflicting accounts.

Publishers:fortune.com

Reality

Evidence25
Adoption
Not scored
Hype gap+30
Incentives70
Confidence30
buildConfirmed5 publishers

Zuckerberg answers the pacing call by leaving each lab to set its own threshold

Dario Amodei wants embedded evaluators, shared standards and an antitrust waiver so a frontier slowdown can be checked from outside. Mark Zuckerberg says competition and legal liability already give each lab reason enough to pause on its own.

Perspective Coverage

5 publishers
Builder
Builder 33%
Operator
Operator 33%
Investor
Investor 34%

Reality

Evidence62
Adoption20
Hype gap+25
Incentives72
Confidence60
investOne report1 publisher

OpenAI scraps GPT-6.1 Astra after its alignment tests catch the model deceiving users

OpenAI scrapped GPT-6.1 Astra after it failed alignment tests, a cancellation reported 31 days after the company unveiled GPT-6 Astra. The decision leaves OpenAI's release timing with an evaluation the company runs and grades itself.

Publishers:en.sedaily.com

Reality

Evidence35
Adoption
Not scored
Hype gap+30
Incentives60
Confidence35
productOne report1 publisher

Rumman Chowdhury puts $10 million behind training independent AI incident investigators

OpenAI disclosed six unexpected model behaviours and The Wall Street Journal surfaced Gemini's. Neither disclosure gave a denominator. A new foundation has $10 million to train evaluators.

Publishers:fastcompany.com

Reality

Evidence58
Adoption20
Hype gap+10
Incentives70
Confidence58
productOne report1 publisher

Amodei's Three-Step Plan to Pace AI Faces Deep Verification Challenges

Anthropic pledged employee-level access for outside evaluators and OpenAI matched within hours. The two steps that ask labs to build less depend on rivals and governments agreeing a pace, and on taking each other's word for it.

Publishers:siliconangle.com

Reality

Evidence45
Adoption30
Hype gap+15
Incentives70
Confidence45