Skip to content

Topic

Model calibration

How closely a model's stated confidence matches its observed accuracy, measured with metrics such as Expected Calibration Error and the Brier score.

Current stories

buildConfirmed7 publishers

Andreessen Horowitz leads a reported $870M round for Jev maker TypeSafe

Andreessen Horowitz led a financing of about $870 million in TypeSafe at a $7.5 billion valuation, Bloomberg reported. The money backs Jev, a model that hands software a choice plus a probability, and its use to builders depends on those probabilities matching how often it is right.

Perspective Coverage

8 publishers
Builder
Builder 38%
Operator
Operator 23%
Investor
Investor 39%

Reality

Evidence50
Adoption30
Hype gap+45
Incentives70
Confidence60
buildOne report1 publisher

OpenAI and AWS reportedly answer Jev with decision models of their own within two weeks

OpenAI and AWS shipped their own decision models within two weeks of TypeSafe AI's September 15 launch of Jev, according to a dev.to account. For routing work, the comparison with an LLM call turns on whether a decision model's probabilities are calibrated.

Publishers:dev.to

Reality

Evidence30
Adoption25
Hype gap+35
Incentives55
Confidence35
buildOne report1 publisher

Open 0.8B model Jeff answers agents' multiple-choice calls and escalates the unsure ones to a 27B

Jeff, an open 0.8B model, returns a probability for each option in one forward pass and sends low-confidence agent decisions to Qwen3.8-27B. The confidence score attached to each answer tells an agent when a small decision is worth the larger model's time.

Publishers:dev.to

Reality

Evidence38
Adoption15
Hype gap+15
Incentives
Not scored
Confidence35
buildConfirmed3 publishers

llama.cpp brings TypeSafe's typed-decision format to five open model families on local hardware

llama.cpp merged a /v1/systemone endpoint on October 2 that returns typed answers with probabilities from five open model families running locally. Whether it can replace hosted classification calls depends on calibration that each team has to measure on its own data.

Perspective Coverage

3 publishers
Builder
Builder 48%
Operator
Operator 29%
Investor
Investor 23%

Reality

Evidence50
Adoption15
Hype gap+5
Incentives55
Confidence55
securityOne report1 publisher

Sophos wants SOCs to test TypeSafe's Jev for accuracy and calibration before it closes alerts

Sophos cites an independent test putting TypeSafe's Jev at 83% on Banking77 with no training examples, 10 points behind a trained classifier. Jev costs about a twelfth as much per email as Claude Haiku 4.5, so SOCs will be tempted to automate at a volume where small error rates add up.

Reality

Evidence55
Adoption
Not scored
Hype gap+20
Incentives40
Confidence50
buildOne report1 publisher

Ollaya's latency table sets a local RTX 4090 against Jev's network round trip

Mert Cobanov's open-source Ollaya serves local decision models behind a Jev-compatible API and reports 8 to 10 ms per request on an RTX 4090. Its comparisons with TypeSafe's hosted service are uncontrolled, so teams have to check accuracy on their own traffic before switching.

Publishers:runtimewire.com

Reality

Evidence38
Adoption
Not scored
Hype gap+25
Incentives60
Confidence35
buildConfirmed14 publishers

Jev turns a catalogue ID from a prompt constraint into part of the decision domain

TypeSafe's Jev answers typed questions with floats and probabilities. An invoice pipeline that handed it classification and catalogue selection still needs a generative model for field extraction and for the note a human reads.

Perspective Coverage

14 publishers
Builder
Builder 53%
Operator
Operator 31%
Investor
Investor 16%

Reality

Evidence55
Adoption40
Hype gap+30
Incentives65
Confidence55
buildOne report1 publisher

Jev's confidence scores cleared 86.5% of graded federal RFQs for automation at 96.7% precision

Jev's confidence scores cleared 641 of 741 quote-graded federal RFQs for automation at 96.7% precision, while a 35B Qwen model's top bucket stayed at 90.1%. In one author's benchmark, calibration decided how much of the queue could skip human review more than Jev's 2.3-point accuracy lead did.

Publishers:dev.to

Reality

Evidence45
Adoption
Not scored
Hype gap+15
Incentives
Not scored
Confidence40
buildOne report1 publisher

A 4-bit Gemma 4 26B on one L4 trails TypeSafe's Jev by 2.1 points overall

A pre-registered run reads Gemma's label logits on one 24 GB GPU and scores it against the hosted API on the same 3,880 records, where it is level on yes/no questions and 4.5 points behind on multiple choice. One temperature fitted on 50 labels closes the calibration gap.

Publishers:dev.to

Reality

Evidence70
Adoption20
Hype gap−12
Incentives35
Confidence55
leadershipOne report1 publisher

TypeSafe's own evals put Jev level with GPT-5.6 Terra at a 75th of the cost per decision

Jev returns typed answers with probability scores and no free-form text, and TypeSafe prices a decision at four hundredths of a cent. Its accuracy figures are scored against other models' probabilities.

Publishers:arize.com

Reality

Evidence34
Adoption22
Hype gap+30
Incentives68
Confidence44
buildOne report1 publisher

A 60-case benchmark puts Jev's usable confidence threshold at exactly 1.000

TypeSafe AI launched Jev with 193.6x and 444.6x multipliers and nothing to re-run. An independent harness measured something else, whether the confidence score is calibrated well enough to route escalations on.

Publishers:dev.to

Reality

Evidence66
Adoption14
Hype gap+38
Incentives48
Confidence58
buildOne report1 publisher

Rounding a judge model's 0.99 into a price tier billed a synthesis as a lookup

A dev.to developer prices each API request after the work by asking a second model how much the answer represents, and the first version threw away the calibration he was paying for by rounding its probabilities into three tiers.

Publishers:dev.to

Reality

Evidence38
Adoption8
Hype gap−12
Incentives45
Confidence52
scienceOne report1 publisher

A 10,000-call probe puts Jev's probability readout at the end of prefill

archerhume timed latency against context length and question order to work out how TypeSafe's closed model returns distributions instead of text. Verifying that those numbers are calibrated takes labelled outcomes he lacked.

Publishers:archerhume.com

Reality

Evidence30
Adoption15
Hype gap+15
Incentives35
Confidence38
buildOne report1 publisher

A model-checked quorum rule absorbed an LLM oracle's jitter across 1,680 chaos rounds

The model's answers wobble by a few hundredths on identical input, so the checkable properties live in a TLA+ spec and a Rust quorum rule that counts only votes whose margin clears a measured noise floor.

Publishers:dev.to

Reality

Evidence55
Adoption12
Hype gap+12
Incentives40
Confidence45
buildOne report1 publisher

Nine models rewrote already-optimal code in all 45 trials of an EffiBench study

In a paper posted on 13 September, nine models across the Claude, GPT and Gemini families edited already-optimal EffiBench solutions in every trial. One added sentence in the prompt recovered 20 refusals.

Publishers:dev.to

Reality

Evidence45
Adoption18
Hype gap+15
Incentives35
Confidence55
buildOne report1 publisher

One yes/no question catches 90% of APPS backdoors at a 2% audit budget

A pilot posted on LessWrong scored ControlArena's backdoor set with a single typed probability from TypeSafe's Jev, reaching AUROC 0.976 for about four cents per thousand submissions on $5 of credit and no baseline run of its own.

Publishers:lesswrong.com

Reality

Evidence42
Adoption10
Hype gap+22
Incentives
Not scored
Confidence35
productOne report1 publisher

TypeSafe leaves stealth with a model whose confidence scores developers have to calibrate themselves

TypeSafe's first model, Jev, answers structured questions with typed output and a confidence measure attached. The account of its launch says developers have to check those scores against real outcomes on their own data first.

Publishers:siliconangle.com

Reality

Evidence26
Adoption7
Hype gap+48
Incentives80
Confidence57
buildOne report1 publisher

A 0.7 confidence gate lets 10 to 20 percent of wrong Whisper tokens through in noise

A developer's write-up of a medical transcription pipeline cites a 2025 study finding that under noise Whisper puts confidence above 0.7 on tokens that are wrong, and that the overconfidence grows as the signal-to-noise ratio falls.

Publishers:dev.to

Reality

Evidence35
Adoption10
Hype gap+28
Incentives65
Confidence40

Earlier coverage