Page-picking model Jev cut wrong-page retrievals from 33% to 3% in Sayora's receptionist test
Sayora's receptionist test found Jev gave the LLM a wrong page on 3% of 168 questions, against 33% for its embeddings. The figure is Sayora's own, from four small-business sites, and transfers only to sites built from single-topic pages.
The Engineer · Build desk
Bar chart comparing how often each system put the right page first: Jev 88% of the time, Sayora's embeddings 70% of the time, per Sayora's blog post.
Right page ranked first, Sayora's test, per Sayora In % of the time
| Item | Value | Claim |
|---|---|---|
| Jev | 88% of the time | 3 |
| Sayora's embeddings | 70% of the time | 3 |
What happened
- Jev put the right page first 88% of the time, against 70% for the production embeddings.
- Sayora kept pages whole and described each one. A lookup tool asks Jev which page answers the caller, with 'None of these' as a valid option.
- The test ran on four customer sites: a 22-page bar and restaurant, a 74-page estate agent, a 75-page electrician and a 232-page multi-trade maintenance company.
- TypeSafe launched Jev on 15 September at $0.042 per million input tokens, with output free.
Why it matters
- What's now possible capability Jev returns a probability for every option and 'none' is one of them. The receptionist can say it does not know and hand the call on instead of reading a wrong page to the caller.
- What it limits constraint A choice question tops out at 255 options and the largest test site had 232 pages, leaving 23 options of headroom. A site past 255 pages exceeds a single question.
- Who has to decide decision Switching moves retrieval from injection on every caller turn to a tool call the receptionist makes when it hits something it does not know. In our view that puts the LLM's own judgement of its gaps in charge of when retrieval runs.
Next story · Build · 2d ago
Filtered queries on Tiger Data's alpha PRISM index can return fewer rows than LIMITConfirmed2 publishers
The baseline filters on one number. Sayora cuts each page into 1,200-character chunks with 200 characters of overlap and embeds them with OpenAI's text-embedding-3-large at 768 dimensions [4]. On every caller turn it injects the closest chunks that score above a cosine similarity of 0.36 [5]. For the caller whose plumber's repair was still leaking, the closest match was a blog post called How to Find a Reliable Plumber [6]. The post is, admittedly, about plumbers. A similarity score measures closeness of topic and wording, and it cannot check that a page answers the question.
The replacement changes two things at once. The unit of retrieval becomes a whole page with a short description [10], and the ranker becomes Jev, a decision model from TypeSafe [7][9]. Embeddings run over those same descriptions would be the control. Without that run, the 33% to 3% gap cannot be split between the model and the page-level index.
On 168 questions, 3% is about 5 wrong pages and 33% is about 55 [18]. Each question moves the rate by about 0.6 points [19].
The questions were model-written. Claude Opus 5.5 read the page text, never the descriptions, and wrote questions the way callers talk [13]. Claude Haiku 4.5 wrote the descriptions, each with a title, a path, a 30-word summary and a few likely questions, at about 100 tokens a page [12]. Sayora also removed pages it said a page picker should never see [17]. Individual property listings went because a feed already covers them. Near-identical "electrician in [town]" landing pages went because a coverage list answers "do you cover X?" better than a choice between 31 copies [17]. That left 403 pages across the four sites [20].
For 3% to transfer, a site needs pages that each cover one topic, which Sayora says describes a small business site [16]. A 30-word summary also has to be enough to tell two pages apart. And callers have to ask questions like the ones Opus wrote.
Sayora's post says of the "none of these" option: "That last part turned out to matter more than the ranking." [14] The post as supplied cuts off before it reports how often "none" was the right answer, so the size of that effect is not in the record.
The price holds up on a back-of-envelope check. At about 100 tokens a page, a 100-page site is roughly 10,000 tokens of options per lookup. At the listed input rate that is about 0.04 cents [21], close to the roughly twentieth of a cent Sayora quotes [15].
What to watch
- A control run of embeddings over the same Haiku-written page descriptions, which would show how much of the 33% to 3% gap belongs to the model.
- Published results on unanswerable questions: how often 'none of these' was chosen correctly, and how often it was chosen when the page existed.
- Results on sites near or above the 255-option limit, where one choice question can no longer hold every page.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence48
- Adoption
- Not scored
- Hype gap+10
- Incentives40
- Confidence50
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
In a test at the start of October, Sayora's production search handed the LLM a wrong page on a third of 168 caller questions.
- [2]
Jev handed the LLM the wrong page 3% of the time, against 33% for Sayora's search.
- [3]
Jev put the right page first 88% of the time, against 70% for Sayora's embeddings.
- [4]
For larger sites, Sayora cuts every page into 1,200-character chunks with 200 characters of overlap and embeds them with OpenAI's text-embedding-3-large at 768 dimensions.
- [5]
On every caller turn, Sayora embeds what the caller said (plus a version with the previous turn) and injects the closest chunks scoring above a cosine similarity of 0.36.
- [6]
For a caller asking who to contact about a plumber's leaking repair, the closest match according to Sayora's production embeddings was a blog post called How to Find a Reliable Plumber.
- [7]
Jev is a decision model that does not generate text; it takes some state and typed questions and returns a choice with a probability for every option.
- [8]
A Jev choice question takes up to 255 options and a request up to 64k tokens.
- [9]
TypeSafe launched Jev on 15 September; it costs $0.042 per million input tokens and output is free.
- [10]
Sayora keeps pages whole and writes each a short description. The receptionist calls a lookup tool when it hits something it does not know; the tool asks the model which page answers what the caller said, with 'None of these' as an option. The best page or two go to the LLM, and on 'none' the receptionist says it does not know and passes the question on.
- [11]
The test used four customer sites: a bar and restaurant with 22 pages, an estate agent with 74, an electrician with 75 and a multi-trade maintenance company with 232.
- [12]
Claude Haiku 4.5 read every page once and wrote a description: the title, the path, a 30-word factual summary and a few questions a caller might ask, about 100 tokens a page.
- [13]
Claude Opus 5.5 read the page text, never the descriptions, and wrote questions the way callers actually talk.
- [14]
Sayora's post says of the 'none of these' option: "That last part turned out to matter more than the ranking."
- [15]
On a 100-page site, a Jev lookup costs about a twentieth of a cent.
- [16]
Sayora says a small business's website is not a pile of documents: someone who knows the business has already organised it into pages, and each page is about one thing.
- [17]
Sayora removed individual property listings, because a feed already covers those, and near-identical 'electrician in [town]' landing pages, because a coverage list answers 'do you cover X?' better than a choice between 31 copies of the same page.
- [18]
3% of 168 questions is about 5 wrong pages; 33% of 168 is about 55.
- [19]
Each of the 168 questions moves the rate by about 0.6 percentage points.
- [20]
The four test sites total 403 pages.
- [21]
A 100-page site described at about 100 tokens a page is roughly 10,000 input tokens, about $0.00042 or 0.04 cents at $0.042 per million tokens.
- [22]
The 232-page test site leaves 23 options of headroom under the 255-option cap.
Sources
2 independent publishers whose own reporting we read for this story.
- dev.toI logged every source ChatGPT used to recommend a B2B vendor. Here's what it actually reads.
1 article · October 15, 2026
- Jev handed our LLM the wrong page a tenth as often as our embeddings (20 minute read)
sayora.ai
1 article · October 15, 2026
Topics and entities
Topics
- LLM Evaluation
- AI Voice Agents
- Retrieval-Augmented Generation
- Embeddings and semantic search