Topic
The practice of testing and benchmarking language models to measure accuracy, reasoning, and performance drift over time.
Grafana Labs proposes turning agent evaluations into SLIs, with a 95% fulfilment target over 30 days leaving a 5% error budget, InfoQ reports. Grafana argues latency and token metrics cannot show an answer was wrong, and budget burn exposes slow drift.
Reality
Sayora's receptionist test found Jev gave the LLM a wrong page on 3% of 168 questions, against 33% for its embeddings. The figure is Sayora's own, from four small-business sites, and transfers only to sites built from single-topic pages.
Reality
Artificial Analysis scores Meta's Muse Wonder Full at 56, two points behind Claude Opus 5.5 at 58. Meta's own model card has Full ahead on long-running agentic tasks, so the right pick depends on how closely your workload resembles those tasks.
Reality
Coinbase says a Qwen3.5-9B model post-trained for under $100 beat Claude Opus 4.5 on its Onramp fraud benchmark. The figures are Coinbase's on private data; the reusable parts are the label-based reward and the tier-only, block-only harness.
Reality
Playtomic ran eight newly released models through its Slack agent Dory on 39 real questions, graded by two judges from different labs. The company says public rankings measure generic skills, so other teams would have to rerun the method on their own questions.
Reality
TII's Falcon-Emirati-7B scores 84.83% on Alyah, an Emirati-dialect benchmark its own researchers helped develop. The model is small enough to trial cheaply for Gulf-facing chat, though every published score so far comes from TII's own testing.
Reality
Microsoft Research's Webwright lifted GPT-5.4 from 33.5% to 60.1% on 200 long web tasks by having it write browsing code from a terminal. Because the baseline steered by screen coordinates, the result compares writing code with clicking pixels.
Reality
Plain RAG and GraphRAG got none of 31 counting and superlative questions right on a 100-question TigerGraph hackathon benchmark, an entrant reports. A COUNT from the graph, set beside the evidence actually read, shows when an answer is incomplete.
Reality
Qwen2.5-3B, wired to a local Wikipedia index, scored 52% on 150 post-cutoff questions it answers none of unaided, up from 33%, in a dev.to author's tests. Each fix targets a measured 3B failure, so a zero-shot 7B gained only 9 points from them, and the two readers' confidence intervals overlap.
Reality
Open-source tool runtape traced an agent's unrequested invoice forward to one sentence in a tool result, 10 of 10 reruns with it against 0 of 10 without. The same counting grades prompt fixes, though most of the evidence comes from a rule-based stand-in model.
Reality
LiveNerf reruns 78 calibrated questions against Claude Opus 5.5 every day, with frozen prompts and a pinned Claude Code CLI. That gives teams building on the model a dated launch baseline to test a suspected regression against.
Reality
A TigerGraph hackathon entry raised exact match from 67% to 99% on 100 questions, with the agent alone accounting for 3 of the 32 points. The rest needed a parser that made Wikipedia infobox fields countable in the graph.
Reality
Bench on the Clocktower, a social-deduction benchmark run for 300 games per model, finds agents playing Good fall for coordinated deception by Evil agents. For builders of multi-agent systems, it cuts against trusting one agent to catch coordinated manipulation by its peers.
Reality
Eval harness agents-md-evals found 25 of 26 assertions passed identically with or without a 755-line AGENTS.md file. The finding rests on that one file, and it assumes the agent loaded the file in the first place.
Reality
Claude Code's new plugin eval showed one developer's skills firing in 5 of 9 relevant runs once all 89 were loaded, down from every run with one skill. The test covers three prompts in one project, but it gives teams that keep rules in skills a way to measure how often those rules get consulted.
Reality
AWS made CloudWatch Omni generally available with 17 built-in evaluators that score an AI agent's answers on live production traffic. For operators, the job moves from confirming an agent is running to deciding whether an automated score is good enough to gate a release.
Reality
Simon Willison says Claude Opus 4.5 and GPT-5.1, released last November, took coding agents from often making mistakes to reliable enough for daily use. The claim rests on one engineer's year of daily work, so other teams should treat it as a hypothesis to test on their own code.
Reality
Grok 4.20 followed an injected wrong step about five times as often with reasoning mode on, in a 15-question Kaggle benchmark entry. On the Anthropic test it uses, that makes the trace a better record of how the model answered and a weaker guard against a bad step.
Reality
Blog vs Bytecode, a 28-item Kaggle benchmark, graded empty proxy responses as wrong and scored DeepSeek-R1 at 17% until a second gateway showed 100%. Once capture was fixed, frontier models lost points by flagging sound code, while a small Gemma model missed most of the planted flaws.
Reality
One developer's support-agent eval suite fails two of its five property bars even though its pooled score is 0.919. The author argues checks like these are what OpenAI lacked when a sycophantic GPT-4o update shipped in April 2025 and was pulled in four days.
Reality
Build · September 21, 2026 · One report1 publisher