Topic
Independent and vendor-run testing of AI models and systems, including benchmarks like MLPerf and ARC-AGI, and checks on speed, cost and quality claims.
Tom's Hardware tested Nvidia's RTX Spark in a 128GB Surface Laptop Ultra against an M5 Max MacBook Pro on agent task time. Its headline calls local AI a strength and gaming a weak spot, but the text reviewed here ends before the AI timings.
Reality
Janas, a C engine by Maurizio Cammalleri, reports 37 tok/s on a 48.4 GB model with 32 GB of RAM, against 2.8 on llama.cpp. Speculative drafting supplies 13 of those 37, so the like-for-like figure is 24 tok/s.
Reality
MLCommons' MLPerf Inference v6.1 adds tests that time a whole RAG pipeline and a 1,007-turn coding-agent replay. Those scores help a buyer size hardware only where the buyer's own pipeline spends its time the way the four-model reference does.
Reality
OpenAI's GPT-6 Astra scored 99.9% on ARC-AGI-3 through a proprietary harness but 62.7% on the benchmark's neutral one, Live Science reported. Because rival labs cannot use that harness, the lower score is the one that can be checked against other models.
Reality
Anthropic's Claude Sonnet 5.5 beats Opus 5.5 at coding for half the per-token price, on its own tests and on Artificial Analysis's. At max effort it writes 60% more tokens per task, so moving coding work down a tier saves nearer a fifth than a half.
Perspective Coverage
4 publishersReality
OpenAI's own benchmarks put GPT-6 Sol and Luna at half the list price and a fraction of a rival's cost per finished task. For defenders, the thing getting cheaper is autonomous tool calls into business systems.
Reality
RoboCurve scored 19 successes in 20 physical attempts on a pair of I2RT YAM arms, at roughly 2.1k output tokens a run. The humanoid pickup described in the same account happened in a simulator.
Reality
Artificial Analysis scores the new Xiaomi model first among open weights and twelve points behind Claude Opus 5.5, and the cheaper Flash tier is the one an operator should put in front of a real queue.
Reality
Bespoke Nimble's LoRA fine-tune of Qwen3.5-9B scored 90% against Jev's 93% on an eval it curated itself, and five more replications landed at scales between 421M and 35B parameters. No standard benchmark exists for the category yet.
Reality
Artificial Analysis now reports how often a provider or model refuses coding-benchmark work on safety grounds. For a buyer the useful part is whether the agent stopped there or switched models and finished.
Reality
Vals keeps its test materials private and charges the model developers it scores, and Andreessen Horowitz has now put $40 million behind that arrangement. Buyers reading the numbers cannot run the test themselves.
Reality
In a paper posted on 13 September, nine models across the Claude, GPT and Gemini families edited already-optimal EffiBench solutions in every trial. One added sentence in the prompt recovered 20 refusals.
Reality
Give Atlas a few images and it generates views a camera never shot, then exports depth, point clouds, and 3D splats you can move through. The invented geometry looks as measurable as the observed, a gap that matters more to a robot than a filmmaker.
Reality
NVIDIA's submission runs the same Qwen3.6-27B as the llama.cpp reference on the same Jetson board and finishes 6.4x sooner. Most of the gap comes from prompt tokens the runtime never has to prefill.
Reality
A dev.to post gives every failed coding-agent run one of four labels and scores skill only over the runs where the harness stayed healthy. SSH drops and disk-full errors get published as their own rates.
Reality
Korea's Dokpamo program gives the benchmarking firm's index 25 of the 100 points that decide which teams advance. In the results published on August 27, the team that led that component finished last of the four.
Reality
A dev.to walkthrough proposes MemoryBench: four tasks, ten metrics and a million-fact store, so buyers can measure recall, latency and write cost themselves before signing.
Reality
Stanford's AI Index puts a 142-fold parameter cut and a more than 280-fold price cut behind one fixed MMLU threshold. That narrows where building your own still pays, and it lands in a state-law count that doubled in a year.
Reality
OpenAI's invite-only Ultrafast tier runs the same GPT-5.6 Sol up to 14 times quicker, while Google halves Gemini Flash pricing until December 31. Latency is now its own budget line.
Perspective Coverage
3 publishersReality