Skip to content

Topic

Vendor-Reported Benchmark Claims

Performance comparisons published by interested parties, where harness, tooling, prompting and concurrency determine applicability.

Current stories

buildOne report1 publisher

Trane put a tool gateway between its HVAC agent and telemetry from millions of assets

AWS says Trane's engineers cut a 20-minute dashboard workflow to a 20-second question in three to four weeks. The multiple rests on timings Trane ran with its own technicians, and the durable part is where the tool calls execute.

Publishers:aws.amazon.com

Reality

Evidence34
Adoption26
Hype gap+32
Incentives82
Confidence56
buildOne report1 publisher

Every Qwen3.8-Omni-Flash workflow ends in text your own tools have to execute

Alibaba's new omni model takes text, images, audio and video across a million-token context and answers in text plus function calls. Rendering and token budgeting stay in the stack you already run.

Publishers:runtimewire.com

Reality

Evidence56
Adoption20
Hype gap+26
Incentives78
Confidence55
buildOne report1 publisher

Gemini 3.8 Live Extended Thinking Can Send turnComplete While the Model Is Still Reasoning

Google's September 15 release pairs a fast speech-to-speech model with one that narrates its own reasoning aloud while tools run. Clients now have to read interactionStatus to know when a turn is actually over.

Publishers:dev.to

Reality

Evidence30
Adoption20
Hype gap+12
Incentives60
Confidence35
buildOne report1 publisher

Stripped GLM-5.3-Flash weights show what Z.ai's MIT license permits

DeAlignAI's downloadable FP8 build is the license working exactly as written, while its self-reported 320-of-320 HarmBench run remains unchecked by any outside researcher and measures compliance, not capability.

Publishers:runtimewire.com

Reality

Evidence46
Adoption20
Hype gap+18
Incentives72
Confidence55
productOne report1 publisher

Ox Alpha was GLM-5.3-Flash, and the number that decides displacement is 18 billion

Z.ai says the model runs at a tenth the cost of its last one. The comparison an operator needs is against the API invoice they already pay, and the release does not make it.

Publishers:siliconangle.com

Reality

Evidence32
Adoption24
Hype gap+38
Incentives72
Confidence44
buildOne report1 publisher

OpenAI moves its inference onto a chip nobody outside OpenAI can buy or benchmark

Jalapeno, co-designed with Broadcom, starts landing in OpenAI's own data centers by the end of 2026. The only efficiency figure is OpenAI's, and the part is not for sale.

Publishers:dev.to

Reality

Evidence28
Adoption10
Hype gap+22
Incentives76
Confidence44
productOne report1 publisher

A 27B-parameter agent beat two frontier models at one task, and the task was chosen carefully

Inherent says its Faraday agent reproduced published findings better than Claude Opus 4.8 and GPT-5.5. The interesting number is not the parameter count but what the account leaves out.

Publishers:techcrunch.com

Reality

Evidence20
Adoption12
Hype gap+55
Incentives80
Confidence58
buildOne report1 publisher

Inco AI's DFlash 2: 21% longer accepted drafts for 1.3% latency and 18.5M parameters

The update adds a path selector and a two-tap convolution rather than layers, recovering most of the accuracy that tripling the drafter bought at 15.2% latency, by the vendor's own numbers.

Publishers:runtimewire.com

Reality

Evidence54
Adoption66
Hype gap+16
Incentives74
Confidence58
buildOne report1 publisher

Open weights caught up on finding bugs. They did not catch up on using them.

Z.ai says GLM-5.3 edges Anthropic's restricted Mythos 5 at vulnerability discovery while losing badly at exploitation. On vendor numbers, the defensive half is commoditising first.

Publishers:dev.to

Reality

Evidence20
Adoption18
Hype gap+45
Incentives82
Confidence30
buildConfirmed3 publishers

OpenAI puts latency on the price list: 750 tokens/sec, gated by workload fit

The Ultrafast preview runs GPT-5.6 Sol on Cerebras hardware for a hand-picked customer list. That makes capacity allocation, not model choice, the constraint your architecture has to survive.

Perspective Coverage

3 publishers
Builder
Builder 42%
Operator
Operator 33%
Investor
Investor 25%

Reality

Evidence42
Adoption24
Hype gap+32
Incentives78
Confidence58