Topic
Metrics describing how fast an AI model generates output tokens and how many requests it can serve concurrently, used to compare inference performance.
OpenAI's Ultrafast tier bills GPT-6.1 Sol at six times its standard per-token rate to return the same model's output faster. Teams now have to pick, request by request, which agent and interactive calls are worth that much for a shorter wait.
Reality
Cloudflare released Clef, open-weight 9B and 27B models that return a probability for each allowed option in a single forward pass. An agent's routing or escalation call becomes a score it can threshold, and so far the only latency figures come from Cloudflare.
Reality
Inception's diffusion model is the fastest endpoint in the cheap tier on both figures, but OpenRouter's median sits at less than half the vendor number and only about 15 percent above Gemini 3.5 Flash-Lite's measured 382 tok/s.
Reality
TypeSafe's Jev answers typed questions in one forward pass with no token stream, and a dev.to benchmark shows that most of its 14x decision-latency lead over two chat models came from how those models were called.
Reality
The vendor's sidecar reads an agent's execution trace and blocks each tool call before it runs. Check Point says the check adds under 100 milliseconds because the expensive context work happens while the agent is still working.
Reality
A dev.to writeup instrumented 1,180 local requests over 24 hours and counted 214 cold model loads, including a summarizer on a 10-minute cron that came up cold every time.
Reality
Flare and Sunburst share identical token rates of eight dollars in and thirty out per million, so what a picture costs is set by the quality tier and by however many tokens the model decides to spend.
Reality
An arXiv preprint treats best-of-N selection and inference latency as one calibration problem, and its Multi-Sequence Verifier scores the whole candidate pool in one pass so a streaming version can stop decoding early.
Reality
A dev.to harness put five coding tasks through four APIs and every run passed its verifier on the first attempt, so the only thing left to compare is clock time, where one trial per cell sets a fragile order.
Reality
Provider prompt caches bill a reused prefix at roughly a tenth of input, so an agent that rewrites its own history to save tokens forfeits the discount and pays to re-prefill everything ahead of the edit.
Reality
A dev.to writeup put caption-driven diffusion TTS against a Style-Bert-VITS2 baseline on the same GPU slice and found a fixed overhead that alone exceeds the baseline's entire generation time, so the model moved offline.
Reality
An arXiv evaluation across OpenAI, Anthropic and Google reports that naive full-context caching can raise latency, while excluding dynamic tool results gives more consistent gains.
Reality
OpenAI's Ultrafast mode for GPT-5.6 Sol, running on Cerebras, is up to 14 times faster than standard processing. The company is already using it for its own incident response, which cuts both ways.
Reality