Skip to content

Topic

Local and on-device inference

Running capable models on consumer GPUs and workstations, including quantization and memory budgeting.

Current stories

buildOne report1 publisher

250 Japanese labels put a 310M encoder 12 points ahead of a hosted decision API

On three Japanese tasks scored against the same 250 frozen rows, a fine-tuned encoder won topic classification by 12 points and tied both polarity tasks, while every zero-shot open-source model in the run lost all three.

Publishers:dev.to

Reality

Evidence58
Adoption15
Hype gap−5
Incentives45
Confidence52
buildOne report1 publisher

Open-weight moderation models land inside the proprietary error band on Bluesky posts

A seven-model comparison on real Bluesky posts reports open-weight sensitivity of 81 to 97 percent against 72 to 98 percent for the proprietary APIs, with the error direction reversing between rudeness and threats.

Publishers:arxiv.org

Reality

Evidence58
Adoption
Not scored
Hype gap+30
Incentives40
Confidence55
buildOne report1 publisher

Unsloth's 10% quant claim is really about which machines can run a 27B model

Dynamic 3.0 ships Qwen3.8-27B GGUFs from 6.2GB up, with an unreproduced accuracy claim attached. The number that matters is the one that decides where the file fits.

Publishers:runtimewire.com

Reality

Evidence34
Adoption45
Hype gap+28
Incentives74
Confidence41
productOne report1 publisher

A 27B laptop model scores like a rented one, and thinks three times as hard to do it

Alibaba's Apache-2.0 Qwen3.8-27B fits in about 17GB and matched near-frontier scores, per Artificial Analysis. It also burned 3.7x the median output tokens getting there.

Publishers:thenextweb.com

Reality

Evidence62
Adoption64
Hype gap+18
Incentives60
Confidence55
buildOne report1 publisher

A 27B Apache-2.0 model in 17GB makes local inference a wiring decision, not a demo

Qwen 3.8 27B ran on a MacBook Pro from a 17GB GGUF and spent 21 minutes on one SVG. Licensing and access stopped being the blocker; latency and KV cache budgeting became the job.

Publishers:dev.to

Reality

Evidence48
Adoption58
Hype gap+18
Incentives62
Confidence52
buildConfirmed6 publishers

Meta's real announcement is the split: 30B on your GPU, everything else behind the API

Muse Glimmer ships as Apache 2.0 weights sized for a 24GB card. Muse Spark 1.2 stays on Muse Code and the Meta Model API. Plan capacity for two tiers, not one.

Perspective Coverage

6 publishers
Builder
Builder 38%
Operator
Operator 31%
Investor
Investor 31%

Reality

Evidence64
Adoption48
Hype gap+18
Incentives78
Confidence70