Edge0's 35B would need 22 GB/s of SSD bandwidth without the file cache
Edge0's Apache-2.0 engine reports a 35B model peaking at 2.9 GB of RAM on a 24 GB Mac mini. Work the decode rate against the active set and most of those expert reads have to be coming from the operating system's file cache.
The Engineer · Build desk
What happened
- Samuel Zeng open-sourced Edge0 on September 10th: an inference framework for running large mixture-of-experts models without loading their full weight sets into memory, plus two models.
- The engine leaves the whole 4-bit checkpoint on the SSD and memory-maps the expert weights, reading only the experts the router picks, 4 of 256 per token on the 35B.
- Edge0's published 35B benchmark reports 2.9 GB of peak active memory on a Mac mini with an M4 Pro and 24 GB of unified memory, decoding at 14.9 to 17.7 tokens per second.
- The repository lists macOS on Apple Silicon as its supported platform, tested on M1 through M4, and the iPhone result stays a demonstration with no iOS build or iPhone benchmark procedure published.
- In Edge0's own OpenCompass testing the 35B pipeline averaged 79.2 across five benchmarks against 83.2 for the full-precision base, and the 8B averaged 69.9 against 72.7.
Why it matters
- What it limits constraint The RAM saving comes out of RAM you already own. The deployment target that gets the reported numbers is a machine with tens of gigabytes free for the file cache.
- What it costs cost The SSD is in the read path for every token of every session, and nobody has published wear figures for that duty cycle.
- Who has to decide decision Anyone sizing a context window is sizing RAM. The 8B's cache went to about 3.3 GB at roughly 3,300 tokens against a 1 GB short-context peak, so the headline memory number is a floor to plan above.
- What doesn't add up contradiction dev.to gives the 35B checkpoint as 19.6 GB from the model cards; runtimewire cites the repository at roughly 23 GB. At 23 GB the checkpoint no longer fits the memory left free on the benchmark machine, and the drive is genuinely in the loop.
Next story · Build · yesterday
NVIDIA's Bolt packs 74B parameters into a claimed 24GB by storing them at 2 bitsConfirmed12 publishers
Three billion active parameters at 4 bits is about 1.5 GB of expert weights per token, by the dev.to write-up's own calculation, which puts roughly half the 2.9 GB peak in the routed experts and the rest in shared weights and cache [14]. Multiply 1.5 GB by the reported decode rate and a cold read of the full active set on every token would need 22.4 to 26.6 GB/s [15]. No consumer SSD sustains that, as the same write-up notes [9]. So most expert reads are being served from the operating system's file cache, and dev.to treats "peak active memory" as the process resident set, with the cached memory-mapped pages sitting outside it [26].
On the benchmark machine there was room for that. Subtract the 2.9 GB resident set from 24 GB of unified memory and 21.1 GB is free, above the 19.6 GB the model card gives for the 4-bit 35B checkpoint [28]. A 24 GB Mac is an odd place to show that a 35B model does not need much RAM. After warmup the checkpoint can sit in the file cache, and the drive only has to serve what the cache evicts [26].
The prefetch design is careful engineering. Edge0 trains a small prerouter network to predict which experts the next step will need, so the runtime can start loading them while the current computation is still running, and frequently selected experts stay cached while less-used weights stay on disk [16]. Edge0 also ships Recover-LoRA, adapters trained to recover quality lost when the base models are cut to 4-bit precision [17]. Those evaluation averages were produced by Edge0 rather than an independent evaluator [10], and the repository calls tool use, multi-step planning and long-horizon autonomy weak [23].
For the 2.9 GB figure to transfer, the target machine needs enough free memory to keep most of the checkpoint cached. dev.to says a Mac with less memory cannot, and flags the whole reading as unchecked against the code [26]. There is no phone build to test it on either way. According to runtimewire, Zeng says the demo used 1 to 2.5 GB of peak memory, with no cloud connection and no remote GPU [27]. The repo's only chip-level mention of an iPhone is a troubleshooting entry: garbled, mixed-language output on Apple A18 and A18 Pro, traced to an older mlx [18]. The reposts that circulated an "iPhone 15 Pro and newer" floor got it from somewhere other than the README or the model cards [12].
Storage-backed inference has prior art. Apple's "LLM in a Flash" research described loading parameters from flash on demand to run models larger than available DRAM, and open-source projects including Swiftlet have shipped Swift and Metal implementations that stream sparse experts from an iPhone's storage, according to runtimewire [21]. What you can install today is a Python package that wants Python 3.10 or newer on an Apple Silicon Mac [19]. Budget the 19.6 GB on the drive you are willing to have read on every token, and note that the project does not say what months of that duty cycle do to a drive [20].
What to watch
- An iOS build or a published iPhone benchmark procedure in the repository, which would make Zeng's demo reproducible by someone other than Zeng.
- A memory and throughput measurement on an Apple Silicon Mac with less free RAM than the checkpoint: does decode hold near 14.9 tok/s when the drive, not the cache, serves each token?
- An independent OpenCompass run against Edge0's own 79.2 average for the 35B pipeline.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence38
- Adoption12
- Hype gap+48
- Incentives55
- Confidence52
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Samuel Zeng (@SamuelZengML) open-sourced Edge0 on September 10th, releasing two language models and an inference framework designed to run large mixture-of-experts models without loading their full weight sets into memory.
- [2]
Edge0-AI open-sourced Edge0, a streaming inference engine under Apache 2.0, plus two preview models built on open sparse MoE bases.
- [3]
Edge0-35B-A3B is built on Qwen3.5-MoE 35B-A3B, with 40 layers and 256 experts.
- [4]
Edge0-8B-A1B is built on inclusionAI's Ling 3.0 tiny, a hybrid pairing multi-head latent attention with MoE, 24 layers and 128 experts, about 1.2B active out of 7.9B.
- [5]
The whole checkpoint stays on the SSD, the expert weights are memory-mapped, and only the routed experts are read as each token comes through; the project's line is that peak memory is bounded by the active set rather than by the parameter count.
- [6]
The published 35B benchmark reports 2.9 GB of peak active memory on a Mac mini with an M4 Pro and 24 GB of unified memory.
- [7]
On the M4 Pro test machine, Edge0 reports 14.9-17.7 tokens per second for the 35B model and 23.9-25.3 tokens per second for the 8B model.
- [8]
The 8B release used 1 GB of peak active memory for short contexts, and Edge0's documentation warns memory rises with context length: a roughly 3,300-token context pushed the 8B model's cache requirements to about 3.3 GB.
- [9]
dev.to calculates that at 15 to 18 tokens a second, pulling a full 1.5 GB active set cold off the drive per token would need over 20 GB/s, and no consumer SSD does that.
- [10]
In Edge0's own OpenCompass testing the 35B pipeline averaged 79.2 across five benchmarks against 83.2 for the full-precision base, and the 8B averaged 69.9 against 72.7; the results were produced by Edge0 rather than an independent evaluator.
- [11]
The Edge0 repository currently lists macOS on Apple Silicon as its supported platform, with an MLX backend tested on M1 through M4 hardware, and Edge0 has not published an iOS build or an iPhone benchmark procedure in the repository.
- [12]
The project says the MLX backend runs on macOS with Apple Silicon (M1 through M4) and that no other platform is supported yet, CUDA included; the "iPhone 15 Pro and newer" floor circulating in reposts appears nowhere in the README or the model cards.
- [13]
The router calls 4 of the 256 experts per token, about 3B active parameters out of 35B.
- [14]
dev.to calculates that 3B active parameters at 4 bits is about 1.5 GB, so roughly half of the 2.9 GB peak is the routed experts and the other half is shared weights plus cache.
- [15]
Reading a 1.5 GB active set cold from storage on every token would require 22.4 GB/s at 14.9 tokens per second and 26.6 GB/s at 17.7 tokens per second.
- [16]
Edge0 trains a small "prerouter" network to predict which experts the next step will need, allowing the runtime to begin loading them while the current computation is still running; frequently selected experts remain cached while less-used weights stay on disk.
- [17]
Edge0 ships what it calls Recover-LoRA, a set of adapters trained to recover quality lost when the base models are reduced to 4-bit precision.
- [18]
Garbled, mixed-language output on Apple A18 and A18 Pro is listed in the repo as a symptom of an older mlx, and it is the only chip-level mention of an iPhone anywhere in the repo.
- [19]
Install is a Python package, and the project wants Python 3.10 or newer on an Apple Silicon Mac.
- [20]
The 35B checkpoint sits on the SSD for the whole session and each token is a storage or cache read; the project does not say what that does to a drive over a few months.
- [21]
Apple's "LLM in a Flash" research described loading model parameters from flash memory on demand to run models larger than available DRAM, and open-source projects including Swiftlet have since shipped Swift and Metal implementations that stream sparse model experts from an iPhone's storage.
- [22]
Edge0 also showed the model generating game logic, structured 3D scenes and interactive web experiences on-device.
- [23]
Both models are previews, and tool use, multi-step planning and long-horizon autonomy are called weak.
- [24]
Both models ship as 4-bit checkpoints, 19.6 GB and 4.55 GB on disk by the model cards.
- [25]
The 35B checkpoint occupies roughly 23 GB of storage, according to Edge0's repository, while the 8B release requires about 4.2 GB.
- [26]
dev.to's reading is that most expert reads must be served from the operating system's file cache, and that "peak active memory" counts the process's resident set while the mmapped pages the OS keeps cached sit outside it; a 24 GB Mac mini can hide a large share of the 19.6 GB checkpoint in that cache and a Mac with less memory cannot. The author says he has not checked this against the code.
- [27]
Zeng says the iPhone demo uses 1-2.5 GB of peak memory, with no cloud connection or remote GPU.
- [28]
On the benchmark machine, 24 GB of unified memory minus the 2.9 GB resident set leaves 21.1 GB free, which exceeds the 19.6 GB model-card checkpoint size.
Sources
2 independent publishers whose own reporting we read for this story.
- dev.toEdge0 streams MoE experts off SSD to fit 35B in 3 GB
1 article · September 11, 2026
- runtimewire.comEdge0 open-sources a 35B model runtime that streams weights from storage
1 article · September 9, 2026
Topics and entities
Topics
- Storage-backed inference
- Quantization formats
- Open-Weight Releases
- Mixture of Experts Architectures
- On-Device LLM Inference
Entities
- MLX
- Apache License 2.0
- Qwen3.5-MoE 35B-A3B