llama-modchip: a 284B model in 57GB of RAM, no GPU

TLDR: llama-modchip is an open source patch set for llama.cpp that runs DeepSeek-V4-Flash, a 284B mixture of experts, in 56.56 GiB of system memory with no graphics card. It enables a desktop with 64GB of DDR5 to hold a model that previously would not fit, at 9 to 15 tokens per second. The difference between llama-modchip and regular llama.cpp is a 1.71 bit weight format plus rewritten matrix kernels, and the cost is that only their patched binary can load the files.

*Caption: The project is eleven days old: 659 stars, 35 forks, Apache 2.0. Screenshot: GitHub.*

The repository went up on October 29 and reached 437 points and 194 comments on Hacker News by November 8. The version that matters is 2.4.0, pushed today.

The wall it breaks

A standard UD-IQ1_S build of DeepSeek-V4-Flash is 76.87 GiB, or about 82.5 gigabytes. A 64GB Linux machine has roughly 62.5 GiB of usable RAM. The model does not fit, and the failure is not subtle: on the unpatched runtime it either runs out of memory or swaps so hard that decode collapses to 5.91 tokens per second.

llama-modchip gets the same checkpoint down to 56.56 GiB, about 60.73 gigabytes. With a 256K context loaded, peak resident memory sits at 57.61 to 58.9 GiB. Zero swap, one machine, no GPU.

*Caption: The project’s own headline numbers. The one that matters is the last: zero GPUs required. Screenshot: GitHub.*

The model is 284.33B total parameters with 13.17B active, spread across 43 attention layers and 256 routed experts. By default every layer evaluates all six routed experts without pruning, which is why the project calls that configuration lossless.

What changed under the hood

Two things.

The weights are re-encoded into a format the project calls Fractal-64 at 1.71 bits per weight. That is what removes the 20 gigabytes.

The kernels are rewritten so the matrix work during prompt processing runs across registers and cache in columns of eight, four, three and two rather than one at a time. On a 16-core Zen 5 chip that took prefill from 25.0 to 44.36 tokens per second, a 77 percent increase, and decode from about 6 to between 9 and 10.

There is also a speculative decoding trick with an unusual name: a Stockfish-derived drafter of 1.64MB that lives in L3 cache and proposes tokens for the main model to accept or reject. With it, decode lands between 11.5 and 14.7 tokens per second.

*Caption: The configuration table. The top row is the unpatched build: out of memory or swapping on 64GB, at 5.91 tokens per second when it runs at all. Screenshot: GitHub.*

The quality claim, read carefully

This is where the project is honest and also where the headline can mislead.

The default configuration, all six experts, is bit-exact against the patched reference: a KL divergence of 0.0000 and 100 percent top-1 and top-3 agreement. That is a strong claim, and it means the lossless path is lossless relative to the same 1.71 bit checkpoint.

The four faster profiles trade quality for speed, and the table states the size of the trade. The “sweet spot” at Top-5 with quadrature costs 0.0042 nats, about 0.42 percent higher perplexity, and keeps 95.8 percent top-1 agreement. The Turbo configuration at Top-4 reaches 13.0 to 16.0 tokens per second with PVS and gives up between 0.0281 and 0.0694 nats.

The caveat is in the comparison itself: these numbers are against the unpatched runtime on the same 1.71 bit file, not against the original 16-bit model. The question “how much worse is this than DeepSeek at full precision” is not answered by the table, and at 1.71 bits per weight some answer exists.

*Caption: The supported checkpoint list, with disk and memory columns rather than marketing claims. Screenshot: GitHub.*

The format lock-in

The Fractal files use a custom tensor type, GGML_TYPE_Q2_0, with a type ID of 40. Stock llama.cpp, Ollama and LM Studio will all reject them with an unknown-type error.

That is the real cost of the project. You are not running standard GGUF on a faster runtime. You are running a fork’s private format, which means every future upgrade path goes through that fork, and a model you download from anyone else will not load without transcoding.

The project ships a transcoder for BF16, Q4_0 and UD-IQ1_S checkpoints, so the pipeline is at least documented rather than hand-waved.

What fits on what

*Caption: Four hardware classes and what each one can hold. The 64GB desktop row is the headline, the laptop row is the useful one. Screenshot: GitHub.*

16 to 32GB laptop or mini PC: a 30B mixture of experts, Nemotron-3.5-30B-A3B, at 14.35 GiB in 3.75 bits. 25 to 60 tokens per second, with a 256K context costing 432MB of KV cache.
64GB desktop: the full 284B DeepSeek-V4-Flash. 44 tokens per second prefill, 9 to 15 decode.
Hybrid, 32 to 64GB plus a 16GB GPU: the attention layers on the card, the routed experts in system RAM. 80.80 tokens per second.
Eight-channel Threadripper: 111.62 tokens per second on the 30B, because eight memory channels feed eight cores properly.

*Caption: The Hacker News discussion the same day: 437 points, 194 comments, mostly arguing about what 1.71 bits per weight costs you. Screenshot: Hacker News.*

Why this matters more than the benchmark

For two years the local AI conversation has been about VRAM. Buy the card, fit the model, count the gigabytes. This project is a reminder that the actual constraint is memory bandwidth and capacity, and that a CPU with eight channels and a lot of cheap DDR5 is a legitimate second path.

A 64GB desktop costs a fraction of a 32GB graphics card at current prices. It will not run an interactive chat at 60 tokens per second, but 9 to 15 is usable for agent work that is not latency critical, and it holds a model class that no single consumer GPU can address at all.

The other lesson is about quantization. At 1.71 bits per weight, DeepSeek-V4-Flash fits in 57GB. That is the whole trick, and it is a direction the community has been walking for a year: shrink the weights, not the ambition.

What to watch

The project is eleven days old with 35 forks and 11 open issues. There is no independent replication yet of the 44 token per second prefill figure, and the benchmarks come from a single machine with a specific CPU. The bit-exact claim is the one to test, because it is checkable: run their eval, then run the same checkpoint unpatched on a machine with enough RAM to hold it.

Until someone does that, treat the numbers as the author’s measurements, clearly documented, on one machine.