kimi-k3-in-c: A 176 KB Engine That Runs a 2.78T-Parameter MoE in 8 GB of RAM

I went looking for something quiet to write about this morning and found a repo that had been sitting on my GitHub trending tab for two days at 2,400 stars. The headline on the README was the kind I keep a mental list of: 2.78 trillion parameters, one CPU, 8 GB of RAM, 176 KB of C. The author is FareedKhan-dev, the project is kimi-k3-in-c, and the technical claim is not “we can sort of run Kimi K3 on a laptop” — it is “we can run Kimi K3 on a laptop and the answer at 8 GB is byte-for-byte the answer at 224 GB.” That single property is the most interesting thing in the codebase, and it is what this post is about.

The first thing to understand is that Kimi K3 is a very large mixture-of-experts model. It is 2.78 trillion parameters across 93 layers, 1.56 TB on disk as shipped, 96 safetensors shards, and 82,432 routed experts. Only 16 of the 896 experts in any layer fire for a given token — about 3.7% of the parameter mass — and the rest of the model is asleep on disk. That asymmetry is the whole reason the project exists. If you treat every parameter as “live” the model is 5.56 TB at bfloat16 and fits on no machine on Earth. If you treat the routed experts as “streamable on demand” the resident set drops by an order of magnitude before you write a single kernel. The author’s number is that the residents-and-streamers split is 113.49 GB, which still does not fit on a consumer laptop — until you stream the dense trunk too, at which point the same model fits in 8.24 GB.

The four reductions

The README is structured as four numbered reductions, and the structure of the reductions is the structure of the project. I am going to go through them because the order is the order in which the engineering decisions were forced:

1. 5,560 GB → 1,560 GB. Every parameter at bfloat16 would be 5.56 TB; the checkpoint as shipped is 1.56 TB because the routed experts already arrive in MXFP4, a microscaling 4-bit float. Each weight is a 4-bit nibble indexing a 16-entry table, every group of 32 weights shares one 8-bit exponent, and the per-weight byte cost is 0.53125 — half a byte plus 1/32. One expert has 33,030,144 parameters, so one expert is exactly 17,547,264 bytes. There are 82,432 of them; that is 1.447 TB, or 93% of the checkpoint. Reduction one is essentially free because the format was already chosen by Moonshot.

2. 1,560 GB → 113.49 GB. The 7% of the checkpoint that is not routed experts is the resident set: 56,743,648,000 parameters at bfloat16, or 113.49 GB. Of that, 108.81 GB is the per-layer dense trunk and 4.70 GB is the embedding table plus the output head. The transition from “model” to “resident set” is the realization that you do not have to load 880 of the 896 experts per layer per token. They are not on the critical path of any single forward pass.

3. 113.49 GB → 8.24 GB. The dense trunk is used on every token, but it is read-only sequential data: layer 0, layer 1, …, layer 92, in the same order on every token. The engine packs the trunk once into a 108.81 GB file where layer L lives at a known offset (a four-minute one-time rewrite) and then reads it as a pinned prefix plus a single ring slot. With a budget of 2.5 GB pinned plus 0.5 GB ring, the resident set is 3 GB, and the peak RSS for the entire 93-layer run is 8.24 GB measured with getrusage. From a 2.78T-parameter model, that is a 675× reduction from the bfloat16 size and a 189× reduction from the shipped checkpoint.

4. The fourth “reduction” is not a memory reduction. It is the property that the output is byte-identical at every memory budget between 8 GB and 224 GB. Twelve budgets spanning a factor of 28, and the token ids are the same in every one. That is the engineering claim the rest of the project exists to defend.

What “byte-identical” actually costs

The author goes to some length to make the byte-identical claim checkable rather than aspirational. Three things I had not seen done this carefully in a small inference engine:

The config reader refuses to guess. The Kimi K3 config has 24 MLA layers and 69 KDA layers, with the layer map being one-based and ending in ..., 92, 93 because both of the last two layers are MLA. A permissive config reader that filled in defaults for missing fields would silently produce a different model — one that still loads, still streams, still emits grammatical English, but has 0 of the 24 global-attention layers. The reader accumulates missing names and exits 2 rather than substituting, and the test suite includes no_layermap.json and bad_layer_index.json to prove the rejection paths fire. I find this reassuring; a model that “works” but is not the model you wanted is the worst failure mode.

The kernels have a floating-point contract. RMSNorm accumulates in double even though every input and output is float; epsilon goes inside the square root rather than outside; the inner dot product is partitioned into four accumulators by i % 4 so a 4-wide vector loop adds the same numbers in the same sequence as the scalar loop. The compiler is told -ffp-contract=off because FMA fusion silently changes the rounding and would break the bit-identical guarantee. A performance change that quietly becomes an accuracy change is the second-worst failure mode after a config that quietly becomes a different model.

The MXFP4 matmul never dequantizes. The naive thing to do with packed 4-bit weights is to decode them into float32 and then call a normal matrix multiply. Pricing that: one expert at 17.55 MB becomes 132 MB once expanded; each token touches 16 experts across 92 layers, so decoding all of them means writing out 194 GB of format conversion per token before a single multiply-accumulate. The matmul reads packed nibbles directly, with one 16-entry E2M1 lookup table and one 256-entry E8M0 scale table, accumulating in double and multiplying by the scale once per group of 32. A scale byte of 255 is the spec’s NaN value, and the engine maps it to 0 — one corrupt byte kills one group of 32 weights instead of turning an entire row into NaN and poisoning every downstream layer. The nibble order is the even weight, the high nibble is the odd weight, and reversing that produces a matrix with every adjacent pair of weights transposed. Statistics are identical; positions are wrong. A verification that checked distributions would pass the wrong matrix. The test compares bit patterns.

The validation ladder has four rungs. The bottom is operation-level fixtures where every test is designed to fail a specific plausible wrong implementation — the SiTU-GLU fixture spans inputs from 0.1 to 1000, because in the near-linear region the bounded tanh is indistinguishable from the identity and a fixture that only tested there would pass an implementation with the caps left out entirely. The next rung is a 13-layer, hidden-128, vocab-256 oracle model with the same tensor graph, which is the smallest model that exercises attention-residual block boundaries (they fire every 12 layers, so a 5-layer model would never see one). The third rung is a per-layer elementwise check against PyTorch, all 93 layers, “worst 0.00x of budget” on every passing layer. The fourth is a full 93-layer forward pass: 163,840 logits compared elementwise, max diff 7.87×10⁻⁶, correlation 1.000000000, and the argmax token (id 2494) agrees with the torch reference on a prompt that decodes to $%&'(.

The author’s caveat about that last rung is worth quoting: “It is agreement on one position of a meaningless prompt: strong evidence about the arithmetic and no evidence at all about output quality.” The README does not paper this over. The first-token run that produces ” Paris.” for the prompt “The capital of France is” is the right answer, but the model is a base model with no chat template, so what follows the period is a JSON-list continuation, not a chat reply.

The interesting design decisions

Three of them jumped out at me, and they are the decisions I expect to influence other MoE engines in the next six months.

First, a pinned prefix is the correct cache for the dense trunk, not LRU. The engine walks layers 0 through 92 in the same order on every token, which is a cyclic scan. A cyclic scan is the pathological case for least-recently-used eviction: by the time layer 0 comes round again, it is the least recently used thing in the cache, so it has always just been evicted. An LRU of 90 slots over a 93-layer cycle achieves a hit rate of exactly zero. Pinning the first N layers gives a deterministic hit rate of N/93, which for N=90 is 96.8%. The README makes the point that the obvious data structure is not merely suboptimal here but wrong in the worst possible direction — returning zero where the trivial approach returns almost one. The author pins 90 of 93 layers at the server preset and the steady-state trunk hit rate is 96.8%, exactly as predicted.

Second, MLA’s KV cache compresses by 53× with no accuracy loss. A normal attention layer with 96 heads stores, for every position, a key and a value for each head, which is 96×320 = 30,720 floats per position per layer. MLA projects the token down into a 512-dim latent plus 64 carried-but-unrotated rope dimensions, caches only that 576-wide slice, and rebuilds the per-head keys and values from it when they are needed. The 53× smaller cache is mathematically identical, and the engine refuses to start rather than discovering the memory limit an hour into a run: context costs about 2.37 MB per position regardless of budget, and the engine computes the available memory up front and exits with “This is a MEMORY limit, not an engine ceiling” if the budget will not hold the request. I like that phrasing. It saves somebody an afternoon of looking for a hardcoded constant that does not exist.

Third, Kimi K3’s Quantile Balancing defeats the expert cache, and the engine proves it. The hot finding in the README’s cache-replay section is that LRU is completely flat from 8 GB to 64 GB: an eightfold increase in cache capacity buys nothing at all over that range, and Belady over the same range climbs from 39% to 62%. The implication is that the weak caching is a property of the model, not a defect in the implementation. Quantile Balancing flattens expert usage across the pool by design, and flat usage is precisely what defeats LRU. With no hot subset, a few gigabytes of arena retain nothing worth keeping, and the right reaction is to spend that memory on the dense trunk instead. The README runs a sweep at a fixed 128 GB budget and finds that giving the trunk memory instead of the expert cache is worth 1.69×. That is a counterintuitive result (RAM is RAM) that only shows up because the workload has no temporal locality at the expert level.

The 1.7× cost curve

The memory ladder is the most useful thing in the README for someone choosing hardware. Twelve budgets, peak RSS measured with cgroup enforcement (MemoryMax=N G -p MemorySwapMax=0, the second flag matters because without it an over-budget rung swaps instead of dying), token times in s/token:

Total GB	Pin layers	Cache GB	s/token	Trunk hit %	GB read/tok	Peak RSS GB
8	0	0.49	32.69	0.0	25.83	8.24
16	3	4.39	32.21	2.8	25.83	16.00
32	11	10.80	31.44	10.3	25.83	31.90
64	27	23.59	28.60	25.4	25.83	63.71
128	60	49.19	29.40	56.5	17.51	128.18
192	90	77.00	21.32	84.7	16.65	191.83
224	90	108.98	19.21	84.7	14.53	223.82

Going from 8 GB to 224 GB takes the per-token time from 32.69 s down to 19.21. That is 28 times the memory for 1.70 times the speed. The jump from 8 GB to 64 GB buys 14%, and the 128 GB rung is actually slower than 96 GB. The reason is that the expert cache does not start to participate until about 36 GB of arena, and adding cache memory at the expense of trunk memory at a fixed budget makes the trunk hit rate drop. The speed is not in the memory; the speed is in the storage bandwidth, and at every budget the engine reads roughly 25.83 GB per token of expert data. The limiting factor is NVMe throughput, which on the test machine (3.2 TB of NVMe, two-socket EPYC 7763, 228 GB RAM) measured 5,373–6,064 MB/s sustained during runs.

The whole-table property is what makes the design land. From 96 GB down to 12 GB the same three tokens come out: 17374,20829,10, decoded as Paris.". Not merely the same ids at every budget but the same right answer.

What is missing or worth watching

The engine is 179,736 bytes — the README’s binary size table lists k3_run at that exact number — and the only dependencies are libm and OpenMP. It targets Linux x86-64 with AVX2 and FMA; AVX-512 is unnecessary; the README explicitly notes that the test machine has no AVX-512 and the engine does not care. The CI matrix is Linux plus Windows for the tokenizer and config reader, and a deliberate test runs the tokenizer under GCC 13.3 on Linux and GCC 16.1 on Windows, feeds them the same 24,499-byte input file, and compares id streams. The md5 sums of stdout differ by one byte. The reason is the line ending the shell added (CRLF on Windows text-mode stdout), not the tokenizer. That is the right kind of cross-platform determinism — deterministic through the parts that matter, not through parts that nobody can control.

The license is Apache-2.0. There is no talk_to_me Python wrapper, no k3 serve --gradio, no chat template, no quantizer. The author’s design rule reads as “this is the engine; if you want a product, write the product.” The benchmark scripts are bash with systemd-run for memory capping, the trace replay is 30 lines of Python, the logits comparison is 40 more. The whole thing is small enough to read in an afternoon, and the test suite — the gate ladder, the byte-exact weight verification, the four-rung oracle — is what makes the byte-identical claim defensible rather than aspirational.

A few things I would watch for in a v0.2. The engine does not support AVX-512, which on a Sapphire Rapids or Zen 4 host would be free throughput; the matmul accumulates in double per group of 32 weights, which is correct but not the most aggressive packing. The tokenizer compares against tiktoken but the BPE loop is a naive greedy merge — at vocab 163,584 this is fine for prompts but not for billion-token pretokenization. The sampling loop is argmax, deliberately, and the README is explicit that greedy is what makes the byte-identical-at-every-budget property hold; a sampler that introduces tie-breaking or RNG would break the tests. The gated MLA’s NoPE rotation has a 64-dim rope slot that is cached and scored but never rotated, and the softmax scale is 1/√192 rather than 1/√128 — getting that wrong is a 22% change in every attention score and produces perfectly readable output, which is the failure mode I would be most worried about porting the code.

The architectural argument I find most durable: the 1.7× cost curve tells you that on this workload the speed is in the storage and the model is in the byte-identical claim, not the benchmark number. A 32-second-per-token engine that produces the same tokens as a 19-second-per-token engine is more useful than a 19-second-per-token engine that produces different tokens depending on how much memory you gave it. The Kimi K3 paper talks about Quantile Balancing being good for training; this engine empirically shows that the same property is exactly what makes an off-the-shelf LRU useless on the inference side, and that the right reaction is to spend the RAM on the trunk. That is a contribution I would not have predicted from the headline “2.78T parameters in 8 GB.”