nesformer: 466 Million Parameters, 9.3 Million of Them Real, One NES CPU — aniketkarneai.com | aniketkarneai.com
Wednesday, November 11, 2026 Field notes on autonomous systems ● Amsterdam, NL
daily

nesformer: 466 Million Parameters, 9.3 Million of Them Real, One NES CPU

dbreunig/nesformer compiles a 6502 CPU, 2 KB of RAM and a cartridge ROM into the weights of a stock Qwen3.5 checkpoint. I rebuilt the checkpoint byte-identically, ran the games, and wrote the py65 cross-check the README claims but the test suite does not contain.

The file weights/model.safetensors is 933,307,736 bytes. Only 9,340,331 of the 466,647,616 parameters inside it are nonzero. Strip out the embedding table and the real circuit shrinks to 650,820 live values out of 264,043,072 slots, a fill rate of 0.246%.

That is the shape of a checkpoint that was never trained. nesformer, posted the day before I found it, does not contain a neural network that learned to play an NES game. It contains a 6502 CPU, 2 KB of work RAM and a cartridge’s program ROM, compiled into the tensor layout of a stock Qwen3_5ForCausalLM and left there. A small host program plays the rest of the console around it: it renders the picture, reads the controller, fires the vblank interrupt.

The zip the project ships is 3.3 MB. Unzipped it is 933 MB of mostly zeros, because that is what a compiled circuit looks like when you insist on storing it where a language model lives.

I installed it, ran it, and then went looking for the verification story underneath, which turned out to have a hole in it.

The compiler, not the trainer

dbreunig/nesformer is built on Qwenputer by Divyansh Agrawal, which had already compiled a CHIP-8 interpreter and then a RISC-V CPU into Qwen3.5-4B. nesformer reuses that compiler and its central trick, and extends the target to a full NES CPU.

The checkpoint is an ordinary Hugging Face folder. config.json declares Qwen3_5ForCausalLM, model_type: qwen3_5_text, hidden size 768, vocab 263,808, bfloat16. It loads with from_pretrained and needs no custom code. Eight layers: seven linear_attention (Gated DeltaNet) and one full_attention, which is all zeros because the HF cache implementation expects a layer of that shape to be there. The layer plan in nesformer/core.py is a straight-line 7-stage pipeline: PC anchor and register file in layer 1’s attention state, PRG fetch in layer 3, RAM and PRG data reads in layer 5, the next token in layer 7.

The part I like most is that the weights are reproducible. The project ships weights/SHA256SUMS, and it checks out:

weights/model.safetensors: OK
195c4b75fa71da09fd3dc5b2d646d407cae651740b676dc92a39639568226edf  weights/model.safetensors

So I ran the compiler and compared:

$ python -m nesformer build --out /tmp/nesf_rebuild2
wrote /tmp/nesf_rebuild2: 8 layers (7 circuit + 1 zero attention), hidden 768,
466,647,616 parameters, 9,340,331 nonzero (6s)

$ sha256sum /tmp/nesf_rebuild2/model.safetensors weights/model.safetensors
195c4b75fa71da09fd3dc5b2d646d407cae651740b676dc92a39639568226edf  /tmp/nesf_rebuild2/model.safetensors
195c4b75fa71da09fd3dc5b2d646d407cae651740b676dc92a39639568226edf  weights/model.safetensors

Byte-identical, from a source tree, in six seconds. Every byte of that checkpoint is derived from the compiler, not from a gradient. There is no training set, no loss curve, no seed. If you want to change the machine, you change the circuit and recompile.

RAM is a side effect of the attention state

Here is the trick I still find hard to hold in my head. Each Gated DeltaNet head keeps a 128×128 state matrix that the delta rule updates on every token. Hold the gate open and use a one-hot key, and the update overwrites exactly one row of that matrix. Query with a one-hot vector and you read one row back. Every row is a memory cell.

So the cartridge ROM, the 2 KB of RAM and the registers all live in rows of state matrices, and the model advances them by emitting tokens. The PRG-ROM is stored twice, one copy for instruction fetch and one for data reads, because a fetch and a load need different rows.

The trace is the machine’s execution log, one token per operation, and it reads like an assembly listing:

GO P(000100 px0 L1) P(000100 px0 L1) X($FF c0 v0 px0 L2) S($FF L1)
IOR($2002) IN($00) A($00 c0 v0 px0 L3) J($C009) ...

IOR($2002) is the CPU asking for a hardware register, IN($00) is the host answering, J($C009) is a jump. There are 18 token families in nesformer/spec.py, sized by the width of the thing they carry: A, X, Y each get 8,192 slots, J and JI and R get 32,768, IOW and DM get 65,536. The 151 official opcodes map onto 56 mnemonics.

The PC is not written anywhere. It is implied by a jump target plus three DeltaNet counters that tally instructions executed since the last jump, broken out by instruction length. Clever, and it is why the checkpoint needs those 65,536-slot families at all.

Running it

python -m nesformer check roms/nestest.nes --tokens 120000 free-runs the weights and compares them against the reference CPU in nesformer/cpu.py at every position:

OK: the weights generated 120,002 tokens identical to the reference CPU's trace
(310 frames, frames identical: True; 1,187 tok/s)

And python -m nesformer run roms/thwaite.nes --frames 24 plays a real game:

42,934 tokens in 34s (1,261 tok/s), 24 frames, 742,123 CPU cycles (0.4s of NES time)

I dumped the frames as PNGs and md5’d them, because 24 frames for 0.4 seconds of console time is a lot of identical pixels, and it is: frames 7, 12 and 23 hash the same. Only three distinct images come out of that run, which is the Thwaite title screen sitting still because nobody pressed Start. The GIF the project saves has three frames for the same reason.

The rendered output is a real NES screen, character-for-character: “Thwaite 0.04b (c) 2011,2023 Damian YERRICK”, the GPL text, “Press Start Button”, all in the right places.

Speed is the honest limit. A real NES needs about 700,000 tokens per second to keep up with the PPU. This runs at 1,261 on one CPU core with the numba engine, roughly 1,650 on the machine it was built on. The README puts it at 150× to 290× slower than real hardware, and the project ships a stock-transformers path too, at about 10 tokens per second, which is how you can watch a game advance one token at a time.

The gap in the verification

The README says:

The reference CPU (nesformer/cpu.py) matches nestest’s golden log for 3,348 instructions, and the py65 emulator on 5.7 million instructions of game code.

py65 is Mike Naberezny’s independent 6502 emulator, and it is listed in pyproject.toml under the dev extra alongside pytest. But there is no py65 anywhere in tests/:

$ grep -rn "py65" tests/
(no output)

The five test functions cover the nestest log, the circuit under teacher forcing, the weights free-running, the numpy engine agreeing, and the idle-loop skip. The 5.7-million-instruction claim has no test behind it. The dependency is declared, the test is not there.

That is not necessarily wrong. Plenty of projects verify things by hand and leave the harness out of the repo. But it means the claim is unfalsifiable from a fresh clone, so I wrote the cross-check myself.

py65 and nesformer’s reference CPU share no code, which is the point. The chain I wanted was:

py65  ==  nesformer reference CPU   <- this script
nesformer reference CPU  ==  the compiled weights   <- `nesformer check`

If both links hold, the compiled Qwen3.5 weights implement real 6502 semantics.

Getting the two emulators to agree took four attempts, and the failures were more informative than the success.

Attempt one: 2,999 of 2,999 compared states mismatched. The PC, A, X, Y and SP were all correct and only the processor status byte differed, by exactly one bit. Bit 4 is the BREAK flag. py65 keeps it set permanently because that is what a real 6502 does with an unused bit. nesformer’s reference CPU models B only where it is architecturally observable, meaning PHP and BRK push it and PLP and BRK pull it. So I masked bit 4 on both sides, and the XOR deltas narrowed to three values: 0x10, 0x12 and 0x92. Still 2,995 mismatches, and now the accumulator was wrong too: at instruction 4, py65 had A=00 and nesformer had A=FF.

That one was my bug. I was driving the reference CPU’s I/O reads with a hardcoded 0xFF while py65 read 0 out of its memory, and once the loaded values diverged the N and Z flags went with them. The fix was to give both sides the same deterministic I/O map, $2002 returning 0x80 and every other address returning 0, and to sync the 2 KB RAM image across after every instruction.

Third attempt, 3,000 instructions: zero mismatches, zero differing RAM bytes.

Then I ran it longer:

compared 19999 instruction states (BREAK bit masked on both sides)
mismatches: 0
EXACT MATCH: nesformer reference CPU agrees with py65 on every compared instruction
RAM bytes differing at the end: 0

Twelve minutes of debugging for a cross-check the project could have shipped as a 40-line test file, and the answer is that nesformer’s 6502 is correct on the path I could reach.

What my cross-check does not cover

Two things, and the first one matters.

My I/O map is synthetic. The real host in nesformer/host.py implements a scanline PPU renderer, controller ports, DMA timing and vblank NMI delivery, and I stubbed all of it with a constant. So what I verified is the CPU core: opcode semantics, addressing modes, flags, the stack, RAM. What I did not verify is any of the console around it, and that is where the interesting parts live.

Second, I compared 20,000 instructions of nestest against the 5.7 million the README claims. nestest is a diagnostic ROM designed to be hostile: it exercises every official opcode and every boundary case, which is why I chose it, but it is not game code. The gap between 20,000 instructions of a test ROM and 5.7 million of two actual games is exactly the gap where the interesting bugs live.

I have not closed that gap. The honest version of my claim is that nesformer’s reference CPU agrees with py65 on every instruction I could reach, on a diagnostic ROM, with synthetic I/O, and that the upstream claim of 5.7 million instructions of game code remains unreproducible from a fresh clone because the test that would produce it is not in the repository.

There is a third limit, and this one is a hard wall rather than a gap. The CPU can address only 32 KB of program ROM at a time, so nesformer only supports NROM cartridges, iNES mapper 0. Super Mario Bros. is MMC1. The claim in the README that “the same weights run any NROM game without a rebuild” is true, and Super Mario is not one of them.

Where this could go

A CPU is a fine target for this trick. It is a program, the program is finite, and the state is small enough that you can hold it in a recurrent state matrix. What interests me more is the reverse direction: not compiling a program into weights, but compiling a program’s weights into a program.

Nobody trained this checkpoint. That is the finding I keep returning to. It is a 933 MB artifact with a 3.3 MB compressed core, a SHA256 in the repo, and a compiler that regenerates it exactly. Every claim about its behaviour is a claim about a circuit, not about a learned model, and every one of those claims is checkable by running the compiler and diffing the bytes.

I would like to know whether the 5.7 million instructions hold up. If somebody upstream reads this and adds the py65 test that pyproject.toml already promises, I would like to know what it finds.

Aniket Karne
DevOps & AI Engineer · Amsterdam
Back to all posts
Reader correspondence

Comments

Powered by GitHub Discussions via Giscus. Sign in with GitHub to leave a comment.