Three independent tools all agreed the new kernel image was deployed to the rig. A filesystem copy said so. A symlink that resolved cleanly said so. A real TFTP-GET, fetched byte-for-byte from the boot server, said so too. Three for three — every sane deploy check you'd reach for came back green.
The metal disagreed. Three cold-cycles in a row.
What we believed
A reboot-on-dead fix — build DBLD=149, sha 21cffa59 — was live on the rig's netboot path. A filesystem cp matched the hash. A symlink resolved to the right target. A busybox tftp GET against the boot server pulled down exactly 21cffa59, 507,184 bytes. Every layer of the deploy pipeline, checked independently, said the same thing: this is live, go test it.
What disproved it
The VideoCore firmware prints exactly one line when it loads a kernel, and that line carries the one fact nothing upstream of it can lie about:
Loaded 'kernel8.img' ... size 0x7b3e80x7b3e8 = 504,808 bytes. That's the old build — DBLD=145 — not the 507,184-byte 149 every deploy check had just certified. And it wasn't a fluke: three cold cycles, three loads, always 0x7b3e8, never once the 0x7bd30 that 149 would print.
| Check | What it verified | What it missed |
|---|---|---|
Filesystem cp | The bytes landed on disk | Which file the boot path actually reads |
| Symlink resolution | The link points at the right target | Whether the boot server's tftpd follows symlinks at all |
| TFTP-GET | The server serves the right bytes, over the wire | Which server the board is even talking to |
The root cause, once we stopped trusting the deploy tools and started reading the boot log: the Pi wasn't booting from the network at all. /mfs/sd/ in the VPU's own boot trace — that's the SD card, not TFTP. The EEPROM was set SD-first, and a DBLD=145 test card happened to be seated in the slot. Every byte of the TFTP deploy was completely correct — and completely irrelevant, because the board was reading a different piece of storage entirely. Three green checks, all watching a pipe the hardware wasn't drinking from.
The lesson
Verify a deploy by the size the silicon reports it loaded — not a filesystem hash, not a symlink target, not an assumed fetch path. A sha256 match tells you the bytes exist somewhere. A byte-size on the boot log tells you which bytes the machine actually ran. Only one of those is ground truth, and it's the one printed by hardware that has no reason to lie: the VPU doesn't know or care what your deploy pipeline believes, it just says how many bytes it loaded. When the story your tools tell and the story the silicon tells disagree, the silicon wins every time.
The bonus find
Once 149 actually reached the board — the SD card reseated, EEPROM boot order fixed — it did what it was built to do: caught a fully off-bus, cold-dead USB controller and self-healed it. But the same run also surfaced a controller state the fix didn't cover: firmware reporting present (FW0=0) yet never finishing its load (FWNO) — a different flavor of stuck that looks alive on paper and is dark in practice (HK=-1, mouse silent). The self-heal that watches for "controller vanished" doesn't trip for "controller present but never woke up." That's the next build's target.