Intended vs. Observed — Debugging the Pi4B Keyboard on Bare Metal

Here is a debugging problem with the difficulty turned up: a USB keyboard is plugged into a Raspberry Pi 4B running sigilOS on bare metal, the login screen renders, and the keyboard does nothing. No characters, no cursor blink in response, nothing. And you cannot attach a debugger — this is real hardware booting over the network, the only window in is a serial cable, and for a long stretch even that was silent. So how do you find a bug you can't watch?

The answer that carried the whole Pi keyboard bring-up was almost embarrassingly low-tech: draw the two paths. Write down, stage by stage, what the machine should do from power-on to a keystroke — and next to it, what the UART and the framebuffer say it actually does. Then find the row where the two columns stop agreeing. That row is the bug.

A two-column diagram of the Pi4B boot path: a shared power-on-to-login stem, then a green 'what should happen' column and a red 'what we see now' column that diverge at the PCIe / VL805 USB fabric. — The map we actually debugged against. Gray = shared boot; green = the intended path; red = what the metal does. They agree all the way to login, then split at the VL805 USB fabric.

Reading the map

The top five stages are gray — power-on, VideoCore firmware, netboot over TFTP, loading the DBLD=100 kernel, and the kernel rendering the PILOGIN screen. Both columns agree here, and that agreement is itself information: the SoC, the network boot, the kernel, and the compositor are all fine. The login screen you can see on the HDMI panel is real. Whatever is wrong is downstream of a working boot.

Then the path forks at the PCIe / VL805 USB fabric — the bcm2711_xhci controller that every USB port on a Pi 4B hangs off of. The green column is what a healthy bring-up looks like:

PCIe root complex up — the RC_BAR2 window opens, UBUS and SCB are configured.
VL805 responds — its config space reads back a valid VIA ID (1106:3483).
VL805 firmware loads — the controller's microcode comes up.
xHCI runs — reset completes, ports are powered.
USB HID enumerates — EnableSlot through to HK=0.
Keyboard types — real scancodes flow (RPT= / RB0:) and the login field fills.

The red column is what the metal actually did, and it goes wrong at the very first fork and stays wrong:

RC_BAR2 reads back 0 — R2LO=0, R2HI=0, the SCB window unset.
VL805 gives no response — config reads come back FFFF FFFF, all-ones.
Firmware load fails — nothing answers on the bus to load.
xHCI is halted — XHLT=1, the host controller never runs.
No enumeration — no HK=, no device ever appears.
Keyboard is dead — the login shows, and input does nothing.

What the divergence told us

This is the payoff of the exercise. The instinct on a dead keyboard is to suspect your own USB or HID code — the enumeration state machine, the descriptor parsing, the keymap. The map says: don't. The two columns split before any of that code runs. The controller answers its config header (so the PCIe link and address routing are alive) but returns all-ones past it and never lets xHCI leave halt. A controller that is half-alive like that — ACKs its identity, then goes dark — is not a controller with a driver bug. It is a controller whose firmware was never loaded.

That reframed the entire hunt. Every fix we could make in software — reset ordering, a self-load trigger, a mailbox request to the VideoCore to reload the microcode — we tried, and each one left the firmware-version register reading zero. The all-ones weren't noise; they were the VL805 telling us it had no microcode to run, and no amount of driver code can conjure microcode that the boot firmware never put there.

Where it landed

So the root moved out of the driver lane and into the firmware / EEPROM lane: the netboot rig simply wasn't provisioning the VL805's microcode at the bootloader stage, so the controller stayed dark and its downstream ports delivered no real power — which is exactly why the keyboard and mouse never even lit up. The fix is to get VL805=1 written into the Pi's bootloader EEPROM so the firmware loads the controller on every cold boot. That update is now applied — VL805=1 is on the EEPROM.

To be precise about status, because on this project "working" means a human watching real pixels, not a green marker: getting the controller provisioned clears the dark-device wall the diagram above is about, and bring-up has moved past it — but the keyboard is not yet confirmed typing end-to-end on metal. The boot now surfaces its next wall further down the path, and that hunt is live. The point of this post isn't a victory lap; it's the method.

The method is the lesson

On a board with no debugger and, for a while, no serial, the tools that actually cracked this were the framebuffer and a two-column list. The standing rule that came out of the Pi bring-up — every debug fact prints on-screen, not just to UART, because the framebuffer is the one output that works on every board — is what made an intended-vs-observed map buildable in the first place. You can't step through metal. But you can write down what should happen, read back what did, and let the first disagreement point at the guilty layer. Here it pointed one layer lower than anyone's code — and that was the whole answer.