The mouse saga: SMI deadlocks, a byte-order guess, and locking a protocol from a raw stack dump
The font bug from the last post was contained: wrong pixels on a screen, visible immediately, fixable by staring at the output. The mouse driver was worse, because for a stretch of several days in mid-February, the failure mode was not "wrong," it was "the machine stops responding, and there is no way to know why from inside the machine itself."
The setup
By build 110, UnoDOS had window dragging working in an emulator. The commit history for the next ten builds is almost entirely mouse-shaped: cursor trails, button display bugs, a keyboard freeze traced to the mouse initialization sequence corrupting the 8042 keyboard controller's configuration (build 111, "restore 8042 config on mouse init failure, fix cursor stack bug"). Then, on February 14, a new class of failure appeared that had nothing to do with drag logic at all: build 162, "Fix mouse init hang on USB boot, add boot progress dots." Build 163 followed: "Skip mouse init on missing 8042, reduce KBC timeouts." The machine was hanging, specifically, when booted from USB rather than floppy, and specifically during mouse setup.
Over the following builds, the diagnosis narrowed in public, one guess at a time:
- Build 164: "Probe BIOS timer before mouse init, skip if not ticking"
- Build 165: "Add numbered boot checkpoints for diagnosis"
- Build 166: "Slash KBC timeouts to 256/4096 iterations for SMI-heavy systems"
- Build 167: "Skip ALL KBC I/O on HD/USB boot to fix SMI deadlock"
That last one is the actual root cause, and it is a genuinely obscure one: System Management Interrupts. Some real PC hardware, particularly the kind you get when booting from USB rather than a floppy the BIOS was written around, intercepts keyboard controller I/O at the firmware level for its own purposes, invisibly, and on certain systems that interception can deadlock against a naive polling loop waiting on the same controller. There is no error code for this. There is no way to ask the hardware "are you currently stuck in an SMI handler." The only symptom is that the machine stops responding, and the only way anyone found the actual mechanism was by systematically removing suspects (missing 8042, BIOS timer not ticking, timeout length) until the one that mattered, avoiding the KBC entirely on that boot path, made the hang disappear.
The second problem: a protocol nobody had written down
Getting past the deadlock did not mean the mouse worked. It meant the driver could now talk to real hardware, and real hardware turned out to disagree with itself about how BIOS mouse callbacks report their data. Build 187, "Use BIOS INT 15h/C2 for mouse init, direct KBC as fallback," moved the driver to the standard BIOS mouse service instead of talking to the controller directly, which should have been the easy, well-documented path. It was not. Build 188: "Fix BIOS mouse callback: AH corruption + fragile byte-order detect." Build 189: "Support both BIOS callback conventions for mouse packets." There were, in effect, two different byte orderings a real BIOS might use to hand back mouse movement data, and no reliable way to know in advance which one a given machine's firmware would choose.
The fix that actually stuck, in build 190, is the one I keep coming back to when I talk about this project's method: "Lock-in BIOS mouse callback convention on first unambiguous packet." Rather than trying to detect the convention up front from documentation or heuristics (which is what the earlier, failed attempts had tried), the driver waits for the first mouse packet whose byte pattern can only be interpreted one way, and locks its interpretation of every subsequent packet to match. Build 191 added a raw stack dump diagnostic specifically to generate the evidence for this: capture exactly what bytes the BIOS callback actually pushed onto the stack, and read them like a detective reads a crime scene rather than guessing from a manual. The dump that settled it came from QEMU's BIOS, the first machine that could reproduce the ambiguity on demand; the build-190 lock-in is what carries that answer safely onto every other machine's firmware. Build 192, the fix that shipped: "Fix BIOS mouse callback byte order from QEMU stack dump." The same day, build 193 declared it a milestone: "Universal PS/2 mouse via BIOS services."
The lesson
Two things came out of this that I have applied to every hardware driver since. First: when there is no manual you can trust, and no error message the hardware will give you, the only ground truth is a raw capture of what the machine in front of you actually did, not a description of what it is supposed to do. The stack dump in build 191 is a small piece of tooling, three commits worth of work, and it is what actually resolved a bug that six earlier build-by-build guesses had not.
Second, and this is the part that generalizes past hardware drivers entirely: an agent guessing at a protocol from documentation will keep guessing plausibly and wrongly, in a loop, for as long as you let it. The move that ended the loop was not a better guess. It was changing the question from "what does the spec say the convention should be" to "what can we observe about this specific machine, right now, that removes the ambiguity." That shift, from asking an agent to reason about a spec toward asking it to design an experiment against the real system, is the same shift that later became the verification harnesses this whole project runs on: render the framebuffer and diff the pixels, execute the built binary at the instruction level and check what it actually did, rather than trusting a description of what it should have done.
The next post is the third in this run of early war stories, and it might be my favorite: a bug where the fix was confidently declared "the ACTUAL root cause," in exactly those words, and then needed four more fixes after that, in the same afternoon.
Source: https://github.com/hmofet/unodos, builds 110 through 193, February 11-15.