28 volts into a 12-volt rail: an autopsy of a bug that was typed, not soldered

The most expensive bug I’ve ever shipped wasn’t in firmware. It was a spreadsheet cell — one transposed line in a bench configuration — and it cost a prototype that took six weeks to build. This is what actually happened, and the taxonomy of mistakes like it.

The night it happened

The bench was a hardware-in-the-loop rig: one power channel for the device under test, an analog command line to the actuator, a digital bus back from the IMU. Nothing exotic. The kind of rig that exists in hundreds of programs, described — if it’s described at all — in a spreadsheet that someone edits the night before a campaign.

That night, someone did. Two rows got transposed during a revision. The DUT’s power input was now described on the channel wired to the 28 V bus. Its absolute maximum was 12 V.

Everything about the setup looked right. The checklist was signed. The config file parsed. The sequence ran. The bench did exactly, precisely, what it was told to do — which is the thing nobody tells you about benches: they are very obedient machines. The DUT drew its last breath in under a second. Six weeks of procurement, assembly, and firmware bring-up, gone. The post-mortem found no faulty component, no bad solder joint, no firmware defect. Just a config that was never checked, because there was nothing to check it with.

The embarrassing part

I write software. My code gets a compiler, a type checker, a linter, CI gates, code review. If I transpose two arguments, three tools fight to be the first to tell me before anything runs. But the configuration that drives actual current into actual silicon — the artifact with the highest blast radius in the whole lab — got a spreadsheet and a signature on a checklist.

Software figured this out decades ago: if a class of mistake is mechanically detectable, you build a machine that detects it, and you run the machine before every single execution. Nobody hand-reviews for missing semicolons anymore. Yet bench configs — where a missing value can mean fire — are still hand-reviewed, at 11 pm, by the person who wrote them.

The taxonomy of typed bench bugs

After that night I started collecting these. Talking to verification teams, the same stories keep surfacing — different benches, identical physics. They cluster into a small, finite set of categories:

• pin.conflict — two signals allocated to the same physical pin. A stimulus shorted into a measurement line. Destroys hardware.
• stimulus.overdrive — a sequence step driving 14 V into a channel rated for 12 V. One keystroke over the rail. Destroys hardware.
• interlock.missing — a 24 V+ channel with no e-stop loop defined. Nothing cuts it when a run goes wrong. Destroys hardware.
• interlock.cuts — the e-stop’s cut-list says bus_pwr. The channel is called bus_power. The e-stop exists; it cuts nothing. Destroys hardware, while looking protected.
• timing.nyquist — sampling a 400 Hz signal at 700 Hz. The data looks fine. It is aliased. Weeks of campaign — void. The bench never even smells smoke.
• timing.budget — steps that sum past the campaign window, discovered when the bench times out at hour eleven. Blocks the campaign.
• sequence.reference — a test step pointing at a channel that was renamed last revision. The bench halts mid-campaign — or worse, doesn’t. Blocks the campaign.

What a pre-flight check looks like

Describe the bench as data — channels, pins, rails, interlocks, sequences — and let a linter read it before the power supply does:

$ standguard.py lint bench.json --strict

[ERROR  ] pin.conflict         channels['actuator_cmd']: pin 'P1.3' already allocated to channel 'bus_power'
[ERROR  ] interlock.cuts       interlocks['estop_loop_1']: cuts references undefined channel 'bus_pwr'
[WARN   ] timing.oversampling  channels['imu_spi']: 2.5× oversampling — review against analysis needs

2 errors · 1 warning · verdict: BENCH UNSAFE TO POWER (exit 1)

Twenty checks today, across eight categories — schema, uniqueness, pins, limits, interlocks, timing, stimulus, sequences. The exit code is the gate: 0 means acceptable, 1 means do not apply power. It wires into CI or into the bench checklist itself.

The part where you come in

The check catalog only gets better one way: real failure modes. Every rule in it exists because a mistake like the ones above actually burned a bench, voided a campaign, or blocked a program. If you’ve been bitten by a typed bench bug — and if you’ve run benches for more than a year, you have — I want the physics, anonymized as much as you need. That’s how rule 21 gets written.