Death by Uptime

About this episode

The episode discusses a disturbing failure mode encountered in a production rack and the team's debugging process to uncover its root cause.

We hit a new (and disturbing!) failure mode recently when a production rack that had been up for several months saw every (!) compute sled's service processor become simultaneously unresponsive. Bryan and Adam were joined by the members of the Oxide team who debugged the vexing issue -- and reached its surprising root cause. In addition to Bryan Cantrill and Adam Leventhal , we were joined by Oxide colleagues, Cliff Biffle , Matt Keeter , and Will Chandler. Previously, on Oxide and Friends: OxF s05e03 – Holistic Engineering with Robert Mustacchi OxF s04e14 – Rebooting a datacenter: A decade later OxF s01e26 – The Pragmatism of Hubris OxF s05e20 – Debugger-Driven Development (omdb) OxF s05e07 – Transparency in Hardware/Software Interfaces OxF s05e31 – Futurelock OxF s05e33 – A Grown-up ZFS Data Corruption Bug Some of the topics we hit on, in the order that we hit them: hubris #2304: STM32H7 Ethernet driver stops yielding CPU after many packets gist — Summarizing the Hubris side of investigations Matt's blog: Hunting a spooky ethernet driver bug If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord…

People in this episode

Hosts: Bryan Cantrill, Adam Leventhal

Guests: Cliff Biffle, Matt Keeter, Will Chandler

Topics covered

• failure modes
• debugging
• service processors
• production systems
• Ethernet driver issues
• hubris

Keywords

• uptime
• debugging
• service processor
• Ethernet driver
• production rack
• failure mode
• hubris

Mentioned in this episode

Organizations: Oxide Computer Company

More episodes of Oxide and Friends

• This Old Repo: LLMs and the Restoration of BattleTris · June 9, 2026 · 1h 18m
• Rooting for the Home Team with Paul Freedman and Bryan Carmel · May 27, 2026 · 1h 2m
• The Tale of Reverso · May 16, 2026 · 1h 7m
• AI in Computer Science Education · May 10, 2026 · 1h 29m
• Mechanical Engineering at Oxide [chapter images] · May 7, 2026 · 1h 25m
• Are LLMs Insufficently Lazy? · May 3, 2026 · 1h 31m