Every major software outage shipped a post-mortem.
The same handful of practices failed every time.
A public, CC BY 4.0 licensed index of major publicly-documented software incidents. Each one mapped to the Concordance engineering protocols that the company's own published root-cause analysis cites as having failed.
Pattern across 8 incidents
Filtered viewThe same 5 engineering practices failed in the majority of these incidents. Click any to see every incident where it failed.
A defective rapid-response content update to the Falcon endpoint sensor was deployed simultaneously to all production hosts, causing kernel-level crashes on ~8.5 million Windows machines worldwide.
A threat actor operating under the pseudonym "Jia Tan" gained maintainer access to the xz utils project over a 2-year campaign and inserted a backdoor into liblzma that targeted OpenSSH on systemd-linked Linux distributions. Caught accidentally by a Microsoft engineer noticing 500ms latency in SSH connections.
A pre-authentication SQL injection vulnerability in Progress's MOVEit Transfer file-transfer software was exploited by the Cl0p ransomware group to exfiltrate data from over 2,700 organisations.
A JNDI lookup feature in the widely-used Apache Log4j Java logging library allowed attackers to trigger arbitrary remote code execution by crafting log messages containing JNDI lookup strings.
A valid customer configuration change exposed a latent software bug in Fastly's edge servers, causing 85% of the network to return errors. Took down major sites including Amazon, Reddit, Twitch, NYT, UK gov.uk, and Stack Overflow simultaneously.
Russian state-affiliated actors compromised SolarWinds's Orion build server and injected malicious code (SUNBURST) into a signed software update, distributing the backdoor to 18,000 customers including US federal agencies.
A missing bounds check in OpenSSL's implementation of the TLS Heartbeat extension allowed remote attackers to read up to 64KB of process memory per request — exposing private keys, session tokens, and user credentials from any TLS-terminating server using affected OpenSSL versions.
A manual software deployment to NYSE's Retail Liquidity Program (RLP) servers updated 7 of 8 production servers; the 8th still ran legacy code that reactivated a dormant test routine ("Power Peg") which placed millions of unintended orders.
Errata: hello@concordancelabs.com