Every major software outage shipped a post-mortem.
The same handful of practices failed every time.
A public, CC BY 4.0 licensed index of major publicly-documented software incidents. Each one mapped to the Concordance engineering protocols that the company's own published root-cause analysis cites as having failed.
Pattern across 10 incidents
Full IndexThe same 5 engineering practices failed in the majority of these incidents. Click any to see every incident where it failed.
A defective rapid-response content update to the Falcon endpoint sensor was deployed simultaneously to all production hosts, causing kernel-level crashes on ~8.5 million Windows machines worldwide.
A threat actor operating under the pseudonym "Jia Tan" gained maintainer access to the xz utils project over a 2-year campaign and inserted a backdoor into liblzma that targeted OpenSSH on systemd-linked Linux distributions. Caught accidentally by a Microsoft engineer noticing 500ms latency in SSH connections.
A threat actor used a service-account credential stored in a personal Google account to access Okta's customer-support case-management system and download HAR files containing session tokens for 134 customer organisations.
A pre-authentication SQL injection vulnerability in Progress's MOVEit Transfer file-transfer software was exploited by the Cl0p ransomware group to exfiltrate data from over 2,700 organisations.
A JNDI lookup feature in the widely-used Apache Log4j Java logging library allowed attackers to trigger arbitrary remote code execution by crafting log messages containing JNDI lookup strings.
A valid customer configuration change exposed a latent software bug in Fastly's edge servers, causing 85% of the network to return errors. Took down major sites including Amazon, Reddit, Twitch, NYT, UK gov.uk, and Stack Overflow simultaneously.
A threat actor exploited an error in Codecov's Docker image creation process to obtain credentials, then modified the Bash Uploader script to exfiltrate environment variables (including secrets) from customer CI environments.
Russian state-affiliated actors compromised SolarWinds's Orion build server and injected malicious code (SUNBURST) into a signed software update, distributing the backdoor to 18,000 customers including US federal agencies.
A missing bounds check in OpenSSL's implementation of the TLS Heartbeat extension allowed remote attackers to read up to 64KB of process memory per request — exposing private keys, session tokens, and user credentials from any TLS-terminating server using affected OpenSSL versions.
A manual software deployment to NYSE's Retail Liquidity Program (RLP) servers updated 7 of 8 production servers; the 8th still ran legacy code that reactivated a dormant test routine ("Power Peg") which placed millions of unintended orders.
Errata: hello@concordancelabs.com