CONCORDANCE LABS · INCIDENT INDEX · v0.1

Every major software outage shipped a post-mortem.
The same handful of practices failed every time.

A public, CC BY 4.0 licensed index of major publicly-documented software incidents. Each one mapped to the Concordance engineering protocols that the company's own published root-cause analysis cites as having failed.

Methodology →·JSON API·CC BY 4.0·Run a free scan →

Pattern across 10 incidents

Full Index

The same 5 engineering practices failed in the majority of these incidents. Click any to see every incident where it failed.

4.6Security AnalysisSENTINEL
4 of 10
3.5Approval Rigor
3 of 10
3.6Code OwnershipSENTINEL
3 of 10
3.9Secrets ManagementSENTINEL
3 of 10
4.2CI Gating
3 of 10
10 of 10 incidents in this view failed at least one Sentinel-10 protocol — Concordance's set of engineering practices most degraded under AI-accelerated development. Velocity Governance thesis →
FilterCivilizational onlyEnterprise onlyAI-related only20242023202120202014
CrowdStrikeJul 2024Civilizational
Falcon sensor channel-file update crashes 8.5M Windows hosts
$10B · 9M affected · 24h downtime

A defective rapid-response content update to the Falcon endpoint sensor was deployed simultaneously to all production hosts, causing kernel-level crashes on ~8.5 million Windows machines worldwide.

4.2 CI Gating5.7 Rollback Capability5.8 Feature Flagging5.4 Release Approval
xz utils (Tukaani Project)Mar 2024Civilizational
CVE-2024-3094 — multi-year social-engineering attack inserts backdoor into xz/liblzma

A threat actor operating under the pseudonym "Jia Tan" gained maintainer access to the xz utils project over a 2-year campaign and inserted a backdoor into liblzma that targeted OpenSSH on systemd-linked Linux distributions. Caught accidentally by a Microsoft engineer noticing 500ms latency in SSH connections.

3.6 Code Ownership3.5 Approval Rigor4.7 Dependency Scanning4.8 Build Reproducibility
OktaOct 2023Enterprise
Okta support-portal HAR file leak exposes 134 customer environments
134 affected

A threat actor used a service-account credential stored in a personal Google account to access Okta's customer-support case-management system and download HAR files containing session tokens for 134 customer organisations.

3.9 Secrets Management3.5 Approval Rigor3.6 Code Ownership
Progress Software (MOVEit)May 2023Civilizational
CVE-2023-34362 SQL injection in MOVEit Transfer leads to mass data theft
$12B · 95M affected

A pre-authentication SQL injection vulnerability in Progress's MOVEit Transfer file-transfer software was exploited by the Cl0p ransomware group to exfiltrate data from over 2,700 organisations.

4.6 Security Analysis4.7 Dependency Scanning2.6 Dependency Management
Apache Software Foundation (Log4j)Dec 2021Civilizational
CVE-2021-44228 — JNDI injection in Log4j enables remote code execution

A JNDI lookup feature in the widely-used Apache Log4j Java logging library allowed attackers to trigger arbitrary remote code execution by crafting log messages containing JNDI lookup strings.

1.3 Ticket Description Quality4.6 Security Analysis2.6 Dependency Management
FastlyJun 2021Civilizational
Single customer config change triggers global Fastly CDN outage
1h downtime

A valid customer configuration change exposed a latent software bug in Fastly's edge servers, causing 85% of the network to return errors. Took down major sites including Amazon, Reddit, Twitch, NYT, UK gov.uk, and Stack Overflow simultaneously.

4.3 Test Coverage5.8 Feature Flagging5.7 Rollback Capability4.2 CI Gating
CodecovApr 2021Enterprise
Codecov bash-uploader supply-chain compromise exfiltrates customer secrets
29,000 affected

A threat actor exploited an error in Codecov's Docker image creation process to obtain credentials, then modified the Bash Uploader script to exfiltrate environment variables (including secrets) from customer CI environments.

4.8 Build Reproducibility3.9 Secrets Management4.2 CI Gating6.1 Incident Response
SolarWindsDec 2020Civilizational
SUNBURST malware injected into Orion build pipeline compromises 18,000 organisations
$100B · 18,000 affected

Russian state-affiliated actors compromised SolarWinds's Orion build server and injected malicious code (SUNBURST) into a signed software update, distributing the backdoor to 18,000 customers including US federal agencies.

4.8 Build Reproducibility3.5 Approval Rigor3.9 Secrets Management4.6 Security Analysis
OpenSSL ProjectApr 2014Civilizational
CVE-2014-0160 — Heartbleed buffer over-read in OpenSSL TLS heartbeat

A missing bounds check in OpenSSL's implementation of the TLS Heartbeat extension allowed remote attackers to read up to 64KB of process memory per request — exposing private keys, session tokens, and user credentials from any TLS-terminating server using affected OpenSSL versions.

4.3 Test Coverage4.6 Security Analysis3.6 Code Ownership3.2 PR Review Quality
Knight Capital GroupAug 2012Civilizational
Knight Capital loses $440M in 45 minutes from incomplete software deployment
$440M

A manual software deployment to NYSE's Retail Liquidity Program (RLP) servers updated 7 of 8 production servers; the 8th still ran legacy code that reactivated a dormant test routine ("Power Peg") which placed millions of unintended orders.

5.4 Release Approval5.7 Rollback Capability5.8 Feature Flagging6.1 Incident Response
Concordance Incident Index v0.1 · CC BY 4.0 · Roadmap: 250 entries at v1.0, scaling to 5,000 with NSF SBIR Phase I.
Errata: hello@concordancelabs.com