Cloudflare R2 crash causes a 59-minute outage, affecting services and leading to up to 13.6% log loss, with swift recovery and key system improvements.
Continue reading
Cloudflare faced a significant service disruption affecting multiple platforms, including its R2 Object Storage service. The outage lasted for 59 minutes and caused a complete failure of operations against R2, along with widespread disruption to several Cloudflare services that depend on R2, including Stream, Images, Cache Reserve, Vectorize, and Log Delivery. This Threatfeed delves deep into the technical details surrounding the incident, its impact on users, and Cloudflare's efforts to mitigate similar events in the future.
Date of Incident: February 6, 2025 Duration: 59 minutes Primary Cause: Human error during abuse remediation Root Cause: Insufficient validation safeguards during routine phishing site remediation Impacted Services: R2, Stream, Images, Cache Reserve, Vectorize, Log Delivery, Durable Objects, Cache Purge, Key Transparency Auditor, Workers & Pages
At approximately 08:12 UTC, the R2 Gateway service was inadvertently disabled during the routine remediation of a phishing site complaint. The action, intended to target a specific phishing URL hosted on R2, mistakenly disabled the entire R2 Gateway service responsible for authenticating and serving requests. As a result, all operations against R2 failed during the initial 59-minute incident window.
R2:
Stream, Images, and Cache Reserve:
Log Delivery:
Vectorize:
Durable Objects:
Cache Purge and Key Transparency Auditor:
Workers & Pages:
| Time (UTC) | Event |
|---|---|
| 08:12 | R2 Gateway service disabled during phishing remediation. |
| 08:14 | Impact begins: R2 operations fail. |
| 08:18 | Critical R2 alerts triggered due to service failure. |
| 08:23 | Sales engineering escalates issue to R2 engineering. |
| 08:33 | Internal incident declared. |
| 08:42 | Root cause identified: R2 Gateway service disabled by error. |
The root cause of the incident was human error during a routine abuse remediation. Cloudflare’s system mistakenly allowed the operator to disable the entire R2 Gateway service instead of the specific phishing endpoint. This issue was compounded by insufficient safeguards in the abuse processing system, which failed to distinguish between internal accounts and customer-facing resources.
The R2 service architecture is built on a separation of concerns, where the Gateway service handles authentication and request routing, while the underlying infrastructure (including the distributed storage subsystem) remains unaffected during failures. However, with the Gateway service down, all operations against R2 were halted. Notably, there was no data loss or corruption during the incident, as the infrastructure components remained intact.
Once the root cause was identified, Cloudflare faced challenges in restoring the R2 Gateway service due to the lack of direct rollback functionality for the product disablement action. The R2 team was forced to engage an operations team with lower-level system access to restore service. After redeploying the R2 Gateway service, client operations were restored, and error rates for dependent services began to normalize.
Cloudflare has committed to a thorough review and improvement of its internal controls to prevent a recurrence of this incident. The company has outlined several key remediation efforts:
Cloudflare has implemented stricter validation safeguards to prevent disabling production services running on internal accounts.
Product disablement actions in the abuse remediation interface have been temporarily disabled while more robust safeguards are added.
Cloudflare is revising how internal accounts are provisioned to ensure they are properly tagged and protected from accidental disablement.
Access to product-disablement actions will be limited to a smaller group of senior operators, with two-party approval required for any ad-hoc disablement requests.
New abuse checks will be added to prevent accidental blocking of internal Cloudflare hostnames and prevent disablement of services linked to internal accounts.
While the February 6th outage lasted less than an hour, its impact was significant, affecting key Cloudflare services relied upon by millions of users. However, the swift recovery, the lack of data loss, and Cloudflare’s immediate commitment to fixing systemic issues demonstrate the company’s dedication to preventing similar incidents in the future.
Cloudflare acknowledges the severity of the incident and is deeply sorry for the inconvenience caused to its customers. The company’s commitment to improving its systems and reducing human error remains a top priority. As Cloudflare continues to enhance its internal controls, users can expect more resilient and reliable services moving forward.
148 malicious npm packages masquerading as student proxy and school Wi-Fi bypass tools. Rather than compromising developers during installation
| 08:46 |
| Attempts to re-enable R2 Gateway service using internal tooling fail. |
| 08:57 | Operations team escalated and begins service restoration. |
| 09:09 | R2 Gateway service redeployed, recovery begins. |
| 09:13 | Impact ends; R2 service begins recovery. |
| 09:36 | Durable Objects error rate returns to normal. |
| 10:29 | Incident closed after monitoring confirms error rates return to normal. |
