Major outage / long-lasting disruption at Cloudflare in November 2023

The Cloudflare Dashboard is currently not or only partially accessible. According to Cloudflare, this is due to a power outage. Many services are currently still offline, including the dashboard, API, Argo Smart Routing, firewall, images and load balancer.

At one of my customers, the dashboard is already partially accessible again, but analytics and logs are not available. The performance of changes is also very poor at the moment.

What I have noticed is that some rules that were created shortly before the outage have simply disappeared. In addition, one customer had an ongoing DDoS attack at the time.

In this context, we already feared that the source IP of our server had been exposed, as we suddenly had unfiltered attacks from “normal” IP addresses on our source system (normally, Cloudflare IP addresses should be visible here).

However, it could also be that Cloudflare switched to a very rudimentary “proxy mode” due to the many outages and the traffic was only passed through, so to speak. This was done without actually revealing the source IP, but with the effect that the visitors’ IP addresses were forwarded directly via the proxy forwarded header.

The further effects of this glitch will be revealed shortly.

Update 4.11.2023

As of this morning, the services should be working again. Cloudflare has also published various statements, including that all Cloudflare security functions should have worked correctly.

Services have now been restored for all customers. Throughout the incident, Cloudflare’s network and security services continued to work as expected. While there were periods where customers were unable to make changes to those services, traffic through our network was not impacted.
Cloudflare – Post Mortem on Cloudflare Control Plane and Analytics Outage

I personally can’t confirm this, the security functions sometimes failed or didn’t work correctly. Traffic was also redirected or forwarded unfiltered, which led to massive problems on the customer side during an ongoing attack.

A backup for the WAF rules was obviously also imported, i.e. a backup was actually imported by Cloudflare.

So it cannot have been “just” a simple power failure.