CloudFlare Global Outage

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > CloudFlare Global Outage

Posted by Simplex-Ed, 05-02-2012, 04:30 PM
Anybody else seeing issues? cloudflare.com seems out and most customers using CF are down.

No tweets on any of their accounts.

Anybody know more?

Posted by Simplex-Ed, 05-02-2012, 04:31 PM
https://twitter.com/#!/CloudFlareSys...84775635894272

edit - new tweet https://twitter.com/#!/CloudFlareSys...85441360023552 they have confirmed it's indeed affecting all locations.

Can only speculate how many websites are out because of this.
Posted by Techy, 05-02-2012, 04:34 PM
Wow, that's a serious issue. Thanks for the update Ed.
Posted by KMyers, 05-02-2012, 04:41 PM
Hello,
I can hit their website and my sites behind cloudflare are working fine. It may be ISP related. Then again on their status page, Miami is the only section that is working properly (and I am in Miami)
Posted by Simplex-Ed, 05-02-2012, 04:44 PM

Quote:

Originally Posted by KMyers

Hello,
I can hit their website and my sites behind cloudflare are working fine. It may be ISP related. Then again on their status page, Miami is the only section that is working properly (and I am in Miami)

Everything is backup now but there was problems in EU at least. CloudFlare accelerated websites didn't even load the CF landing page -- they were completely down.

Kinda makes me wonder how easy it'd be for CF to wipe out a huge amount of websites. Does anybody know how many they host at all?

Posted by DotVPS-J, 05-02-2012, 04:52 PM
Did this cause sites just using there protection or people using it as DNS management too?
Posted by damoncloudflare, 05-02-2012, 06:01 PM
My apologies for the service outage (definitely our fault). Here's the explanation:

While tuning Asia performance, an improper router configuration was pushed, causing upstream provider to misroute traffic.
Posted by (Stephen), 05-02-2012, 08:04 PM

Quote:

Originally Posted by damoncloudflare

My apologies for the service outage (definitely our fault). Here's the explanation:

While tuning Asia performance, an improper router configuration was pushed, causing upstream provider to misroute traffic.

What safeguards will be put in place now, as you grow in locations and in customer base, you will undoubtedly have times of needing to push items to all locations at once, but should there not be an additional confirmation of some sort for that type event?

(just for thought)

Things seem to be running better now but prior to now for the last 36 hours we've actually had pretty terrible performance not related to our backend (loading IP direct to us was sub second loads, but via CF some images were loading for 50+ seconds)
Posted by damoncloudflare, 05-02-2012, 08:14 PM
"

Quote:

Originally Posted by (Stephen)

What safeguards will be put in place now, as you grow in locations and in customer base, you will undoubtedly have times of needing to push items to all locations at once, but should there not be an additional confirmation of some sort for that type event?"

We have put in safeguards to prevent a similar issue from happening (a pretty good explanation on our blog about the what & how we're addressing).

Things seem to be running better now but prior to now for the last 36 hours we've actually had pretty terrible performance not related to our backend (loading IP direct to us was sub second loads, but via CF some images were loading for 50+ seconds)

That definitely seems unusual & we haven't had any issues lasting that long. Any details you can send to our support team would be helpful (what you were using to test, files, traceroute to the domain, etc.).
Posted by flam316, 05-02-2012, 08:26 PM

Quote:

Originally Posted by Simplex-Ed

Everything is backup now but there was problems in EU at least. CloudFlare accelerated websites didn't even load the CF landing page -- they were completely down.

Kinda makes me wonder how easy it'd be for CF to wipe out a huge amount of websites. Does anybody know how many they host at all?

I think they control around 200,000+ sites.

Also, CloudFlare sites were up on my ISP. Which is good because this means most of the NY metro area didn't notice any outage.
Posted by TadasPet, 05-03-2012, 09:52 AM
Official blog is @ http://blog.cloudflare.com/todays-outage-post-mortem