Cloudflare network down?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Cloudflare network down?

Posted by VN-Ken, 09-15-2012, 11:51 AM
Our websites are going up and down like a yoyo, and cloudflare's website itself is also down.

Anyone else also experiencing trouble?
Posted by Kingfish85, 09-15-2012, 11:52 AM

Quote:

Originally Posted by VN-Ken

Our websites are going up and down like a yoyo, and cloudflare's website itself is also down.

Anyone else also experiencing trouble?

Same here. Issue seems to be with CloudFlare. Hopefully they fix it soon.
Posted by Ethernet Servers, 09-15-2012, 11:53 AM
It is indeed down. I was wondering what was going on when my forum just died on me. Server was up and running fine yet the website was down. Then I realized it was CF down.
Posted by EvolutionCrazy, 09-15-2012, 11:54 AM
Can't reach anything from most of the Europe (tested on multiple PRO and BUSINESS plans)... even their site is not reachable...

https://twitter.com/CloudFlareSys/st...97504950956032

This is the biggest/widespread cloudflare downtime I've seen since I've been a customer (nearly a year)

What's going on?

Connectivity to and from nlayer seems fine here...
Code:
  1    <1 ms    <1 ms    <1 ms  internet.gateway [192.168.0.200]
  2     *        *        *     Richiesta scaduta.
  3     *        9 ms     *     172.17.81.21
  4    10 ms    10 ms    10 ms  172.17.80.1
  5    18 ms    17 ms    17 ms  172.17.9.233
  6    19 ms    19 ms    16 ms  172.17.10.81
  7    17 ms    19 ms    15 ms  pos1-10-0-0.milano26.mil.seabone.net [195.22.192.29]
  8    30 ms    29 ms    30 ms  xe-2-2-0.parigi52.par.seabone.net [195.22.210.96]
  9    36 ms    36 ms    35 ms  xe-0-0-2.cr1.cdg1.fr.nlayer.net [69.22.139.49]
 10    36 ms    36 ms    36 ms  as13335.xe-0-0-1-302.cr1.cdg1.fr.nlayer.net [63.141.223.18]
 11    37 ms     *       37 ms  cf-173-245-61-250.cloudflare.com [173.245.61.250]
Posted by Amitz, 09-15-2012, 11:55 AM
Same here from Germany/Europe.
Posted by kamal_s, 09-15-2012, 11:56 AM
same here in canada
Posted by HostingOwner-SS, 09-15-2012, 11:56 AM
Same here, cloudflare main website is down aswel.
Posted by cmaniac, 09-15-2012, 11:56 AM

Quote:

Originally Posted by AH - George

It is indeed down. I was wondering what was going on when my forum just died on me. Server was up and running fine yet the website was down. Then I realized it was CF down.

Just another confirmation, was on a few websites behind CloudFlare and then poof,

http://speedcap.net/img/f40593839c04...8/88282e2a.png

But as of right now - CloudFlare.com seems to be responsive and websites are up *but slow*.
Posted by TravisT-[SSS], 09-15-2012, 11:57 AM
All down here too ;/
Posted by EvolutionCrazy, 09-15-2012, 11:57 AM
seems up now
Posted by VN-Ken, 09-15-2012, 11:59 AM

Quote:

Originally Posted by EvolutionCrazy

it's up now

Ditto. CF seems up as are our websites. Hopefully it stays that way.
Posted by cmaniac, 09-15-2012, 11:59 AM

Quote:

Originally Posted by EvolutionCrazy

it's up now

//edit not more.... LOL

No, it is online. Just refresh a time or two, you'll see.

It's dropping random connections, who knows why.

http://isup.me/cloudflare.com
Posted by flam316, 09-15-2012, 12:02 PM
I think it's nLayer that was the issue. Most their network runs on them and most of their network was down because of an "upstream issue."

Anyways, it's all back up for me in NY metro.
Posted by andryus, 09-15-2012, 12:03 PM

Quote:

Looking into an issue with one of our upstream providers, affecting multiple colos
Tweeted Sat Sep 15 15:42:48 +0000 2012

By cloudflare via their system status page.
Posted by EvolutionCrazy, 09-15-2012, 12:06 PM
from the US it was fine (or at least most of the US facilities)

Still can't understand why I wasn't able to reach the site and why it was still forcing me to to the broken facilities.... instead of rerouting everything immediately on the locations not affected :|
Posted by flam316, 09-15-2012, 12:09 PM

Quote:

Originally Posted by EvolutionCrazy

from the US it was fine (or at least most of the US facilities)

Still can't understand why I wasn't able to reach the site and why it was still forcing me to to the broken facilities.... instead of rerouting everything immediately on the locations not affected :|

It was an upstream issue which affected US and EU locations... which probably made it difficult to route around considering that they only use a couple upstreams in those areas.
Posted by EvolutionCrazy, 09-15-2012, 12:10 PM
backend down again?

This website is offline
No cached version is available

http servers seems fine... but no connectivity to the origin servers...

100% and 2500% SLAs
Posted by DotVPS-J, 09-15-2012, 12:13 PM
It seems there DNS system is down too , It's not just there Cloud proxy system.. as my domains aren't resolving either.
Posted by EvolutionCrazy, 09-15-2012, 12:16 PM
dns are reachable and working from here (italy - telecom italia) - reverse proxy is reachable but not working...
Posted by AlanDu, 09-15-2012, 12:28 PM
Yeah, it's really annoying, I am losing valuable website visitors :'(.
Posted by nexusbroadcast, 09-15-2012, 12:43 PM
Cloudflare has been down for me since 11 AM. I have changed my dns back to my servers - that will bring your websites back online until this issue is resolved.
Posted by flam316, 09-15-2012, 12:45 PM
You guys are still seeing downtime? It's been up for more than 45 minutes for me.
Posted by EvolutionCrazy, 09-15-2012, 12:46 PM
still down... more then 1 hour since the initial downtime... no update... no tech info.... what is going on?
Posted by nexusbroadcast, 09-15-2012, 12:50 PM
It just came back up for me as I finished switching the dns off them. switching it back now.. arghh - would be nice to hear something from cf about the downtime.
Posted by EvolutionCrazy, 09-15-2012, 01:00 PM
up again since a few minutes ... hope this time it will last
Posted by AlanDu, 09-15-2012, 01:07 PM
It's back up, however my website is now loading very slowly.

Come on Cloudflare, don't let us down.
Posted by Amitz, 09-15-2012, 02:38 PM
Going up and down. Switched to my own DNS servers now...
Posted by UpReseller, 09-15-2012, 02:54 PM
Wow, one of my customers saw downtime when using CloudFlare. Thanks for this thread.
Posted by damoncloudflare, 09-15-2012, 03:28 PM
CloudFlare did have a system issue this morning that started around 8:40 AM PST. The issue is now resolved and we sincerely apologize to our customers for the access issues this would have caused to your sites. Again, we do apologize for the issues that were apparently caused by an upstream issue and lasted for about 1.5 hours.

Please note that we do try to post known network or datacenter issues on our Twitter handle.
Posted by damoncloudflare, 09-15-2012, 03:30 PM
Quote:
Originally Posted by EvolutionCrazy
Can't reach anything from most of the Europe (tested on multiple PRO and BUSINESS plans)... even their site is not reachable...

https://twitter.com/CloudFlareSys/st...97504950956032

This is the biggest/widespread cloudflare downtime I've seen since I've been a customer (nearly a year)

What's going on?

Connectivity to and from nlayer seems fine here...
Code:
  1    <1 ms    <1 ms    <1 ms  internet.gateway [192.168.0.200]
  2     *        *        *     Richiesta scaduta.
  3     *        9 ms     *     172.17.81.21
  4    10 ms    10 ms    10 ms  172.17.80.1
  5    18 ms    17 ms    17 ms  172.17.9.233
  6    19 ms    19 ms    16 ms  172.17.10.81
  7    17 ms    19 ms    15 ms  pos1-10-0-0.milano26.mil.seabone.net [195.22.192.29]
  8    30 ms    29 ms    30 ms  xe-2-2-0.parigi52.par.seabone.net [195.22.210.96]
  9    36 ms    36 ms    35 ms  xe-0-0-2.cr1.cdg1.fr.nlayer.net [69.22.139.49]
 10    36 ms    36 ms    36 ms  as13335.xe-0-0-1-302.cr1.cdg1.fr.nlayer.net [63.141.223.18]
 11    37 ms     *       37 ms  cf-173-245-61-250.cloudflare.com [173.245.61.250]
I do apologize for the service issues this morning that started around 8:40 AM PST. An apparent issue with upstream caused this and we do apologize to all of our customers for the service issues caused (the issue lasted about 1.5 hours).
Posted by damoncloudflare, 09-15-2012, 08:08 PM
Just posting a little bit more info about what we found today. Again, we apologize for the service issue.

We had a network interruption that affected many of our customers today. It was caused by a combination of factors, primarily a routing issue with one of our upstream bandwidth providers combined with an overly aggressive rate limit we applied to our network in order to mitigate a large denial of service (DDoS) attack directed at one of our customers.

The issue began at 16:36 GMT. We were able to revert the overly aggressive rate limit and route around the problem with our upstream provider by 17:00 GMT.

The issue affected different portions of our network of 23 global data centers differently. The problems were most pronounced in the EU region (and especially our London, Amsterdam, Paris and Frankfurt data centers) where we were at the regional peak of our traffic levels and therefore most likely to exceed the overly aggressive rate limit.

CloudFlare is designed to keep sites fast, safe and reliable so any outage is unacceptable to us. Our engineering and ops teams are have been working since the time of the outage to add additional protections to guard against what happened today and ensure it will not happen again in the future.
Posted by shovenose, 09-15-2012, 09:01 PM
Thanks for the detailed information, Damon!
Posted by damoncloudflare, 09-17-2012, 02:06 PM

Quote:

Originally Posted by shovenose

Thanks for the detailed information, Damon!

Would prefer it if we didn't have to post something like that, of course. We have a more detailed explanation on the blog about the issue as well.
Posted by CW Mike, 09-17-2012, 02:16 PM

Quote:

Originally Posted by damoncloudflare

Would prefer it if we didn't have to post something like that, of course. We have a more detailed explanation on the blog about the issue as well.

Yeah mate you guys worked your socks off and migrated it I think you all did good mate and I saw the blog post when you tweeted it.