Cloudflare Outage - Confirmed - Knowledgebase

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Cloudflare Outage - Confirmed

Cloudflare Outage - Confirmed

Posted by SajanP, 03-03-2013, 05:54 AM
Cloudflare's DNS is not resolving any domains that I know and have confirmed to be hosted through Cloudflare.

Went down around 3:45AM CST.

By not resolving any of the domains on the Cloudflare DNS, whether using CloudFlare or not, all of those sites are essentially down.
Posted by Branzone, 03-03-2013, 05:55 AM
Was about to make the same thread. Yes it's down here as well.
Posted by Mikeambrose3, 03-03-2013, 05:56 AM
Yep. Hopefully it will be fixed soon.
Posted by Hoopla-Brad, 03-03-2013, 05:58 AM
Their routes have been withdrawn. It looks like a network issue.
Posted by n!ghtmare, 03-03-2013, 05:58 AM
Yes. I don't use cloudflare for any sites but my host uses their DNS, now my email is not working..
Posted by SajanP, 03-03-2013, 06:03 AM
Quite a good chunk of the internet I use is just...gone.
Posted by UberTricep, 03-03-2013, 06:04 AM
All of my sites are currently down that use Cloudflare too. Worst things is I cannot even go on their website to disable it.
Posted by CyberHostPro, 03-03-2013, 06:04 AM
hi

it is down for me too including my own site that is used with their DNS.
Posted by Hoopla-Brad, 03-03-2013, 06:04 AM
Just came back.
Posted by Hosting55, 03-03-2013, 06:05 AM
ehh, all of my CF sites are down, including cpanel powered sites
also cpanel cloudflare modules are down, I can't edit my cpanel CF pages...
how can I remove CF from my cpanel sites without using cpanel CF module?
Posted by n!ghtmare, 03-03-2013, 06:05 AM

Quote:

Originally Posted by jamesa00789

All of my sites are currently down that use Cloudflare too. Worst things is I cannot even go on their website to disable it.

you disable it from your domain registrar. If you just turned off the cloudflare logo in their control panel your site would still be down because their dns is down.
Posted by Ethernet Servers, 03-03-2013, 06:06 AM
https://twitter.com/cloudflaresys
Posted by dqh, 03-03-2013, 06:08 AM
I can not believe cloudflare will have this status
Posted by Branzone, 03-03-2013, 06:08 AM

Quote:

Originally Posted by Hosting55

ehh, all of my CF sites are down, including cpanel powered sites
also cpanel cloudflare modules are down, I can't edit my cpanel CF pages...
how can I remove CF from my cpanel sites without using cpanel CF module?

For cPanel hosted sites: Edit the DNS for each domain. Need to change your www CNAME entry to "yourdomain.com." instead of the cloudflare hostname. I just did this and our site is back up without cloudflare. Also note that the DNS change will take however long the TTL is for the entry, ours was 300 because of situations like this so it worked almost instantly.
Posted by SajanP, 03-03-2013, 06:10 AM

Quote:

Originally Posted by Branzone

Edit the DNS for each domain. Need to change your www CNAME entry to "yourdomain.com." instead of the cloudflare hostname. I just did this and our site is back up without cloudflare. Also note that the DNS change will take however long the TTL is for the entry, ours was 300 because of situations like this so it worked almost instantly.

That will only work for those hosted on a cPanel server using the cPanel plugin. This won't work for anyone using Cloudflare directly.
Posted by CyberHostPro, 03-03-2013, 06:12 AM
resellerclub management area down who too i think use CF, grrr i am paying for the professionl package with CF to avoid downtime!! i think its time to move back to my own dns.
Posted by Branzone, 03-03-2013, 06:12 AM

Quote:

Originally Posted by SajanP

That will only work for those hosted on a cPanel server using the cPanel plugin. This won't work for anyone using Cloudflare directly.

Fixed.
Posted by UberTricep, 03-03-2013, 06:13 AM

Quote:

Originally Posted by n!ghtmare

you disable it from your domain registrar. If you just turned off the cloudflare logo in their control panel your site would still be down because their dns is down.

Alright, but changing the name servers could take hours. Also I cannot remember all of the DNS records to use with the domain registrar as I can't go on the Cloudflare site and look them up. Looks like I'm going to sit and wait
Posted by Hosting55, 03-03-2013, 06:15 AM

Quote:

Originally Posted by Branzone

For cPanel hosted sites: Edit the DNS for each domain. Need to change your www CNAME entry to "yourdomain.com." instead of the cloudflare hostname. I just did this and our site is back up without cloudflare. Also note that the DNS change will take however long the TTL is for the entry, ours was 300 because of situations like this so it worked almost instantly.

thanks, mine is 1400
Posted by Hosting55, 03-03-2013, 06:17 AM
fck.... resellerclub management is also down, they are on CF too!!!
I cannot change DNS to my domains.
oh god...
Posted by TmzHosting, 03-03-2013, 06:18 AM
It seems like CloudFlare is having major issues at the moment:

CloudFlare ‏@CloudFlare
We are experiencing a system wide issue. Our team is investigating. Will continue to update with information.

Some of our corporate websites run through CF, but now we can't even login to change anything since everything seems down. Who else is experiencing issues?

- Daniel
Posted by n!ghtmare, 03-03-2013, 06:18 AM
I hope the FBI finally shut them down. These morons think that they're above the law, refusing to shut down blatantly illegal sites, actually claiming that they CAN'T shut them down because they're "just a proxy service".
Posted by XiNiX, 03-03-2013, 06:19 AM
Confirmed its Down.
Our Server Down Since 6+ Hrs.
Posted by Showfom, 03-03-2013, 06:20 AM
Same here, their official website is down.
Posted by Ethernet Servers, 03-03-2013, 06:20 AM
If only people would stop tweeting cloudflare and let them fix the issue.
Posted by EvolutionCrazy, 03-03-2013, 06:20 AM
the whole network is down! not just the dns!
Posted by TmzHosting, 03-03-2013, 06:21 AM
CloudFlare ‏@CloudFlare
I don't have an ETA yet. As soon as I do, I will share.

This must be affecting a lot of people.

- Daniel
Posted by SajanP, 03-03-2013, 06:23 AM

Quote:

Originally Posted by EvolutionCrazy

the whole network is down! not just the dns!

Hoopla-Brad has confirmed that their routes are...gone.

They've literally VANISHED from the internet. Poof!
Posted by Neso, 03-03-2013, 06:24 AM
This affecting all people, even biggest sites ... really do not know what to say ...
Posted by XiNiX, 03-03-2013, 06:25 AM

Quote:

Originally Posted by Hosting55

fck.... resellerclub management is also down, they are on CF too!!!
I cannot change DNS to my domains.
oh god...

My Goodness.
Bitten by the Same Bug Mate..
Posted by LV-Matt, 03-03-2013, 06:25 AM

Quote:

Originally Posted by TmzHosting

It seems like CloudFlare is having major issues at the moment:

CloudFlare ‏@CloudFlare
We are experiencing a system wide issue. Our team is investigating. Will continue to update with information.

Some of our corporate websites run through CF, but now we can't even login to change anything since everything seems down. Who else is experiencing issues?

- Daniel

Any experienced technician should never be able to not login to change something, I am struck here by your <<comment>> especially considering you are offing a managed service.

For the Love of god, have you never heard of your computers Host file (Mac, Windows and Linux all have one). All our sites may be down, but we can still access them all as we have our hosts files edited accordingly.

Here is some help for you: http://helpdeskgeek.com/windows-7/windows-7-hosts-file/

http://decoding.wordpress.com/2009/0...-os-x-leopard/

I'm shocked that you would allow such an eventuality to happen.
Posted by Kevin Hillstrand, 03-03-2013, 06:27 AM
Frustrating. So much for the 100% uptime guarantee on the business plan.
Posted by Hosting55, 03-03-2013, 06:27 AM

Quote:

Originally Posted by XiNiX

My Goodness.
Bitten by the Same Bug Mate..

I hope resellerclub.com is not registered by resellerclub)
Posted by NetworkPanda, 03-03-2013, 06:29 AM

Quote:

Originally Posted by SajanP

Hoopla-Brad has confirmed that their routes are...gone.

They've literally VANISHED from the internet. Poof!

Their servers are not completely offline, from our tests, some of them respond for a few seconds and then they go offline again. It appears that there is a 90-95% packet loss on the entire network.

Edit: The packet loss of their nameservers now dropped to 50%, this is a good sign, maybe the problem will be fixed soon.
Posted by milliondollar, 03-03-2013, 06:29 AM
Hope it does not take too long for them to fix it. All my websites are down.
Posted by eva2000, 03-03-2013, 06:30 AM
Yup i have one site on Cloudflare that isn't resolving - luckily it's a not so important site. Important sites are on DNSMadeEasy's DNS.
Posted by Hosting55, 03-03-2013, 06:30 AM
resellerclub is up now at my end, I have changed my domains DNS, hope all will come back soon
Posted by EvolutionCrazy, 03-03-2013, 06:30 AM

Quote:

Originally Posted by Kevin Hillstrand

Frustrating. So much for the 100% uptime guarantee on the business plan.

well... thing about who had the 2000% plan AHAHAHAHAHA

btw: http://www.4chan.org still down too...

it has been 30minutes ... this is really beyond what one could expect from a service like that... everything went down! even their managment interface!

i wish I could at least login into the managment interface to export the bind config and reimport it back into http://www.incapsula.com/
Posted by dqh, 03-03-2013, 06:34 AM
Server returned error: 502 Server returned error information for request
Header are:

HTTP/1.1 502 Bad Gateway
Server: cloudflare-nginx
Date: Sun, 03 Mar 2013 10:34:48 GMT
Content-Type: text/html
Content-Length: 579
Connection: keep-alive
CF-RAY: 495ad24ce98041e
Set-Cookie: __cfduid=df27fc509640d335b7614ab7b3398c7971362306888; expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/; domain=.cloudflare.com
Posted by NetworkPanda, 03-03-2013, 06:36 AM
Things are gradually improving, the packet loss of their nameservers dropped from 90% (before 10 minutes) to 20% now. Some domains have started resolving again and we are now seeing traffic to our customers' sites which use CloudFlare gradually returning to normal.

Not back to 100% fully operational, but at least there is some improvement, let's hope it continues like this.

Their web site is still down, but I think their primary concern now is fixing the nameservers (so that all their customers sites come back online) and then fix the site.
Posted by Kailash12, 03-03-2013, 06:37 AM

Quote:

Originally Posted by Hosting55

I hope resellerclub.com is not registered by resellerclub)

Yes resellerclub.com is using Cloudflare DNS. I am not sure why they need to use third party DNS whereas they are offering managed DNS service to their own clients... Hopefully they will up soon...
Posted by Herenow, 03-03-2013, 06:43 AM
This is they're first world wide outage... correct?

All my websites are down... Well this will be for sure the topic of the week... Even if they're not down due to attacks, i guess probably this isnt caused by attacks due to they're network design. But it would be really intresting if it was indeed a attack IMO...

Probably a system failure?
Posted by cmaniac, 03-03-2013, 06:45 AM
Just here to confirm, I figured there'd be a thread.
I saw on their Twitter, that they were having 'issues'. Hope it's not too much longer as I am paying a good chunk a month for the Business package.
Posted by milliondollar, 03-03-2013, 06:51 AM
Cloudflare website back online. My websites back too
Posted by Daniel15, 03-03-2013, 06:52 AM
Someone must have tripped over the cable connecting The Cloud.
http://xkcd.com/908/

Glad to see they're slowly coming back up though
Posted by SajanP, 03-03-2013, 06:55 AM
Proxy service shows signs of life at 4:55AM CST. DNS has been back for roughly 15 minutes.
Posted by cmaniac, 03-03-2013, 06:55 AM
My website's back online, so are a few others behind CloudFlare. Seems like it's over.
Posted by dqh, 03-03-2013, 06:57 AM
01hr 00min down time..
Posted by Showfom, 03-03-2013, 06:59 AM
@CloudFlare:

Network update --> DNS coming back online. Your site should start to be accessible again.

https://twitter.com/CloudFlare/statu...69116802486272
Posted by Hosting55, 03-03-2013, 06:59 AM
all of my sites are back
Posted by LemonDrizzle, 03-03-2013, 07:00 AM
I honestly don't get why people decide to put in an additional point of failure in their infrastructure by introducing Cloudflare to the mix. More harm than good in my opinion, I hope that they still don't occasionally force legitimate potential customers and traffic to enter a captcha to continue - imagine that conversion loss.
Posted by ElitePixels, 03-03-2013, 07:01 AM
looks like they are resolving the issues now.

Most of the websites are back, CloudFlare itself is back now.

But the question is, how many are going to continue using it ?
I'm definitely not going to use CloudFlare after this issue.
Posted by Kailash12, 03-03-2013, 07:02 AM
They are up now...
Posted by SajanP, 03-03-2013, 07:03 AM

Quote:

Originally Posted by LemonDrizzle

I honestly don't get why people decide to put in an additional point of failure in their infrastructure by introducing Cloudflare to the mix. More harm than good in my opinion, I hope that they still don't occasionally force legitimate potential customers and traffic to enter a captcha to continue - imagine that conversion loss.

When it comes to Anycast DNS, proxy service, and global CDN...there are many that would trust a third party to handle that then set it up themselves. For many that NEED these things, Cloudflare is one of the only options.

The same reason that the vast majority of the planet don't run their own mail servers, but use the likes of GMail/Hotmail instead.
Posted by (Stephen), 03-03-2013, 07:03 AM
CF seems to be showing some signs of life now.
Posted by XiNiX, 03-03-2013, 07:04 AM
ALL my Sites are ONLINE.. FINALLY !!
Posted by LemonDrizzle, 03-03-2013, 07:05 AM

Quote:

Originally Posted by SajanP

When it comes to Anycast DNS, proxy service, and global CDN...there are many that would trust a third party to handle that then set it up themselves. For many that NEED these things, Cloudflare is one of the only options.

The same reason that the vast majority of the planet don't run their own mail servers, but use the likes of GMail/Hotmail instead.

Plenty of Anycast DNS and CDN providers out there, with proven track records and less rampant abuse that will likely lead to an eventual forced shutdown. Using as a part of a business infrastructure is nothing short of silly, especially given the issues they have had - and especially given the consistent abuse on the platform.

If you're running a serious business, please consider using a more reliable and less abuse tolerant platform.
Posted by EvolutionCrazy, 03-03-2013, 07:11 AM

Quote:

Originally Posted by SajanP

When it comes to Anycast DNS, proxy service, and global CDN...there are many that would trust a third party to handle that then set it up themselves. For many that NEED these things, Cloudflare is one of the only options.

The same reason that the vast majority of the planet don't run their own mail servers, but use the likes of GMail/Hotmail instead.

I'm not sure about that... there are plenty of alternatives... cloudflare seems just one of them... and these downtimes shouldn't happen...

I use DynECT for some of my domains and those in multiple years HAVE NEVER EVER showed the amount of downtime that cloudflare has collected :|
Posted by LemonDrizzle, 03-03-2013, 07:13 AM

Quote:

Originally Posted by EvolutionCrazy

I'm not sure about that... there are plenty of alternatives... cloudflare seems just one of them...

I use DynECT for some of my domains and those in multiple years HAVE NEVER EVER showed the amount of downtime that cloudflare has collected :|

I use Zerigo, personally. They did have an issue in July 2012 as the result of a wide scale DDoS attack - but they've made improvements to the platform as a result of this, so as opposed to some providers who have continued to have downtime; things have been solid.
Posted by SajanP, 03-03-2013, 07:16 AM

Quote:

Originally Posted by EvolutionCrazy

I'm not sure about that... there are plenty of alternatives... cloudflare seems just one of them... and these downtimes shouldn't happen...

My point is downtimes can happen to anyone. I think people are making a bit of a knee jerk reaction. What Cloudflare provides for free is an amazing deal for many sites.

With that said, when was the last time your site was completely down because of Cloudflare. This is the first time I can remember something like this happening like this.

I think people are having a bit of a knee-jerk reaction to a service many people use for free.

*I don't use Cloudflare for any of my sites, but a few of my clients do.
Posted by LemonDrizzle, 03-03-2013, 07:19 AM

Quote:

Originally Posted by SajanP

My point is downtimes can happen to anyone. I think people are making a bit of a knee jerk reaction. What Cloudflare provides for free is an amazing deal for many sites.

With that said, when was the last time your site was completely down because of Cloudflare. This is the first time I can remember something like this happening like this.

I think people are having a bit of a knee-jerk reaction to a service many people use for free.

*I don't use Cloudflare for any of my sites, but a few of my clients do.

Oh, I'm not disputing the fact it's a great service for free. I'm just pointing out that using it for a business seems a tad silly.
Posted by EvolutionCrazy, 03-03-2013, 07:19 AM
well.. i think people complaining here are using the paid service... at least I'm paiyng them each month...

if I was using a free version I wouldn't complain much
Posted by (Stephen), 03-03-2013, 07:20 AM

Quote:

Originally Posted by SajanP

My point is downtimes can happen to anyone. I think people are making a bit of a knee jerk reaction. What Cloudflare provides for free is an amazing deal for many sites.

With that said, when was the last time your site was completely down because of Cloudflare. This is the first time I can remember something like this happening like this.

I think people are having a bit of a knee-jerk reaction to a service many people use for free.

*I don't use Cloudflare for any of my sites, but a few of my clients do.

1. This is not the first time for CF.
2. CF paid and free not seemingly isolated from each other, good for the free, not for the paid....
3. No amount of money spent anywhere, can prevent all downtime. It is all in the recovery. 1 hour at this was not too good, but we shall see how it is handled.
Posted by SajanP, 03-03-2013, 07:20 AM

Quote:

Originally Posted by LemonDrizzle

Oh, I'm not disputing the fact it's a great service for free. I'm just pointing out that using it for a business seems a tad silly.

Ah, yes. My apologies. I do agree with that.

That said, it'd be pretty hard for an anycast service with the amount of locations they have to go down completely like it just did...twice.
Posted by mg-, 03-03-2013, 07:23 AM

Quote:

Originally Posted by LemonDrizzle

I honestly don't get why people decide to put in an additional point of failure in their infrastructure by introducing Cloudflare to the mix. More harm than good in my opinion, I hope that they still don't occasionally force legitimate potential customers and traffic to enter a captcha to continue - imagine that conversion loss.

Because services like cloudflare do wonders for websites that receive traffic
Posted by Bodybuilder, 03-03-2013, 07:25 AM

Quote:

Originally Posted by SajanP

*I don't use Cloudflare for any of my sites, but a few of my clients do.

Your site is on Cloudflare
sajanp.com
Posted by LemonDrizzle, 03-03-2013, 07:26 AM

Quote:

Originally Posted by mg-

Because services like cloudflare do wonders for websites that receive traffic

I have a site that has on average, anywhere between 4,000 and 6,000 active users at any point in the day. Around 100 API requests per second, and far more direct site requests per second. I introduced Cloudflare temporarily and it did more harm than good - consistent error messages of "Cloudflare does not have a copy..." etc. Another platform we're bringing online is estimated to receive around 15,000 requests per second at normal levels, and roughly four times that at peak hours. Cloudflare is definitely not on our agenda.

So, blanket statements such as "Cloudflare does wonders for websites that receive traffic" is a tad misleading.
Posted by (Stephen), 03-03-2013, 07:27 AM

Quote:

Originally Posted by mg-

Because services like cloudflare do wonders for websites that receive traffic

For a while they didn't, as they didn't receive traffic for an hour.
Ta ding!

Sorry, just had to add a light hearted reply among all the stress!
Posted by SajanP, 03-03-2013, 07:28 AM

Quote:

Originally Posted by Bodybuilder

Your site is on Cloudflare
sajanp.com

Fair enough, . You'll see that it's only for DNS and some testing. Which is why I happened to notice right away as it went down.

You can dig noppix.com and see we run our own DNS just fine.

My points remain.
Posted by Bodybuilder, 03-03-2013, 07:39 AM

Quote:

Originally Posted by SajanP

Fair enough, . You'll see that it's only for DNS and some testing. Which is why I happened to notice right away as it went down.

You can dig noppix.com and see we run our own DNS just fine.

My points remain.

You made the statement "*I don't use Cloudflare for any of my sites"

So you lied?? You need to admit the truth.

Say out loud "You i use Cloudflare for DNS, my site and some testing"stickout
Posted by SajanP, 03-03-2013, 07:42 AM

Quote:

Originally Posted by Bodybuilder

You made the statement "*I don't use Cloudflare for any of my sites"

So you lied?? You need to admit the truth.

Say out loud "You i use Cloudflare for DNS, my site and some testing"stickout

lmfao. Don't make me do it! It was only to play with their Rocketloader JS stuff and make sure it works for a client's site that has more javascript than should be legal. I don't use cPanel on the dev server nor do I have the cPanel plugin...so I had to move the entire domain over.

However, you are right *mumbles*...I do use cloudflare...*/mumbles*
Posted by mg-, 03-03-2013, 07:46 AM

Quote:

Originally Posted by LemonDrizzle

I have a site that has on average, anywhere between 4,000 and 6,000 active users at any point in the day. Around 100 API requests per second, and far more direct site requests per second. I introduced Cloudflare temporarily and it did more harm than good - consistent error messages of "Cloudflare does not have a copy..." etc. Another platform we're bringing online is estimated to receive around 15,000 requests per second at normal levels, and roughly four times that at peak hours. Cloudflare is definitely not on our agenda.

So, blanket statements such as "Cloudflare does wonders for websites that receive traffic" is a tad misleading.

I guess the thousands of sites that receive more traffic than you that don't have those problems must be an indication of sorcery. I guess there's really no benefit to things like anycast, minify/combine, image resizing on the fly for mobile devices, cdn, their railguns system.

But hey.. you had a bad experience.. that's why I hate broccoli.
Posted by SajanP, 03-03-2013, 07:49 AM

Quote:

Originally Posted by mg-

I guess the thousands of sites that receive more traffic than you that don't have those problems must be an indication of sorcery. I guess there's really no benefit to things like anycast, minify/combine, image resizing on the fly for mobile devices, cdn, their railguns system.

But hey.. you had a bad experience.. that's why I hate broccoli.

Nah, that's not it. I do have one other client that's hosted on a $15 VPS. Which if you saw the amount of requests per page (300+), you'd realize that Cloudflare is the only that it allows it to happen with 512MB RAM.

http://screen.sajanp.com/2013-03-03-05-47-16.png

They peak about 200 active visitors according to Google Real Time and the site runs like a champ.
Posted by Bodybuilder, 03-03-2013, 07:49 AM

Quote:

Originally Posted by SajanP

lmfao. Don't make me do it! It was only to play with their Rocketloader JS stuff and make sure it works for a client's site that has more javascript than should be legal. I don't use cPanel on the dev server nor do I have the cPanel plugin...so I had to move the entire domain over.

However, you are right *mumbles*...I do use cloudflare...*/mumbles*

I was playing with you. When should I come over for coffee??

I hope you got a cow because I'm taking all the cream.
Posted by LinuxNerd, 03-03-2013, 07:53 AM
I have to say it's a little strange for this system wide outage to occur given Cloudflare's setup. I hope there aren't bigger issues at play. At least for the legitimate customers/clients they have. Their DNS system is nice... But it's obviously vulnerable to something.
Posted by LemonDrizzle, 03-03-2013, 07:53 AM

Quote:

Originally Posted by mg-

I guess the thousands of sites that receive more traffic than you that don't have those problems must be an indication of sorcery. I guess there's really no benefit to things like anycast, minify/combine, image resizing on the fly for mobile devices, cdn, their railguns system.

But hey.. you had a bad experience.. that's why I hate broccoli.

Example of one personal site. As somebody who has consulted for more than one top 20 worldwide traffic sites in terms of performance, stability and architecture - I can tell you now that most would not consider running Cloudflare after today. "Hey, our redundant system went down." Not something I ever expect to hear.

Today has proven that Cloudflare is a bad idea for any serious business. It was already a bit of a laughing stock outside of the realms of people looking for cheap improvements, if only due to the almost unchecked abuse that was related to it, and the fact that the operators don't seem to give a damn about it.

Anycast DNS is available elsewhere. Minified content is hardly special. Image resizing on the fly for mobile devices is actually done by a lot of mobile operators (take Telefonica, Everything Everywhere and Vodafone as an example). If I want a CDN, I can look at things like Akamai or Cloudfront.

I have an infrastructure in place that does everything Cloudflare can do, only difference is - I wasn't down for an hour today losing traffic, reputation; and money.
Posted by Bodybuilder, 03-03-2013, 07:57 AM

Quote:

Originally Posted by LemonDrizzle

I have an infrastructure in place that does everything Cloudflare can do, only difference is - I wasn't down for an hour today losing traffic, reputation; and money.

Like a boss
Posted by mg-, 03-03-2013, 08:07 AM

Quote:

Originally Posted by LemonDrizzle

Example of one personal site. As somebody who has consulted for more than one top 20 worldwide traffic sites in terms of performance, stability and architecture - I can tell you now that most would not consider running Cloudflare after today. "Hey, our redundant system went down." Not something I ever expect to hear.

Today has proven that Cloudflare is a bad idea for any serious business. It was already a bit of a laughing stock outside of the realms of people looking for cheap improvements, if only due to the almost unchecked abuse that was related to it, and the fact that the operators don't seem to give a damn about it.

Anycast DNS is available elsewhere. Minified content is hardly special. Image resizing on the fly for mobile devices is actually done by a lot of mobile operators (take Telefonica, Everything Everywhere and Vodafone as an example). If I want a CDN, I can look at things like Akamai or Cloudfront.

I have an infrastructure in place that does everything Cloudflare can do, only difference is - I wasn't down for an hour today losing traffic, reputation; and money.

For companies that can afford to have the proper infrastructure in place then of course you wouldn't.. there's also the thousands of high traffic websites that don't have the cash to cover every different aspect required to guarantee uptime and have all the benefits to cut down on hardware costs.

For sites that don't sell a product or service directly but rely on income through other areas - it isn't feasible to spend what's required... well they could, but it wouldn't exactly be lining pockets.

So yes, would I put a shopping cart on it? No. But hey, every company is at risk of screwing up. Azure https blob/db was down for almost a day... because they let their ssl cert expire.

On a side note.. it doesn't matter who you've consulted. Top consultants in any field in any sector will bash their peers because of platform choice, methods, etc. Doesn't mean anything.

Every single company, platform, software, vehicle, toaster, diaper, peanut butter SUCKS.

I'll find an expert in any field that says so.
Posted by LemonDrizzle, 03-03-2013, 08:09 AM
Route 53 is cheap, minification is free, Akamai via Rackspace is only $0.12 per GB with no commitment required.

Good infrastructure doesn't necessarily cost the world.
Posted by LinuxNerd, 03-03-2013, 08:12 AM
It's unfortunate, no doubt. I just wonder if it's related to their blind-eye style abuse handling. As was said in this thread (and many areas on this board I have read recently) they effectively reply to any and all abuse with "we're a proxy service, go tell someone who cares".

Edit: Wrong URL.
Posted by SajanP, 03-03-2013, 08:14 AM

Quote:

Originally Posted by LemonDrizzle

Route 53 is cheap, minification is free, Akamai via Rackspace is only $0.12 per GB with no commitment required.

Good infrastructure doesn't necessarily cost the world.

Only? The screenshot of the stats I linked to above shows Cloudflare handling 8 mill requests and over 100GB/week. Let's say on a CDN that client gets billed for 400GB @ $0.12/GB...that's $50/mo + the fee for the requests (Cloudfront).

For a 2 person, overnight popular news site in a small country with zero revenue...that's a pretty big deal. May not be to you, me, and others here, but from the shoes of the person who was only paying $5/mo for shared hosting just a year ago...that's quite a monthly expense to add on top of what they are already paying...for a hobby.

Stick to your top sites and awesome infrastructing, please stop crapping on people who use Cloudflare because they just went down for an hour.

Rackspace has gone down.
Amazon has gone down.
Hell...an IBM datacenter in NZ was down for DAYS a couple weeks ago.

Edit: http://screen.sajanp.com/2013-03-03-05-47-16.png

Yeah. For this site paying just $15/mo TOTAL...adding $50/mo for CDN is a bit much.
Posted by Bodybuilder, 03-03-2013, 08:17 AM

Quote:

Originally Posted by mg-

Every single company, platform, software, vehicle, toaster, diaper, peanut butter SUCKS.

Did you take my peanut butter??
Posted by NetworkPanda, 03-03-2013, 08:27 AM

Quote:

Originally Posted by LinuxNerd

It's unfortunate, no doubt. I just wonder if it's related to their blind-eye style abuse handling. As was said in this thread (and many areas on this board I have read recently) they effectively reply to any and all abuse with "we're a proxy service, go tell someone who cares".

Here's a full update: http://blog.cloudflare.com/todays-outage-post-mortem

Have to laugh at the damage minimization:

Was about an hour here.

This link was about another outage, May 2012. They have not yet posted an explanation about the outage which occurred today.
Posted by LinuxNerd, 03-03-2013, 08:32 AM
Yeah sorry, realized after I saw the date. Apparently some edge routers caused a stuff up.

http://techcrunch.com/2013/03/03/clo...metallica-com/
Posted by LemonDrizzle, 03-03-2013, 08:36 AM
At any time, has the entire AWS infrastructure gone down? I am aware that specific regions and availability zones have gone down, but I don't recall the entire platform dropping off of the face of the Earth. I could be wrong on this point.

In regards to Rackspace going down, I can't really comment on them - I am only aware that their CDN is powered by Akamai, and nobody can really argue against the fact that Akamai is damn near as good as it gets.

My comment in regards to Akamai was mostly targeted at people saying "they won't talk to me sub-$3k!". Maybe not direct, but you can go through somebody else to get access to the same infrastructure.

My argument against Cloudflare for people to use services such as those listed is for people running an actual business. Not the hobbyist news site, but people running a business generating viable revenue.
Posted by SajanP, 03-03-2013, 08:45 AM

Quote:

Originally Posted by LemonDrizzle

At any time, has the entire AWS infrastructure gone down? I am aware that specific regions and availability zones have gone down, but I don't recall the entire platform dropping off of the face of the Earth. I could be wrong on this point.

In regards to Rackspace going down, I can't really comment on them - I am only aware that their CDN is powered by Akamai, and nobody can really argue against the fact that Akamai is damn near as good as it gets.

My comment in regards to Akamai was mostly targeted at people saying "they won't talk to me sub-$3k!". Maybe not direct, but you can go through somebody else to get access to the same infrastructure.

My argument against Cloudflare for people to use services such as those listed is for people running an actual business. Not the hobbyist news site, but people running a business generating viable revenue.

To the first point, no, Amazon as a whole has not gone off the face of the planet. However, they have had an incident where all their products were listed at $49.99 for a about 30 minutes. (Awesomely, they honored their goof ups and fullfilled the orders I think). Many of the Netflix outages are S3 related as well.

Yes, AZs go down and when you're affected...your thought is never, 'well at least other Amazon customers are up.' Down is down. Bring the point full circle, down is down. Whether your on your own infrastructure, paying out the nose, or paying nothing...it's always a possibility and looking down on one thing vs the other is silly.

Point is again, nobody's perfect.

To the second bolded point, I completely agree and that's not what I was arguing against. Obviously if you're a business and generate revenue, you're wanting to set your own stuff up. There's no doubt. Nobody would trust a business stealing wifi off someone else to process their payments.

At the same time, there ARE non tech related businesses on this planet. For many of them, Cloudflare is a good solution and one I'd continue to recommend. I personally have never experience them going down before, just tonight. As it turns out it's their 3rd time in history.

For a business without many IT resources or budget, Cloudflare will continue to be a good option for an easy CDN and optimizer. Try teaching the local flower shop owner how to use W3 Total Cache, and explaining when to purge the cache and how to invalidate CDN objects. That person would rather absorb 5 hours of downtime in 2 years than deal with the headache (in their view) of setting things up properly.

*I understand most FLOWER SHOPS won't need Cloudflare, but...you get the point.
Posted by LemonDrizzle, 03-03-2013, 08:47 AM
People relying on a single AWS availability zone are doing it wrong ;-). I've experienced a number of false positive sandbox requests with Cloudflare, and far too high a number of "this site is unavailable" messages when using Cloudflare. This sentiment appears to be echoed over on HN.
Posted by SajanP, 03-03-2013, 08:50 AM

Quote:

Originally Posted by LemonDrizzle

People relying on a single AWS availability zone are doing it wrong ;-).

I can't figure out if you're trolling or not now. If you want to have a contest on who's willy is bigger, you've won. Go back to consulting for your worldwide top 20 sites.

There ARE use cases for AWS where you deploy to a single AZ. Maybe not in your world, but they exist.

If your point right now is that any provider other than Cloudflare is 100% perfect and better for all businesses and cases that generate revenue, you're simply out of your mind.

Edit: Sorry. I meant regions not AZs. It's 7AM...holy hell. I'm headed to bed.
Posted by LemonDrizzle, 03-03-2013, 08:52 AM

Quote:

Originally Posted by SajanP

I can't figure out if you're trolling or not now. If you want to have a contest on who's willy is bigger, you've won. Go back to consulting for your worldwide top 20 sites.

There ARE use cases for AWS where you deploy to a single AZ. Maybe not in your world, but they exist.

If your point right now is that any provider other than Cloudflare is 100% perfect and better for all businesses and cases that generate revenue, you're simply out of your mind.

That was an obvious tongue-in-cheek comment, please don't get your panties in a twist. I'm referring to people who use AWS for HA solutions and then complain when the single AZ they are in goes down (and yes, people do that and complain). I am not referring to people with use cases that are viable for a single availability zone, that's a different thing.

Edit: I was going to say (in response to your edit), there are far more reasons for single-region as opposed to single-az.
Posted by SajanP, 03-03-2013, 08:58 AM

Quote:

Originally Posted by LemonDrizzle

That was an obvious tongue-in-cheek comment, please don't get your panties in a twist. I'm referring to people who use AWS for HA solutions and then complain when the single AZ they are in goes down (and yes, people do that and complain). I am not referring to people with use cases that are viable for a single availability zone, that's a different thing.

Edit: I was going to say (in response to your edit), there are far more reasons for single-region as opposed to single-az.

Why would anyone be deploying to the same AZ for anything remotely related HA?

With that said, I have seen some scripts hardcode the AZ for absolutely no reason.

Which is your point, and I'm not arguing against you. Just saying, there are many people that use Cloudflare that shouldn't be looked down upon for doing so.

/night
Posted by LemonDrizzle, 03-03-2013, 09:06 AM

Quote:

Originally Posted by SajanP

Why would anyone be deploying to the same AZ for anything remotely related HA?

With that said, I have seen some scripts hardcode the AZ for absolutely no reason.

Which is your point, and I'm not arguing against you. Just saying, there are many people that use Cloudflare that shouldn't be looked down upon for doing so.

/night

I have no clue, sadly - people do it though.

Oh, I'm not looking down upon them - I'm just bemused as to why some people put what they deem "mission critical" sites on there. It all depends on the use case scenario at the end of the day.
Posted by SajanP, 03-03-2013, 09:13 AM

Quote:

Originally Posted by LemonDrizzle

I have no clue, sadly - people do it though.

Oh, I'm not looking down upon them - I'm just bemused as to why some people put what they deem "mission critical" sites on there. It all depends on the use case scenario at the end of the day.

I kid you not. There's a popular blog article out there somewhere with a tutorial on how to setup a 'CDN'...by serving content on from the same server, just a different Apache VirtualHost.

http://www.riyaz.net/blogging/setup-own-cdn/890/

Edit: Just looked at it again, and it's not even a different VHost. The whole tutorial is pointing a CNAME to your main content.

Also, I did mean regions above as I assumed we were talking redundancy. AWS has two regions in the US and there's no reason not to use both if that's your goal. Only reason you'd really want to use the same region is if you don't want to pay for the bandwidth in and out of your difference components or replications (again, if the goal is redundancy).

Okay, now I promise. If you ping me, I will ignore. I need sleep. Shoot me an email if you're up for bickering further.
Posted by TheLie, 03-03-2013, 10:01 AM
Seems to be resolved now.
Posted by net, 03-03-2013, 10:05 AM

Quote:

Originally Posted by Zhang

Seems to be resolved now.

How long it went down? Do you know what was the problem?
Posted by TheLie, 03-03-2013, 10:05 AM
30-45min, a failure in Junipers FlowSpec implementation:
http://blog.cloudflare.com/todays-ou...t-mortem-82515
Posted by Steven, 03-03-2013, 10:33 AM

Quote:

Originally Posted by LemonDrizzle

Oh, I'm not looking down upon them - I'm just bemused as to why some people put what they deem "mission critical" sites on there. It all depends on the use case scenario at the end of the day.

Good marketing..
Posted by NetworkPanda, 03-03-2013, 11:01 AM
They have posted a detailed blog entry for today's outage here:
http://blog.cloudflare.com/todays-ou...t-mortem-82515
Posted by Appdeveloper, 03-03-2013, 01:19 PM
If it wasn't for CloudFlare, My site (hosted by HostGator) would have 100% uptime for the past 3 months.

CloudFlare has these "minute downtimes" at random as well.
Posted by LemonDrizzle, 03-03-2013, 01:26 PM

Quote:

Originally Posted by Zhang

30-45min, a failure in Junipers FlowSpec implementation:

I wouldn't blame Juniper as much as CF's operational policy of the ops team pushing untested changes globally.
Posted by patchwork, 03-03-2013, 03:15 PM
The CF outage got me thinking and wondering if my DNS settings are correct

When CF went offline I swapped my DNS settings back to my old settings to minimize the impact.

I was just changing them back to CF when I started wondering if I was doing the settings wrong.

Would it actually be better to set sites to use 3 or 4 DNS settings as shown below, so if cloudflare was unavailable for what ever reason the original DNS would take over again adding a level of redundancy.

mark.ns.cloudflare.com
pat.ns.cloudflare.com
DNS1.zzzzzzzzzzzz.com
DNS2.yyyyyyyyyyyy.com

Would this work and be better?
Posted by NetworkPanda, 03-03-2013, 03:56 PM

Quote:

Originally Posted by patchwork

The CF outage got me thinking and wondering if my DNS settings are correct

When CF went offline I swapped my DNS settings back to my old settings to minimize the impact.

I was just changing them back to CF when I started wondering if I was doing the settings wrong.

Would it actually be better to set sites to use 3 or 4 DNS settings as shown below, so if cloudflare was unavailable for what ever reason the original DNS would take over again adding a level of redundancy.

mark.ns.cloudflare.com
pat.ns.cloudflare.com
DNS1.zzzzzzzzzzzz.com
DNS2.yyyyyyyyyyyy.com

Would this work and be better?

Yes, this will work in terms of availability of your site, but some random requests to your site will not be passing through CloudFlare even when it is online, they will be reaching your site directly.

When a DNS query is made for a domain, one random nameserver is used, between these specified in the registrar. It does not always start from the 1st, then go to the 2nd etc. For example, some random queries will directly check the 3rd or the 4th. This is how DNS system works. So, some random queries will be directly checking your 3rd and 4th nameservers you have setup and the visitors will not be entering your site through CloudFlare, even if the CloudFlare DNS are online.

If this is ok for you, then you can setup the nameservers this way.
Posted by patchwork, 03-03-2013, 04:55 PM

Quote:

Originally Posted by NetworkPanda

Yes, this will work in terms of availability of your site, but some random requests to your site will not be passing through CloudFlare even when it is online, they will be reaching your site directly.

When a DNS query is made for a domain, one random nameserver is used, between these specified in the registrar. It does not always start from the 1st, then go to the 2nd etc. For example, some random queries will directly check the 3rd or the 4th. This is how DNS system works. So, some random queries will be directly checking your 3rd and 4th nameservers you have setup and the visitors will not be entering your site through CloudFlare, even if the CloudFlare DNS are online.

If this is ok for you, then you can setup the nameservers this way.

I've been searching around on the net and it seems there is a lot of different views on which order they are chosen

Here one quite good page
http://serverfault.com/questions/130...nameserver-hit

So far I'm leaning towards 3 DNS entries.

If a client chooses randomly then it will choose CF 66% of the time.
if it chooses sequentially then it will pick CF 99% of the time.
If it chooses based on response time then CF could well win and be chosen.
Geographic based choices could also choose CF over my DNS most of the time.

I could be way off, but this could give the extra redundancy while still receiving the benefits of CF possibly 80% of the time (maybe even higher).

Pete
Posted by CNSERVERS, 03-03-2013, 05:06 PM
That's why Nlayer no longer support Flowspec.
Posted by LinuxNerd, 03-03-2013, 05:32 PM

Quote:

Originally Posted by patchwork

So far I'm leaning towards 3 DNS entries.

If you're only using them for DNS it wouldn't be so bad. Perhaps a few of your look ups may be faster/slower depending on locations. I've seen websites make use of cloudflare and other DNS services, including their own.

Just make sure you remember to edit records in two locations
Posted by UpReseller, 03-04-2013, 04:08 AM
We are glad, because this is temporary issue only.
Posted by blueraindesigns, 03-04-2013, 04:19 AM
Personally I'm impressed at how they handled it, and considering how big a deal it was, the downtime was pretty minimal. Doesn't hurt my perception of them at all.
Posted by SoftDux, 03-04-2013, 05:58 AM
So, and not to flame CloudFlare, how does one actually get good uptime from a service like this? I know everyone has problems, etc, but surely something like CloudFlare (aka CDN) is supposed to give better uptime due to their distributed setup?
Posted by Time4VPS, 03-04-2013, 06:03 AM

Quote:

Originally Posted by SoftDux

So, and not to flame CloudFlare, how does one actually get good uptime from a service like this? I know everyone has problems, etc, but surely something like CloudFlare (aka CDN) is supposed to give better uptime due to their distributed setup?

When you propagate wrong configs to that "distributed setup" you get distributed error.

Biggest and most often errors cause is human. Same for CF - hardware / network perfected, techs / procedures - not. And here you go... 1 hour total outage is huge drop in a market like CDN's.
Posted by IRCCo Jeff, 03-04-2013, 10:34 AM

Quote:

Originally Posted by SoftDux

So, and not to flame CloudFlare, how does one actually get good uptime from a service like this? I know everyone has problems, etc, but surely something like CloudFlare (aka CDN) is supposed to give better uptime due to their distributed setup?

Redundancy through distributed setup assumes that you're not pushing untested changes using a rather experimental protocol to globally distributed edge routers on the fly, without the appropriate engineers standing by.
Posted by charlie23, 03-04-2013, 01:24 PM

Quote:

Originally Posted by time4vps

And here you go... 1 hour total outage is huge drop in a market like CDN's.

I completely agree with you
Posted by layer0, 03-04-2013, 02:25 PM

Quote:

Originally Posted by SoftDux

So, and not to flame CloudFlare, how does one actually get good uptime from a service like this? I know everyone has problems, etc, but surely something like CloudFlare (aka CDN) is supposed to give better uptime due to their distributed setup?

Cloudflare cannot increase the uptime of your site. Ultimately, your origin server still needs to be up for your site to work properly. Cloudflare can only increase the downtime of your site, as you're adding in another layer that represents a point of failure.

Maybe a decent idea is to manage the DNS yourself, and use a combination of Cloudflare + Incapsula with round robin A records, and have a system which removes the A record of the failed service (if there is one) automatically. That's assuming you are really hard set on using one of these gimmicky services to begin with.
Posted by Bodybuilder, 03-04-2013, 02:27 PM
@layer0

Want to start poll about Cloudflare??
Posted by layer0, 03-04-2013, 02:32 PM

Quote:

Originally Posted by Bodybuilder

@layer0

Want to start poll about Cloudflare??

Feel free to do so if you wish; in my opinion there is typically no performance benefit if your origin server is already reasonably close to your target audience (which isn't hard to accomplish). If you really have a scattered user base you may see improvements in some situations, but it'll probably hurt performance for some users too. This is based on real world results, not theoretical.
Posted by Bodybuilder, 03-04-2013, 02:36 PM
I don't use Cloudflare..... Was just thinking about the poll, lol

Might start WW3, lol
Posted by blueraindesigns, 03-04-2013, 02:52 PM

Quote:

Originally Posted by layer0

Cloudflare cannot increase the uptime of your site. Ultimately, your origin server still needs to be up for your site to work properly.

When cloudflare is working normally, which before this was pretty much always... then even if your origin server goes down, cloudflare serves your site to visitors if you have that setting enabled. When that happens, it lets the visitor know that is happening (at least on the free version, I don't know about the paid versions).
Posted by patchwork, 03-04-2013, 02:55 PM

Quote:

Originally Posted by layer0

Cloudflare cannot increase the uptime of your site. Ultimately, your origin server still needs to be up for your site to work properly. Cloudflare can only increase the downtime of your site, as you're adding in another layer that represents a point of failure.

Maybe a decent idea is to manage the DNS yourself, and use a combination of Cloudflare + Incapsula with round robin A records, and have a system which removes the A record of the failed service (if there is one) automatically. That's assuming you are really hard set on using one of these gimmicky services to begin with.

It has saved me a lot of bandwidth and server resources and pages do load a lot faster.
Posted by patchwork, 03-04-2013, 03:05 PM

Quote:

Originally Posted by layer0

Feel free to do so if you wish; in my opinion there is typically no performance benefit if your origin server is already reasonably close to your target audience (which isn't hard to accomplish). If you really have a scattered user base you may see improvements in some situations, but it'll probably hurt performance for some users too. This is based on real world results, not theoretical.

I have traffic from all over the world and it has only helped my sites, we had 83 million requests in the last 30 days and CF served 51 million of them, that's a huge saving in server resources. It means my server can concentrate on generating dynamic pages and CF does the rest.
Posted by layer0, 03-04-2013, 03:24 PM

Quote:

Originally Posted by Bodybuilder

I don't use Cloudflare..... Was just thinking about the poll, lol

Might start WW3, lol

Over the years on WHT especially recently people have this tendency to overcomplicate hosting sometimes. It's the same comparison I'll make that I've got single server setups that are more reliable than some gimmicky cloud hosting services that supposedly have ridiculous amounts of redundancy. The more layers you add to your infrastructure the more stuff can break. More chance of human error too.

Quote:

Originally Posted by blueraindesigns

When cloudflare is working normally, which before this was pretty much always... then even if your origin server goes down, cloudflare serves your site to visitors if you have that setting enabled. When that happens, it lets the visitor know that is happening (at least on the free version, I don't know about the paid versions).

Not really. If you have a dynamic site like most people, if your origin is down you are screwed.

Quote:

Originally Posted by patchwork

I have traffic from all over the world and it has only helped my sites, we had 83 million requests in the last 30 days and CF served 51 million of them, that's a huge saving in server resources. It means my server can concentrate on generating dynamic pages and CF does the rest.

I do understand the benefits from a resource saving standpoint, but in all the tests we've done there seriously isn't any major performance improvement, and often times a reduction.

If you want to optimize the performance of static objects, then offload them to a real CDN like edgecast who doesn't have all these outages and doesn't require you to share a resource pool with tons of kiddie sites on the same network like Cloudflare would.
Posted by glinski, 03-04-2013, 03:57 PM
I'm having trouble with my site since friday...

A lot of CloudFlare errors like "Site offline. No cached version avaiable"

My hosting company(Knownhost) could not find any downtime from their end. Could it be this outage ?

Anyone with a similar problem ?
Posted by Orien, 03-04-2013, 04:01 PM

Quote:

Originally Posted by glinski

I'm having trouble with my site since friday...

A lot of CloudFlare errors like "Site offline. No cached version avaiable"

My hosting company(Knownhost) could not find any downtime from their end. Could it be this outage ?

Anyone with a similar problem ?

Contact CloudFlare support and they'll help you out.

Since this outage is resolved, I'm closing this thread. If anyone would like to continue discussing CloudFlare, feel free to create a new discussion thread in our main forums.