Cloudflare issues/packetloss

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Cloudflare issues/packetloss

Posted by AG-Arco, 03-23-2013, 09:25 AM
It seems that a lot of Cloudflare sites (including their own) don't seem to be loading right now.

Does anyone know what's happening?
Posted by Benjamin|UNIT, 03-23-2013, 09:26 AM
Everything is just fine from here.
Posted by AG-Arco, 03-23-2013, 09:27 AM
I'd show you a screenshot of Justping, however, my site is behind Cloudflare and it seems to be down from here as well unfortunately. There's packet loss for most European locations.

Quote:

Amsterdam2, Netherlands: Packets lost (30%) 29.4 29.6 30.0 2400:cb00:2048:1::8d65:71d8
Florida, U.S.A.: Okay 28.8 29.0 29.3 141.101.113.216
Amsterdam3, Netherlands: Packets lost (90%) 28.5 28.5 28.5 2400:cb00:2048:1::8d65:71d8
Sydney, Australia: Okay 0.3 0.8 1.3 141.101.123.216
New York, U.S.A.: Okay 11.8 11.9 12.2 2400:cb00:2048:1::8d65:71d8
Stockholm, Sweden: Packets lost (60%) 87.2 90.1 93.4 2400:cb00:2048:1::8d65:7bd8
Santa Clara, U.S.A.: Okay 3.5 4.2 9.2 141.101.113.216
Vancouver, Canada: Okay 5.0 5.1 5.5 2400:cb00:2048:1::8d65:7bd8
London, United Kingdom: Packets lost (70%) 35.9 36.1 36.6 2400:cb00:2048:1::8d65:71d8
Madrid, Spain: Okay 44.8 45.0 45.3 141.101.113.216
Padova, Italy: Okay 118.9 119.7 121.8 2400:cb00:2048:1::8d65:7bd8
Singapore, Singapore: Okay 4.3 5.3 6.2 141.101.113.216
Austin, U.S.A.: Okay 6.8 7.0 7.3 2400:cb00:2048:1::8d65:71d8
Cologne, Germany: Packets lost (100%) 2400:cb00:2048:1::8d65:71d8
MÃ¼nchen, Germany: Packets lost (40%) 44.8 45.2 45.7 2400:cb00:2048:1::8d65:7bd8
Amsterdam, Netherlands: Okay 1.0 1.1 1.7 2400:cb00:2048:1::8d65:7bd8
Shanghai, China: Okay 142.6 143.4 144.6 141.101.123.216
Hong Kong, China: Okay 155.6 155.9 156.6 2400:cb00:2048:1::8d65:7bd8
Melbourne, Australia: Okay 92.9 93.1 93.5 2400:cb00:2048:1::8d65:71d8
Lille, France: Packets lost (30%) 18.6 25.7 31.8 141.101.123.216
San Francisco, U.S.A.: Checkpoint temporarily not available - - - -
Zurich, Switzerland: Packets lost (60%) 44.2 44.6 44.9 2400:cb00:2048:1::8d65:71d8
Mumbai, India: Okay 99.8 123.2 141.9 141.101.123.216
Auckland, New Zealand: Okay 23.9 24.1 24.4 141.101.123.216
Groningen, Netherlands: Okay 15.6 18.8 20.7 2400:cb00:2048:1::8d65:71d8
Antwerp, Belgium: Okay 3.5 13.9 54.9 141.101.113.216
Dublin, Ireland: Okay 14.6 14.8 15.1 141.101.123.216
Kharkov, Ukraine: Okay 53.3 53.6 53.9 141.101.113.216
Manchester, United Kingdom: Packets lost (10%) 16.7 17.8 19.2 141.101.123.216
Vilnius, Lithuania: Okay 32.0 32.5 33.1 141.101.123.216
Bucharest, Romania: Okay 133.7 142.7 154.7 141.101.123.216
Bangkok, Thailand: Packets lost (100%) 141.101.123.216
Kuala Lumpur, Malaysia: Okay 199.5 199.7 200.2 141.101.123.216
Jakarta, Indonesia: Packets lost (100%) 141.101.123.216
Cape Town, South Africa: Okay 159.1 161.9 163.5 141.101.113.216
Glasgow, United Kingdom: Okay 1.3 1.5 2.0 141.101.123.216
Lisbon, Portugal: Okay 44.4 44.6 45.4 141.101.123.216
Chicago, U.S.A.: Okay 1.3 22.6 206.0 2400:cb00:2048:1::8d65:71d8
Dallas, U.S.A.: Okay 1.7 2.4 4.2 141.101.113.216
Istanbul, Turkey: Okay 46.0 46.6 47.5 141.101.123.216
Gdansk, Poland: Packets lost (70%) 56.6 59.0 63.4 2400:cb00:2048:1::8d65:71d8
Cairo, Egypt: Okay 54.9 56.4 59.0 141.101.113.216
Buenos Aires, Argentina: Okay 128.4 131.0 132.1 141.101.113.216
Belgrade, Serbia: Okay 28.3 28.9 29.3 141.101.113.216
Toronto, Canada: Okay 1.2 1.4 1.4 141.101.113.216
Athens, Greece: Packets lost (90%) 131.1 131.1 131.1 2400:cb00:2048:1::8d65:71d8
Frankfurt, Germany: Packets lost (80%) 40.6 40.8 41.0 2400:cb00:2048:1::8d65:71d8
Sofia, Bulgaria: Okay 38.4 38.6 38.9 141.101.113.216
Budapest, Hungary: Okay 15.5 15.6 15.8 141.101.123.216
Sao Paulo, Brazil: Packets lost (10%) 106.6 106.9 107.1 141.101.113.216
Paris, France: Okay 8.3 8.5 8.8 141.101.113.216
Mumbai, India: Okay 96.1 97.3 102.3 141.101.113.216
New Delhi, India: Okay 106.4 110.0 120.7 141.101.123.216
Chicago, U.S.A.: Okay 2.7 5.0 7.0 2400:cb00:2048:1::8d65:7bd8
Bangalore, India: Packets lost (100%) 141.101.123.216
Montreal, Canada: Okay 19.8 21.6 27.6 2400:cb00:2048:1::8d65:7bd8
Tokyo, Japan: Okay 1.4 1.6 1.8 141.101.123.216
Milan, Italy: Packets lost (60%) 45.9 46.6 48.2 2400:cb00:2048:1::8d65:71d8
Hangzhou, China: Okay 301.5 314.0 325.8 141.101.113.216
Copenhagen, Denmark: Okay 116.9 117.0 117.4 2400:cb00:2048:1::8d65:7bd8
Phoenix, U.S.A.: Okay 9.6 9.8 10.0 141.101.123.216
Miami, U.S.A.: Okay 33.2 33.9 37.1 141.101.113.216

Posted by NetworkPanda, 03-23-2013, 09:38 AM
It appears that the outage affects their main site too. It is 2 hours now we can't access cloudflare.com to modify the DNS entries for some domains.

It gives errors like this

Quote:

This website is offline

No cached version is available.

The most likely causes:

The server is down for maintenance
There may be a network problem
The site may be experiencing excessive load

Posted by AG-Arco, 03-23-2013, 09:40 AM
I have just checked and my US servers can still access all the sites perfectly.

Will this be covered under the SLA for Pro users? They promise at least 99.9% uptime. :/
Posted by Benjamin|UNIT, 03-23-2013, 09:40 AM
I use CloudFlare And my site is up @ https://edgegameservers.com
And CloudFlare's website works just fine for mwe as well.
Posted by AG-Arco, 03-23-2013, 09:43 AM

Quote:

Originally Posted by Benjamin|UNIT

I use CloudFlare And my site is up @ https://edgegameservers.com
And CloudFlare's website works just fine for mwe as well.

Only European users are affected by this outage.

When I try to go to your site I see Error 103 "(net::ERR_CONNECTION_ABORTED): Unknown error."
Posted by Loon, 03-23-2013, 09:54 AM
Slight title change, as there are no issues for some. Appears up for me.
Posted by NetworkPanda, 03-23-2013, 10:03 AM
After some further checks: We can't access CloudFlare from Europe and Georgia, USA (I don't know if it affects only this US location or the entire east coast).
We are able to access it without any problems from Los Angeles, California and other west coast locations.
Posted by AG-Arco, 03-23-2013, 10:09 AM

Quote:

Originally Posted by NetworkPanda

After some further checks: We can't access CloudFlare from Europe and Georgia, USA (I don't know if it affects only this US location or the entire east coast).
We are able to access it without any problems from Los Angeles, California and other west coast locations.

I've got the same readings here. Cloudflare doesn't seem to know yet.
Posted by HocusPocus, 03-23-2013, 10:18 AM
Yep most of my sites that are hosted on cloudflare are down right now. A simple wget timesout after 60 seconds, however if i access the site directly i get a response within 120ms.

When i get time next week, i'll almost certainly be moving away from them.
Posted by AG-Arco, 03-23-2013, 10:24 AM
I know it's Saturday, but come on... They have to have someone at their NOC or global sysadmin team that works weekends.. 2 hours and not even a word from them seems pretty much below average.
Posted by NetworkPanda, 03-23-2013, 10:29 AM

Quote:

Originally Posted by chulo arco

I know it's Saturday, but come on... They have to have someone at their NOC or global sysadmin team that works weekends.. 2 hours and not even a word from them seems pretty much below average.

The previous outage which occurred on a Sunday about 4 am EST was resolved within 50 minutes. For this one we have no any news and it already more than 2 hours...
Posted by F-DNS, 03-23-2013, 11:52 AM

Quote:

Originally Posted by NetworkPanda

For this one we have no any news and it already more than 2 hours...

WHT isn't the sole source of news. Cloudflare are pretty good at posting announcements on Twitter whenever they spot a problem...

https://twitter.com/cloudflaresys

Follow them and sign up for free SMS alerts and your phone will buzz whenever they tweet, and the text of the tweet is contained in the SMS so you don't even have to be on the Net
Posted by IRCCo Jeff, 03-23-2013, 12:25 PM
It's fairly routine that multiple CloudFlare nodes are under attack, and of course not at all surprising that those nodes would have packet loss. Their engineering spends copious amounts of time fiddling with FlowSpec and diverting traffic to isolate attacked sites.
Posted by Appach, 03-28-2013, 08:47 PM
Seems normal for me.
Posted by Violent Injection, 03-28-2013, 10:14 PM
Prolly because the thread is from 5 days ago.
Posted by paradisewebdesign, 03-28-2013, 10:38 PM
http://blog.cloudflare.com/