Hetzner massive packet loss

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Hetzner massive packet loss

Posted by Neso, 11-17-2013, 09:49 AM
Hi, is anybody else have problems with hetzner, it look like they are 2 days under some sort of attack, and most of my servers with them have massive problems, packet loss and downtime ...
Posted by gingir, 11-17-2013, 09:50 AM
Same here
Posted by EvolutionCrazy, 11-17-2013, 09:54 AM
yeah... since tonight I keep getting issues...

http://www.hetzner-status.de/en.html

They keep posting issues about "attacks" but things keeps working badly... looks like they are not able to handle it.
Posted by Neso, 11-17-2013, 09:55 AM
Now almost all my 15 servers is down, I have expecting this to happen sometime, and already started to moving clients to better provider ...
Posted by gingir, 11-17-2013, 10:01 AM
Every server I have with Hetzner is currently not reachable.
Posted by NetworkPanda, 11-17-2013, 10:22 AM
Our servers are back online now, so it seems they fixed the issues.
Posted by Neso, 11-17-2013, 10:24 AM
No they did not: http://cloudmonitor.ca.com/en/ping.php check here your server IP, most of locations have still massive packet loss.

My servers are online only from some locations, but there is still random offline messages.

I am waiting for Hetzner representative here from WHT to update us, with something useful, and not like on their status page ...
Posted by EvolutionCrazy, 11-17-2013, 10:26 AM
http://bitcoinity.org/markets/mtgox/USD

still unreachable...

Was the attack targeting them? Bitcoin during the last few days has finally started to wake up
Posted by gingir, 11-17-2013, 10:27 AM
Nothing changed at my end... every server I have with Hetzner is still unreachable.
Posted by Hosting55, 11-17-2013, 10:28 AM
Hi

is anybody have outages at DC20?
One of my server in DC20 unreachable since 60 minutes.

other servers in DC10, DC12, DC14 are all up and works well.
Posted by gingir, 11-17-2013, 10:31 AM
What is RZ20?
I thought all their datacenters were called DC19, DC20, DC21 and so on.
Posted by Neso, 11-17-2013, 10:32 AM

Quote:

Originally Posted by gingir

What is RZ20?
I thought all their datacenters were called DC19, DC20, DC21 and so on.

I think it is same as DC, DC 19 is only that have problems, I have servers in other DC and they look OK for now.
Posted by NetworkPanda, 11-17-2013, 10:34 AM

Quote:

Originally Posted by gingir

What is RZ20?
I thought all their datacenters were called DC19, DC20, DC21 and so on.

RZ, DC are the same thing.

We are in DC20 as well, our servers are all online after a downtime of 18 minutes, so it seems that your turn will come soon and you will be back online as well.
Posted by Hosting55, 11-17-2013, 10:34 AM

Quote:

Originally Posted by gingir

What is RZ20?
I thought all their datacenters were called DC19, DC20, DC21 and so on.

RZ is for RechenZentrum, datacenter in german.
Posted by gingir, 11-17-2013, 10:34 AM
I am on 19 and 20, both not reachable.
Posted by Criot, 11-17-2013, 10:40 AM
Looks like they're probably going to have to implement some protection into their network.
Posted by wndml, 11-17-2013, 10:40 AM
Yes, we see some packet loss on our dedis.
Posted by nero16, 11-17-2013, 10:41 AM
DC19 huge packet loss!!!!!!
Posted by gingir, 11-17-2013, 10:44 AM
19 and 20 still not reachable.

I am starting to browse other companies' prices on their websites.
Any recommendations?
Posted by WebHostDog, 11-17-2013, 10:46 AM
They are saying on their website that facility 19 is under attack.
Posted by EvolutionCrazy, 11-17-2013, 10:46 AM
vps in rz19 online here
Posted by Neso, 11-17-2013, 10:50 AM
I know they are low cost provider, but to allow this problem to impact complete DC, and for 2 days ... I really do not know what to tell ...

They ware last good low cost provider until this, it look like they do not have any protection at all for their DC ...

Almost 2 hours all servers in DC are only reachable from some parts of world, no any useful information, nothing ...
Posted by digitallog, 11-17-2013, 10:58 AM
I have done some ping and tracert test to hetzener server and their network is up.Not sure which portion of their clients are affected.Also i noticed fdcservers chicago are facing same problem like them.
Posted by gingir, 11-17-2013, 10:59 AM
DC19 is still unreachable
Posted by nero16, 11-17-2013, 11:00 AM
what you on about... its completely dead now!
Posted by gingir, 11-17-2013, 11:02 AM
Which other services are you guys using? I need an alternative to this... mmm...
Posted by EvolutionCrazy, 11-17-2013, 11:04 AM

Quote:

Originally Posted by gingir

Which other services are you guys using? I need an alternative to this... mmm...

look for networks with either ddos protections or way higher bandwidth capacity...

networks like hetzner that does nullroute you when going like beyond 100k pps are doomed to fail in these situations as that's a clear sign they cannot hold
Posted by gingir, 11-17-2013, 11:08 AM

Quote:

Originally Posted by EvolutionCrazy

look for networks with either ddos protections or way higher bandwidth capacity...

Can you recommend anything besides OVH?
I can pay a 20-25% more what I am paying Hetzner if it can spare me this.

DC19 is still unreachable, my servers are down.
Posted by nero16, 11-17-2013, 11:09 AM
http://www.hetzner-status.de/en.html
They working hard! atleast they trying their best and good update info!!!

Appreciate their continues notice update!
Posted by digitallog, 11-17-2013, 11:11 AM
They are trying their best to solve the problem and they are providing updates on their website.They are budget provider what more you guys can expect from them?Just be patient and lets see how it goes.
Posted by EvolutionCrazy, 11-17-2013, 11:15 AM

Quote:

Originally Posted by gingir

Can you recommend anything besides OVH?
I can pay a 20-25% more what I am paying Hetzner if it can spare me this.

DC19 is still unreachable, my servers are down.

You can take into consideration online.net as an alternative.

Webtropia has similar prices but even more issues on the network IMHO
Posted by Neso, 11-17-2013, 11:19 AM
There are no better low budget provider from Hetzner, if you have money then just move to higher end providers, I did this for about 50% of my clients, and after this I will slowly move rest of them ...

Also this problem started almost 48 hours, and they said it was resolved, you can see on their status page ...

Also I can not accept that, even if they are low end provider, that their complete DC go down, if they can not protect individual servers from DDOS, they could at lest invest money to protect DC ...
Posted by gingir, 11-17-2013, 11:26 AM

Quote:

Originally Posted by EvolutionCrazy

You can take into consideration online.net as an alternative.

Webtropia has similar prices but even more issues on the network IMHO

Thanks EvolutionCrazy, I will take a look.

Our Hetzner servers are still unreachable since 01:30 CEST. Packet loss began around 11:00.
Posted by Syslint, 11-17-2013, 11:30 AM
DC20 seems fully down .
Posted by gingir, 11-17-2013, 11:37 AM
DC19 is down too since 01:30 CEST.
Posted by RSNET-John, 11-17-2013, 11:39 AM
I'm surprised they are manually null routing the IPs. With someone of their size I would expect them to have an automated attack detection system.
Posted by gingir, 11-17-2013, 12:04 PM
DC19 is still down since 01:30 CEST... come on Hetzner...
Posted by Neso, 11-17-2013, 12:24 PM
Same here, still offline for most part of worlds all servers in DC 19, I really do not know what to tell anymore to clients in US ...

I am really sorry now that I did not moved clients faster.
Posted by gingir, 11-17-2013, 12:27 PM
I feel so guilty too.
Where are you moving to Neso? I might be following...

All my Hetzner servers are down since 1:30.
Posted by Neso, 11-17-2013, 12:32 PM

Quote:

Originally Posted by gingir

I feel so guilty too.
Where are you moving to Neso? I might be following...

All my Hetzner servers are down since 1:30.

Atjeu, they are not cheap but they have one of most stable networks and fastest support, already moved most of my US clients, but now will move most likely all that I can.

There are no other option, Hetzner was best from all low budget providers, but after this there is no point anymore to gamble, other providers like OVH etc. are much worse.
Posted by wndml, 11-17-2013, 12:42 PM
I dont know if this is a coincidence, but I noticed increased ping times (2x - 3x) to a server I got at Leaseweb Frankfurt DC too and that started at the same time Hetzner problems did.

This looks like something rather huge.
Posted by softshop011, 11-17-2013, 12:43 PM
We are not expiriencing any packetloss to/from Europe, however i'm seeing a 40% packetloss to some USA locations! Could this be due to icmp throttling/blocking or are they still under an attack ?!
Posted by Neso, 11-17-2013, 12:46 PM

Quote:

Originally Posted by softshop011

We are not expiriencing any packetloss to/from Europe, however i'm seeing a 40% packetloss to some USA locations! Could this be due to icmp throttling/blocking or are they still under an attack ?!

Most likely, in last replay over ticket they told me that they are filtering still traffic and that they are under heavy DDOS attack.

Large portion of US is blocked.
Posted by softshop011, 11-17-2013, 12:57 PM
How convinient for OVH and their re-launch at soyourstart.com, hetzner attacked and 'downed' just 2 days before their launch... I'm sure we'll be seeing their ddos protection packages on all configs.
Posted by softshop011, 11-17-2013, 01:14 PM

Quote:

Originally Posted by softshop011

soyourstart.com

ahh the url is soyoustart.com, chosen by the OVH owner Oles himself. What a dumb domain name.
Posted by gingir, 11-17-2013, 01:16 PM

Quote:

Originally Posted by softshop011

How convinient for OVH and their re-launch at soyourstart.com, hetzner attacked and 'downed' just 2 days before their launch... I'm sure we'll be seeing their ddos protection packages on all configs.

Where can I read more about this?

My servers at Hetzner's are still down since 01.30 CEST...
Packet loss started at 11.00...
Posted by Neso, 11-17-2013, 01:16 PM
OVH have one of worst support, they replay to simple questions in about 2-7 days, I do not even wish to think how they would handle this, and also after their debacle over selling etc. nobody normal would use them even for backup server.
Posted by digitallog, 11-17-2013, 01:23 PM

Quote:

Originally Posted by Neso

OVH have one of worst support, they replay to simple questions in about 2-7 days, I do not even wish to think how they would handle this, and also after their debacle over selling etc. nobody normal would use them even for backup server.

Maybe your are using their France facility.But if you see reviews of there ovh Canada facility.They are providing good support and have many happy clients.
Posted by strex, 11-17-2013, 01:28 PM

Quote:

Originally Posted by Neso

OVH have one of worst support, they replay to simple questions in about 2-7 days, I do not even wish to think how they would handle this, and also after their debacle over selling etc. nobody normal would use them even for backup server.

I have 17 servers there and never had any problems with them.
Their support is also very fast, my last broken hard drive was replaced in less than 20 minutes.
Posted by malinka*, 11-17-2013, 01:30 PM
Hi

My EX6S server with Hetzner went down 18.00 located at DC 20.

Hope they fix it soon.

They have DDoS attacks on several of their subnets.

Regards
Malinka
Posted by gingir, 11-17-2013, 01:34 PM
Hetzner, CPR yourself please! All servers unreachable since 13.00 CEST!

Quote:

Originally Posted by strex

I have 17 servers there and never had any problems with them.
Their support is also very fast, my last broken hard drive was replaced in less than 20 minutes.

Where are you in Canada or in France?
Posted by Neso, 11-17-2013, 01:38 PM
OVH is maybe good in Canada, but France is really bad ...

Hetzner is really funny now: http://www.hetzner-status.de/en.html, they say; "Currently we're successfully filtering the malicious traffic" yes, when you block half US, Europe and rest of world ... still most servers in DC 19 have large packet loss...
Posted by softshop011, 11-17-2013, 01:40 PM
Hetzner status page :
November 17, 2013 5:45:00 PM CET
Currently we're successfully filtering the malicious traffic, the attack is still running at about 60 Gbit/s

Thats a huge attack. Not sure any host could mitigate it easely. I wonder if its a botnet or a dns reflection attack...
Posted by NetworkPanda, 11-17-2013, 01:43 PM
It seems that they are currently filtering all incoming and outgoing UDP traffic on the affected servers/datacenters. This basically means that you don't have DNS working on affected servers, while everything else (HTTP, SSH, FTP etc.) should be working.

So, if your nameserver is the same server where your sites are hosted, you might need to setup temporarily another nameserver for them.
Posted by peterodua, 11-17-2013, 01:45 PM

Quote:

Originally Posted by softshop011

Hetzner status page :
November 17, 2013 5:45:00 PM CET
Currently we're successfully filtering the malicious traffic, the attack is still running at about 60 Gbit/s

Thats a huge attack. Not sure any host could mitigate it easely. I wonder if its a botnet or a dns reflection attack...

ddos-guard says that they can mitigate up to 100 gbit/s
Posted by gingir, 11-17-2013, 01:48 PM
Are they doing it manually or what?
Posted by Neso, 11-17-2013, 01:51 PM

Quote:

Originally Posted by gingir

Are they doing it manually or what?

Most likely :p

Yesterday they fixed this after 24 hours (attack stopped), and that was most likely only some sort of test attack ...

I fear that this will not be resolved soon, most likely in about 12 hours if we have luck ...
Posted by peterodua, 11-17-2013, 01:52 PM

Quote:

Originally Posted by gingir

Are they doing it manually or what?

I don't know. I had an attack of 30 gbit/s and they handled it.

What is the best alternatives to hetzner? I can pay more to have more stable server. Mine is still unreachable in DC19
Posted by malinka*, 11-17-2013, 01:53 PM
Yes now my server is up.

I am so glad.

Regards
malinka
Posted by Neso, 11-17-2013, 01:56 PM
Finally, servers are UP for most locations, maybe we have little luck
Posted by wartungsfenster, 11-17-2013, 01:58 PM

Quote:

Originally Posted by gingir

Where can I read more about this?

My servers at Hetzner's are still down since 01.30 CEST...
Packet loss started at 11.00...

http://www.hetzner-status.de/en.html#2367

60gbps DDoS, quite fair to allow them a some time to sort it
Posted by gingir, 11-17-2013, 01:59 PM
Better move my DNS servers away from this s*** now that I have the chance.
Posted by NetworkPanda, 11-17-2013, 02:02 PM
Except for 15 minutes earlier today, our servers were online all this time and we had only UDP traffic blocked (not an issue for us, since we use a DNS cluster across various countries). Now it seems that UDP is unblocked by Hetzner and working again so gradually everything is returning to normal.
Posted by Criot, 11-17-2013, 02:04 PM

Quote:

Originally Posted by gingir

Better move my DNS servers away from this s*** now that I have the chance.

Any host having a 60gbps DDoS would have exactly the same issues, even if their network had protection. If their network had protection it probably wouldn't be down so long yes, but at the end of the day, there isn't many providers who'd handle 60gbps attack, unless all of their customers are paying them hundreds a month.
Posted by gingir, 11-17-2013, 02:04 PM

Quote:

Originally Posted by NetworkPanda

Except for 15 minutes earlier today, our servers were online all this time and we had only UDP traffic blocked (not an issue for us, since HTTP was working). Now it seems that UDP is unblocked and working again so gradually everything is returning to normal.

Thankyou, this is what I am going to do, never again DNS on here.
Posted by softshop011, 11-17-2013, 02:07 PM

Quote:

Originally Posted by peterodua

ddos-guard says that they can mitigate up to 100 gbit/s

ddos-guard is a garage host, i doubt they could handle a 1gb/s attack
Posted by NetworkPanda, 11-17-2013, 02:08 PM

Quote:

Originally Posted by gingir

Thankyou, this is what I am going to do, never again DNS on here.

Any datacenter provider facing a 60 Gbps UDP DDoS attack would have similar or worse problems, so moving your DNS to another provider does not always mean you are safe and that this will never again occur to you.
Posted by Rack_server, 11-17-2013, 02:11 PM
All UDP packets are filtered and voip / voice server owner are just pulling their hair.
Posted by gingir, 11-17-2013, 02:13 PM

Quote:

Originally Posted by NetworkPanda

Any datacenter provider facing a 60 Gbps UDP DDoS attack would have similar or worse problems, so moving your DNS to another provider does not always mean you are safe and that this will never again occur to you.

I know but if it has real DDoS mitigation at least it won't last this long.
Any suggestion is welcome.
Posted by IRCCo Jeff, 11-17-2013, 02:14 PM

Quote:

Originally Posted by Criot

Any host having a 60gbps DDoS would have exactly the same issues, even if their network had protection. If their network had protection it probably wouldn't be down so long yes, but at the end of the day, there isn't many providers who'd handle 60gbps attack, unless all of their customers are paying them hundreds a month.

.. or thousands.
Posted by Rack_server, 11-17-2013, 02:28 PM
Seems problem really solved at last. Getting ping from lookingglass of level3 from all sites.
Posted by peterodua, 11-17-2013, 02:44 PM
Mine is working too. At last.
Posted by softshop011, 11-17-2013, 02:52 PM

Quote:

Originally Posted by Rack_server

All UDP packets are filtered and voip / voice server owner are just pulling their hair.

Are you still experiencing throttling issues with UDP traffic ? I just ran iptraf between two hetzner servers in the same DC, UDP traffic seems fine, 200mbit/s - 400mbit/s
Posted by Hosting55, 11-17-2013, 02:54 PM

Quote:

Originally Posted by Rack_server

Seems problem really solved at last. Getting ping from lookingglass of level3 from all sites.

Not for me, one of my server is in DC20 and still down.
Posted by gingir, 11-17-2013, 09:07 PM

Quote:

Originally Posted by Hosting55

Not for me, one of my server is in DC20 and still down.

Any news? Can you reach it now?
Posted by JoJo MoJo, 11-18-2013, 03:43 AM
you'd expect a hoster of this size to take some measures against that happening... Truth is ddos protection does cost a lot, but then they should at least have some way of limiting the collateral damage it inflicts on the shared infrastructure.
Posted by Hosting55, 11-18-2013, 04:37 AM

Quote:

Originally Posted by gingir

Any news? Can you reach it now?

yes it is working now
Posted by alexnuke, 11-18-2013, 09:38 AM
Still having packet loss and server goes unreachable randomly.
Posted by gingir, 11-18-2013, 09:59 AM

Quote:

Originally Posted by alexnuke

Still having packet loss and server goes unreachable randomly.

Which DC are you in Alex?
Posted by alexnuke, 11-18-2013, 10:17 AM
Quote:

Originally Posted by gingir

Which DC are you in Alex?

I am at DC 19. Approximate 5% avg packet loss to 50% packet loss randomly.
Code:
 Host                                                                                                            Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. vl236-c4-19-c6-1.pnj1.choopa.net                                                                              0.0%   102    0.6   0.6   0.3   0.8   0.1
 2. vl166-br1.pnj1.choopa.net                                                                                     0.0%   102    1.5   2.8   0.3  11.1   2.7
 3. r1.nyiix.init7.net                                                                                            0.0%   102    8.8   7.2   1.2  12.4   3.7
 4. r1lon1.core.init7.net                                                                                         0.0%   102   70.5  71.3  68.7  79.5   2.5
 5. r1nue1.core.init7.net                                                                                         0.0%   102   83.5  83.4  83.1  84.3   0.1
 6. gw-hetzner.init7.net                                                                                          0.0%   102   82.3  83.9  81.9 126.7   7.3
 7. core12.hetzner.de                                                                                             5.0%   101   83.4  83.7  83.2  91.7   1.1
 8. core21.hetzner.de                                                                                             0.0%   101   86.0  86.0  85.8  86.3   0.1
 9. juniper3.rz19.hetzner.de                                                                                      0.0%   101   84.8  86.1  84.6 117.3   4.9
10. hos-tr1.ex3k24.rz19.hetzner.de                                                                                0.0%   101   88.1  86.9  85.5  91.0   1.4
11. static.35.122.76.144.clients.your-server.de                                                                   0.0%   101   84.8  85.5  84.6  88.9   0.9
Posted by wndml, 11-18-2013, 11:33 AM

Quote:

Originally Posted by alexnuke

I am at DC 19. Approximate 5% avg packet loss to 50% packet loss randomly.

The 5% you see is ICMP throttling, Hetzner does that at their routers. The faster the packets, the greater the throttling. Try reducing packet rate with mtr -i switch.

Real packet loss is seen from a point onward. You have all 0% after core12.hetzner.de