Network issues with ATJEU?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Network issues with ATJEU?

Posted by Masud, 07-02-2012, 05:36 AM
Dear friends @ WHT,

I am with ATJEU for 10+ years now and never faced regular network issues with them but the recent uptime report and several client complaint suggests that they are facing some regular network issues these days.

Even right now 2 of my servers are going through same issue.

Looks like they are aware of it but somehow they are unable to explain the problem.

Anyone else with ATJEU facing this?

Regards,
Masud
Posted by Ankit, 07-02-2012, 05:51 AM
I have a VPS with a provider that hosts at ATJEU that is also down right now. Hopefully this gets fixed soon.
Posted by Masud, 07-02-2012, 06:24 AM
Yeah, looks like they are under DDoS, its time they sort this out for good!
Posted by Ville:, 07-02-2012, 03:47 PM
My vps with a provider their is still down.
Posted by pdqso, 07-02-2012, 04:58 PM
An attack against a provider is going on. Here is what the company emailed:

Quote:

On July 2nd, the datacenter we're hosted in, Atjeu Hosting, in Phoenix AZ, unplugged our network drop due to a single inbound DDoS attack on one of our clients that, despite at 700mbit, completely crippled their network and sent them into a panic.

As of this writing, we're up to 8 hours of downtime, which is something completely unacceptable to me personally, and to the goals of our company, so we're contacting other datacenters to look for a new home.

Posted by atjeu, 07-02-2012, 11:58 PM
We cancelled that client because they have had 3 large ddos attacks in less than 2 months. They sell $2 VPS machines mostly to chinese clients and they are a ddos target. This client was warned and given chances so it wasn't like this was the first offense. We owe it to the rest of our clients to keep our network clean.

Large ddos attacks have been on the rise big time this year with all data centers. Part of protecting against them is to not have clients who are targets. We are also doing massive network upgrades and are upgrading all switches and even our edge router to be able to better handle these kinds of things.
Posted by Masud, 07-03-2012, 01:43 AM
I hope that solves the problems!
Posted by ZKuJoe, 07-03-2012, 02:03 AM

Quote:

Originally Posted by atjeu

We cancelled that client because they have had 3 large ddos attacks in less than 2 months. They sell $2 VPS machines mostly to chinese clients and they are a ddos target. This client was warned and given chances so it wasn't like this was the first offense. We owe it to the rest of our clients to keep our network clean.

Large ddos attacks have been on the rise big time this year with all data centers. Part of protecting against them is to not have clients who are targets. We are also doing massive network upgrades and are upgrading all switches and even our edge router to be able to better handle these kinds of things.

1) Why are you releasing information about your client's activities and your client's customers on a public forum?
2) Why not nullroute the target IP instead of unplugging all of the machines (this can even be automated)?
Posted by SeriesN, 07-03-2012, 02:09 AM
700mbit brought your system down to its knee? And you call your self a DC?
Posted by layer0, 07-03-2012, 02:21 AM
Seems that the 700mbps figure came from the client. It's possible that the actual attack was even larger.
Posted by SeriesN, 07-03-2012, 02:26 AM

Quote:

Originally Posted by layer0

Seems that the 700mbps figure came from the client. It's possible that the actual attack was even larger.

Per ATjeu's reply "That" was the only client with issue. And Per Clients contact "They were Notified" about 700 mbit?

Am I missing any link here?

And An Easy question, Why not Null route the ip? It is not a DC's concern to whom the clients sell his service or how much he i charging them while being under the law and DC TOS. And how can a DC know if most of those Users are Chinese? Hmm suspicious.
Posted by DeltaAnime, 07-03-2012, 02:31 AM

Quote:

Originally Posted by SeriesN

Per ATjeu's reply "That" was the only client with issue. And Per Clients contact "They were Notified" about 700 mbit?

Am I missing any link here?

And An Easy question, Why not Null route the ip? It is not a DC's concern to whom the clients sell his service or how much he i charging them while being under the law and DC TOS. And how can a DC know if most of those Users are Chinese? Hmm suspicious.

They probably have some sort of SFLOW setup that tells them what ASN's/etc they talk to the most.

Francisco
Posted by Steven, 07-03-2012, 04:04 AM

Quote:

Originally Posted by ZKuJoe

1) Why are you releasing information about your client's activities and your client's customers on a public forum?
2) Why not nullroute the target IP instead of unplugging all of the machines (this can even be automated)?

It's actually not an uncommon practice to pull down whole server.s I can think of 3 other providers off the top of my head that do that
Posted by Masud, 07-03-2012, 04:41 AM
I have been a happy client of ATJEU for years now but last night's incident effected 2 of my most important clients and it has kinda raised a few questions.

Jeff from ATJEU is in touch with me and I hope he/they can come up with a solution which convinces me and my clients to stay aboard.
Posted by ZKuJoe, 07-03-2012, 09:16 AM

Quote:

Originally Posted by Steven

It's actually not an uncommon practice to pull down whole server.s I can think of 3 other providers off the top of my head that do that

That sounds pretty unreasonable to me. I'm a small provider and I would never lets a DDOS attack against one client impact another. I wouldn't even let my DC be impacted from an attack against one client. I would hope any sized DC would have the same, if not better systems in place than I do.
Posted by atjeu, 07-03-2012, 10:58 AM

Quote:

Originally Posted by SeriesN

700mbit brought your system down to its knee? And you call your self a DC?

This was an incoming attack that totally maxed out our incoming gig circuit. That means that the attack was much LARGER than that. It has difficult to say when the attack is so large. In that case, there are two options, null route(which is what we did) or use a cloud mitigation service like Prolexic, Verisign or Black Lotus. Most clients do not want to pay for mitigation services however as it can literally be 10k to 30k for a one time mitigation.

@ZKuJoe
1) Why are you releasing information about your client's activities and your client's customers on a public forum?
2) Why not nullroute the target IP instead of unplugging all of the machines (this can even be automated)?

I hate to get into details but because of the amount of misinformation that has released on this event, I feel that it is necessary to clear the air. Their forum posts make it sound like they are an innocent reseller who is very careful and was blind sighted but us shutting them down. This is simply untrue.

We did null route but because this is the third time this has happened inside of two months and because their business model is such that we expect ddos problems to be a regular occurrence with them, we felt we had to make the difficult decision to shut them down in the interest of the rest of our clients.

@SeriesN
we knew what they were doing because we have talked to them at length about their business model. They are a colo client so we know them personally.

Everyone needs to realize that a large scale DDOS attack is not something that is always a simple thing to troubleshoot. Depending on the nature and size of the attack, it can be illusive and often automatic mitigation of large attacks produces false positives which wrongly block legitimate traffic. When manually null routing, the greater network can still be effected for a few minutes in the best case scenario. In this incident, our network analysis tools were not picking up the target IP as they normally should and that made this take a lot longer to fix. When this only happens every once in a while it is not that big of a deal. When it happens on a regular basis, it is not acceptable. In the past, we might see one or two large ddos attacks in a 12 month period. In the last year, these numbers have sky rocketed. This is why we are making sweeping network upgrades across our whole facility. We are also looking at partnering with a third party mitigation company after our network upgrades are complete.