Limestone networks offline

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Limestone networks offline

Posted by stablehost, 11-24-2010, 09:16 PM
... once again, sigh.

Looks like 80% packet lost, perhaps another DDOS.
Posted by SC-Daniel, 11-24-2010, 09:17 PM
Yep, extremely high latency and major packet loss... again :-/
Posted by Karl, 11-24-2010, 09:18 PM
Same =\

10chars
Posted by Dexqt, 11-24-2010, 09:30 PM
Oddly enough, everything is working fine here except HTTP traffic.
Posted by Karl, 11-24-2010, 09:34 PM

Quote:

Originally Posted by Dexqt

Oddly enough, everything is working fine here except HTTP traffic.

Yeah I did a ping on watchmouse.com and there were no problems from Melbourne
Posted by VoidX, 11-24-2010, 09:37 PM
Seems to be down for me as well, waits...
Posted by Arsenico, 11-24-2010, 09:43 PM
Same here, 3 servers are being not accesible from anywhere. And we can't contact limestone's staff when they have this kind of problems.

I'm getting really mad with this situation.
Posted by VoidX, 11-24-2010, 09:44 PM

Quote:

Originally Posted by Arsenico

Same here, 3 servers are being not accesible from anywhere. And we can't contact limestone's staff when they have this kind of problems.

I'm getting really mad with this situation.

Meh, nothing we can do, I am just as frustrated as you.
Posted by Karl, 11-24-2010, 09:45 PM
I think we all are Arsenico.
Posted by jackpx, 11-24-2010, 09:46 PM
yes. two servers .... 80% packet lost
Posted by Arsenico, 11-24-2010, 09:47 PM
I think so man. A ddos attack is really bad and difficult, but this is happening at least once a week.
Dossed's customer should leave limestone, and find a ddos protected company.
Posted by Ithiel, 11-24-2010, 09:48 PM
I have 3 dedicated servers with them, and they're all down.

It's the middle of the day here, and I've got clients all over me screeming about 'why is my website down!'.

Limestone need to get this fixed NOW!
Posted by VoidX, 11-24-2010, 09:49 PM
The attack is probably directed @ the limestone data center.
Posted by Entrio, 11-24-2010, 09:51 PM
extremely slow and high packet loss. From australia here
Posted by Karl, 11-24-2010, 09:51 PM

Quote:

Originally Posted by VoidX

The attack is probably directed @ the limestone data center.

Probably an angry ex-customer Haha.
Posted by paulg1981, 11-24-2010, 09:51 PM
+1 for me too
Posted by Arsenico, 11-24-2010, 09:52 PM

Quote:

Originally Posted by VoidX

The attack is probably directed @ the limestone data center.

Do you think so? that is not what i would want. Neither limestone's CEO
When the attack is directed to the company itself, it's very difficult to take quick actions.
Well, gotta eat something, and pray. it's all i can do.
Posted by VoidX, 11-24-2010, 09:52 PM

Quote:

Originally Posted by Karl

Probably an angry ex-customer Haha.

Haha no, I'm hosted at LimeStone. Migrated here last week.
Posted by paulg1981, 11-24-2010, 09:53 PM
Is DDOS covered in the service uptime guarantee? Ironically I would look but their site won't load :-)
Posted by Karl, 11-24-2010, 09:53 PM

Quote:

Originally Posted by VoidX

Haha no, I'm hosted at LimeStone. Migrated here last week.

I meant the attack is probably from an angry ex-customer
Posted by VoidX, 11-24-2010, 09:54 PM

Quote:

Originally Posted by Karl

I meant the attack is probably from an angry ex-customer

Oh, misread
Maybe they are all working as a team
Posted by Karl, 11-24-2010, 09:56 PM

Quote:

Originally Posted by VoidX

Oh, misread
Maybe they are all working as a team

Haha, no problem. That would be pretty awesome
Posted by Entrio, 11-24-2010, 09:56 PM
it seems to be going up and down all the time. Sometime it will load right away other it wont load at all.
Most of the time it just slowpaces
Posted by VoidX, 11-24-2010, 09:57 PM
What will they compensate us for the down time?
50% discount for next month server bill?
Posted by Ithiel, 11-24-2010, 09:57 PM
Limestone are based in the DataBank (formerly the Federal Reserve Building - http://databank.com/), and they're just one of many companies that run out of that frankly colossal facility.

If the attack is directed at their data centre, then it means that there will be a whole heap of other hosting/server providers that are down at the moment. Since the outage category here on WHT doesn't seem to be overflowing with pissed off people from every other hosting company under the sun as well, I'd imagine the attack is limited to just limestone, rather than the data centre itself.
Posted by VoidX, 11-24-2010, 09:58 PM
A staff from LimeStone should give us some sort of update here asap.
Posted by paulg1981, 11-24-2010, 09:59 PM
Not a good day for a servce interruption with all the BF deals on the horizon either
Posted by Karl, 11-24-2010, 10:00 PM

Quote:

Originally Posted by VoidX

A staff from LimeStone should give us some sort of update here asap.

Last time this happened we heard nothing for a long time, until someone posted here, however, they didn't give an explanation as to what happened or why it happened. Basically just said there was a lot of latency.
Posted by SC-Daniel, 11-24-2010, 10:00 PM
Haha, after about 15 minutes of loading, the RW login page is starting to render.

Hopefully this is resolved soon :-/ Second outage of this nature in 2 weeks.
Posted by Dexqt, 11-24-2010, 10:01 PM

Quote:

Originally Posted by Arsenico

I think so man. A ddos attack is really bad and difficult, but this is happening at least once a week.
Dossed's customer should leave limestone, and find a ddos protected company.

Or invest in some serious Cisco mitigation equipment.
Posted by paulg1981, 11-24-2010, 10:08 PM
Just had to share my sentiments

http://www.youtube.com/watch?v=yo3uxqwTxk0

Good for a laugh while I wait
Posted by Bortman, 11-24-2010, 10:13 PM

Quote:

Originally Posted by Dexqt

Or invest in some serious Cisco mitigation equipment.

No kidding. We are considering migrating from LSN if they can't get their act together, and we have a lot of servers with them, so it would be a real pain. Having random downtimes of 1-2 hours every other week is not acceptable even if we were on totally cut-rate hosting.
Posted by Arsenico, 11-24-2010, 10:13 PM

Quote:

Originally Posted by Dexqt

Or invest in some serious Cisco mitigation equipment.

it depends on the size of the attack. They could have a cisco mitigation equipment, but they are down anyway.

In my country, we had a huge ddos attack. To a hosting company, the impact was:
International connection from almost all of the ISP ..down.
Datacenter, down.
Others datacenters.. down too.

lol. The attack was so big, that it has a huge impact. the hosting company stayed down for almost 2 weeks. And the problems for the international connections too. (it was coming and going.)

So, it depends on the size man.
Posted by VoidX, 11-24-2010, 10:20 PM
I want a update nao!
Posted by SC-Daniel, 11-24-2010, 10:21 PM
Seems back to normal, or at least stabilizing...
Posted by jackpx, 11-24-2010, 10:21 PM
One hour .. 70% packet lost
Posted by paulg1981, 11-24-2010, 10:21 PM
LSN is back up for me. Excellent
Posted by Arsenico, 11-24-2010, 10:22 PM
Yes it came back a few minuts ago. =) Glad to see this!

I'm still mad. but well, i can handle it. since i understand the impact of ddos attacks.
Posted by Dexqt, 11-24-2010, 10:23 PM

Quote:

Originally Posted by paulg1981

Just had to share my sentiments

http://www.youtube.com/watch?v=yo3uxqwTxk0

Good for a laugh while I wait

LOL, the best part is the video is screwed and all the comments refer to "FIX IT"
Posted by jackpx, 11-24-2010, 10:25 PM
yes... back online
Posted by Mike - Limestone, 11-24-2010, 10:26 PM
Everything is indeed back on-line. I'm expecting an update from our Network Operations Center shortly; their first priority is on stabilizing the network as soon as possible in events such as these.

Thank you, everyone, for your patience. SLA tickets are welcome.

-mike
Posted by Deroba, 11-24-2010, 10:30 PM
Mike, mind sharing us what was the cause or the problem of this downtime?
Posted by Karl, 11-24-2010, 10:30 PM

Quote:

Originally Posted by Mike - Limestone

Everything is indeed back on-line. I'm expecting an update from our Network Operations Center shortly; their first priority is on stabilizing the network as soon as possible in events such as these.

Thank you, everyone, for your patience. SLA tickets are welcome.

-mike

Could you update this thread later to let us all know what happened?

Regards,
Karl
Posted by VoidX, 11-24-2010, 10:31 PM

Quote:

Originally Posted by Karl

Could you update this thread later to let us all know what happened?

Regards,
Karl

+1, do it...
Posted by Jacob Wall, 11-24-2010, 10:34 PM

Quote:

Originally Posted by Ithiel

Limestone are based in the DataBank (formerly the Federal Reserve Building - http://databank.com/), and they're just one of many companies that run out of that frankly colossal facility.

If the attack is directed at their data centre, then it means that there will be a whole heap of other hosting/server providers that are down at the moment. Since the outage category here on WHT doesn't seem to be overflowing with pissed off people from every other hosting company under the sun as well, I'd imagine the attack is limited to just limestone, rather than the data centre itself.

Not true, they have their own blend of network carriers(Telia, Savvis, and Internap) and that's not the same links that Databank shares. Anyways, most companies within the Databank pull their own bandwidth in since it's a carrier neutral facility. The Databank "network" only does a few Gbps IIRC.
Posted by Mike - Limestone, 11-24-2010, 10:45 PM

Quote:

Originally Posted by Karl

Could you update this thread later to let us all know what happened?

Regards,
Karl

Hi Karl (and others who asked),

Yes, I am seeing about obtaining an official statement concerning the scope and impact of the network disruption. Thanks again for your patience.

-mike
Posted by Karl, 11-24-2010, 10:54 PM
Thank you Mike!
Posted by KrisFromLimestone, 11-26-2010, 10:15 PM
On 11/24/2010 some users experienced degraded network connectivity stemming from a network attack that took place around the time. The incident was roughly 70 minutes in duration lasting from 6:55pm to 8:05pm CST before it was fully mitigated by our network engineers.

This attack was related to a similar attack that we observed several weeks ago and unfortunately both attacks ceased before the effects they were causing at certain points in the network could be fully analyzed. However, we have gathered sufficient data during this incident to determine the area of the network being exploited and have put additional monitors in place to assist in gathering relevant data that can be used to more quickly mitigate such an attack if it occurs again.

We apologize for any inconvenience this has caused you and your clients and we remain committed to providing excellent service. If you are one of our valued clients and have any additional inquiries pertaining to this incident please feel free to open a ticket with our Support department. Our team would be happy to help!