Corespace out all day?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Corespace out all day?

Posted by holyearth, 07-14-2012, 07:14 PM
Corespace down most of today...sucks I just signed up for their services too...
Posted by CoreSpace, 07-14-2012, 07:31 PM
I apologize you are facing issues, we are mitigating a very large DOS attack that has been affecting a portion of our customers over the last few hours. We are working to solve the issue as soon as possible, I apologize again for the issue.
Posted by holyearth, 07-14-2012, 08:09 PM
My server has been down for at least 6 hours...can't you guys isolate/null-route the client being targeted?
Posted by CoreSpace, 07-14-2012, 08:20 PM
It appears that it grew very rapidly and is affecting nearly all of our network, It has been escalated to our main providers and is being worked at the highest levels at Level3's security segment as well as our Corero rep is onsite helping to mitigate the attack as well. I do apologize for the issues this is causing, it's the largest attack we have seen. Please bare with us while we work though the problem.
Posted by DeltaAnime, 07-14-2012, 08:37 PM
Why not just nullroute the attacked IP?

It isn't very hard.

Francisco
Posted by CoreSpace, 07-14-2012, 08:50 PM
It's not attacking a single IP, the DOS is hitting all 5 /16's. It's been escalated to all the highest levels across all of our providers. It seems to be hitting at a rate greater then 2 million packets per second.
Posted by CoreSpace, 07-14-2012, 10:38 PM
Just found out through our other industry friends... we are not the only one fighting a large DOS today. There are a few other DC's with the same issue.
Posted by nazran, 07-14-2012, 11:12 PM
CORESPACE as one of your COLO clients may I suggest that you have a phone line and or internet page outside of your facility that doesn't go down when you go down? It is bat-s**t-crazy that you have no way for your customers to reach you in a scenario like this.
Posted by CoreSpace, 07-14-2012, 11:44 PM
I agree, and to be honest our fail-over NOC phone is normally always in the DC but as of 3pm the on staff colo employee left with it on accident in a rush to the ER. We retrieved the NOC phone and can receive all calls again, and gladly he is doing fine. I can only apologize for the issues you are facing and we are rigorously working to fix the issue. We will be transparent and forthcoming with all we are working on if you have any additional question please ask.
Posted by ricalien, 07-14-2012, 11:48 PM
Any predict on when service will resume?
Posted by holyearth, 07-14-2012, 11:59 PM
Service is still down...9+ hours...the corespace website seems to be back up so maybe this will be over soon?
Posted by CoreSpace, 07-15-2012, 12:02 AM
I don't have an answer for you right now, all I can say we are doing everything we can to overcome the issue currently.
Posted by CoreSpace, 07-15-2012, 12:05 AM
they have already cut the DOS in half down to about a million packets per second .... so the light is growing. This is one of the largest DDOS's I've seen in my time in the hosting industry.
Posted by georgia_tech_swagger, 07-15-2012, 12:10 AM
Yes please @ a network status page outside of the DC.

I've been calling all day to just a busy signal ... but this came right up when googling "Corespace down".

It was an adventure just finding the number to call -- I've been in that datacenter since it was CI Host at another physical location.... and then CI Host at the current address .... and then Constellate .... and now CoreSpace.

I was toying with the idea of Amazon EC2 on demand failover .... looks like I need to be more serious about it.
Posted by georgia_tech_swagger, 07-15-2012, 12:33 AM
BTW ... for other people with servers in there ... everything that is not port 80 has been working (albeit with heavy packet loss) for the last several hours.
Posted by CoreSpace, 07-15-2012, 12:41 AM
I really appreciate the info and the review you gave us. I already passed that port 80 info on, thanks all the extra help is always appreciated.
Posted by CoreSpace, 07-15-2012, 12:47 AM
Thanks again Jeremy!
Posted by georgia_tech_swagger, 07-15-2012, 01:40 AM
Port 80 seems to be limping along now ... but 22 (SSH) is hopeless.
Posted by georgia_tech_swagger, 07-15-2012, 01:50 AM
Spoke too soon ... but 80 was working (slowly) there for a good 5 mins.
Posted by chuyskywalker, 07-15-2012, 02:07 AM
My server seems to be responding...okish. Certainly not back to 100% yet, but at least serving pages and requests again.
Posted by BadWillHunting, 07-15-2012, 02:13 AM

Quote:

Originally Posted by georgia_tech_swagger

BTW ... for other people with servers in there ... everything that is not port 80 has been working (albeit with heavy packet loss) for the last several hours.

Sure, GTS, but it's pretty-much unusable anyway... I will gladly rep the fact that we're under a DOS attack, I know them well.
Posted by BadWillHunting, 07-15-2012, 02:16 AM
...about the Penn State stuff.
Posted by georgia_tech_swagger, 07-15-2012, 02:17 AM

Quote:

Originally Posted by BadWillHunting

...about the Penn State stuff.

Hahahahahhaha
Posted by chuyskywalker, 07-15-2012, 02:23 AM
On the bright side, we're all getting a 50% discount on this months bill since this outage well exceeded the 5 hours of packet loss.

htt p://corespace.com/pdf/sla.pdf <-- Sorry, can't make real links yet.
Posted by BadWillHunting, 07-15-2012, 02:34 AM
explanation? I'm with you guys, nothing by my support, no matter what may ever come.

Quote:

Originally Posted by georgia_tech_swagger

Hahahahahhaha

Posted by georgia_tech_swagger, 07-15-2012, 02:42 AM
The Penn State reference. That has been most of the realignment talk the last week.
Posted by CoreSpace, 07-15-2012, 09:15 AM
The attack is not ranges from 1gbps to 10gbps, this is much larger than a simple DDOS it's as full scale as it gets. Per Corero/Level3 this is the second largest they have seen.
Posted by CoreSpace, 07-15-2012, 09:20 AM
CI host/Constellate is a separate company with different owners and board members than CoreSpace has. This is the first issue of this magnitude we have faced like this. I'm sorry you had previous issues with CI host and they way they handled their issues.
Posted by idextrus, 07-15-2012, 09:43 AM
I have been with CI Host since 2003 and I was really hoping that with the Corespace amalgamation, previous lack of communication issues would be a thing of the past. I just started my vacation yesterday and have had to deal with this issue ever since. They NEED to have an offsite location/web site to post information for serious issues such as this. I am beyond angry and will have to likely spend the remainder of my holiday appeasing all of my affected clients!
Posted by chuyskywalker, 07-15-2012, 10:10 AM
Getting pretty close to near 24 hours of network issues -- I'm a patient person, since CIHost, Constellate, and now Corespace have been very good to me, but this is getting a tad bit ridiculous.
Posted by holyearth, 07-15-2012, 10:35 AM
Still down today :-(

Losing my patience, I just prepaid for 6 months of service!
Posted by MP Admin, 07-15-2012, 01:09 PM

Quote:

Originally Posted by CoreSpace

CI host/Constellate is a separate company with different owners and board members than CoreSpace has. This is the first issue of this magnitude we have faced like this. I'm sorry you had previous issues with CI host and they way they handled their issues.

Like others here, our company was with C I Host, which changed to Constellate, which changed to Corespace.

With C I Host we had constant problems. With Constellate, operations for us at least were relatively stable. With Corespace, this has pretty much been the first issue with them ... with you. Unfortunately, as you've operated yesterday, and throughout today, it's as if Corespace is back to being as unreliable as C I Host was.

But none of this really matters. What matters is how you deal with the massive problems that are occurring as I post this message. Please reply to the following questions. Thank you.

1. When will your clients be able to call support? Your phone number has been constantly busy since Saturday, July 14th.

Suggestion: At least have your number forwarded to a message stating your current status, and what you're doing about it.

2. As best as you can determine, when can you realistically expect to be fully operational?

Suggestion: Since you evidently use Corespace to to "Stop DDoS Attacks", you use them to "Protect Infrastructure", and you use them to "eliminate downtime", work with them to temporarily take over your operations by assisting you with contact information until you're able to manage business on your own again.

Ironically, I couldn't register in this forum using my regular email address. That's because Corespace has shuttered the email addresses associated with our affected domains.

In any case, please do your best to at least get your phone number working again, and to keep your clients updated about what's going on.

The way I found you here is because you're showing up with a search of "Corespace down" in Google. That's unfortunate. Thankfully, however, at least Corespace is being represented here for updates. I applaud you for this, and it gives me hope that you'll be fully operational in a timely fashion.

On behalf of our own company, I selfishly hope for the best since what affects you is currently affecting us. Good luck.

Edited to correct a typo.
Posted by MP Admin, 07-15-2012, 01:41 PM

Quote:

Originally Posted by MP Admin

2.Since you evidently use Corespace to to "Stop DDoS Attacks" ...

Sorry, folks. Evidently, I can only edit for corrections one time with each post. As such, please make note of a mistake I made in what I posted (see my own quote above).

"Since you evidently use Corespace" should have been "Since you evidently use Corero" - sorry about that.
Posted by CoreSpace, 07-15-2012, 02:05 PM
Current update: We are adding in DDOS services from Black Lotus to help eliminate the DDOS attack, I believe we are looking to have the issue finalized hopefully in the next few hours. Thanks again for all of our customers affected by the DOS attack for bearing with us.
Posted by MP Admin, 07-15-2012, 02:26 PM

Quote:

Originally Posted by CoreSpace

Current update: We are adding in DDOS services from Black Lotus to help eliminate the DDOS attack, I believe we are looking to have the issue finalized hopefully in the next few hours. Thanks again for all of our customers affected by the DOS attack for bearing with us.

Good communications is a very big part of customer service, so thank you very much for your update. It's very much appreciated.
Posted by idextrus, 07-15-2012, 02:34 PM
Yes, please keep the updates coming. I am mandated to update all of my clients who have been affected by this, so anything that I can pass along is greatly appreciated.
Posted by Waylon2000, 07-15-2012, 03:27 PM
I've been a client of CI Host/Constellate/CoreSpace for almost ten years (started with CI Host).

Overall the company has been fairly reliable but has dropped the ball badly on me several times. The abandonment of the small-hosting side of the their business and the shotgun 'migration' of those sites to 'X7 Hosting' was a complete disaster. I had several clients whose sites were basically destroyed by the ham-handed conversion. (X7 may well be the most inept, technically-challenged company I have ever dealt with, bar none. Staffed by idiots, run by morons.)

We've had better luck with their dedicated servers, but there have been a few times that's been problematic too.

Not having an offsite number for people to call or an offsite status page is just crazy negligent in my opinion.

This latest DDOS coupled with the somewhat ineffective response is not sitting well with me. Both my servers have been off the air nearly 100% since yesterday.

Is HTTP there? Nope.
Can I use FTP? Nope.
Can I login via SSH? Nope.

Can any of my clients get to the 50+ sites I have at Corespace? Nope.

Soooooo....it's been down 2 days now. When will Corespace be back in operation??
Posted by MP Admin, 07-15-2012, 04:11 PM

Quote:

Originally Posted by Waylon2000

I've been a client of CI Host/Constellate/CoreSpace for almost ten years (started with CI Host).

That's pretty much the same with our company as well.

Interestingly, and this is my own thing to deal with once Corespace becomes operational again, it seems that I'm paying double for what I'm getting with my server. But I expect this will be corrected once I point this out to their accounting department. They'll either need to upgrade the server we're leasing with what's supposed to be provided, or they'll have to lower their price to what we're actually supposed to be paying. If not, then moving to another host will be the thing to do.

In any case, I guess another thing to do is to maintain our own redundant operations by mirroring what we have at Corespace with another web host. Thankfully, as I've looked around, prices have come down with evidently very reliable host providers to such a point that we can now duplicate our server at the same cost that we're paying with Corespace.

In other words, what we were once getting with a $99 dedicated server we can get for $49. So any one of us can have our own redundancy for what we're already paying each month.

Logistically, I haven't figured out these details yet, but now more than even for those of us with Corespace, it seems that we should have our own redundancy in place. Clearly, there's no redundancy there.

Thankfully, there are a good number of web hosts that we can rely upon, and any one of us can easily point our domains to other domain name servers as needed or desired.

Time will tell whether or not Corespace is going to be able to resolve their problems. I hope for their sake, and ours, it will be sooner than later. After all, to be perfectly frank, I'm not in the frame of mind to be moving over to another server; at least not at this time.
Posted by Texaschwag, 07-15-2012, 04:16 PM
This is redic!

We have servers that must be live by morning! We will surely be moving to Colo4 after this experience. I feel that all Datacenters will go down sooner or later, but communication is the key. No email or a way to contact...

The phones are just now up and I have been rotting in the queue for thirty minutes.... actually the line just went dead again.
Posted by writespeak, 07-15-2012, 04:43 PM
A reminder from the description for this forum:

Quote:

This forum is provided to discuss current outage issues and to allow a way for customers and providers to communicate. Comments by non-customers will be removed.

Thank you for your cooperation.

Lois
Posted by BadWillHunting, 07-15-2012, 04:47 PM
So, seriously, who pissed-of some muslim-extremist-group, on their hosted-forums?

I have asked USAF IT to monitor this situation, it's getting a tad over-the-top.
Posted by MP Admin, 07-15-2012, 04:58 PM

Quote:

Originally Posted by writespeak

A reminder from the description for this forum:

Thank you for your cooperation.

Lois

Have any non-customers posted here? I just presumed that anyone trying to get help from Corespace are clients like our company.

Can you please clarify why you felt compelled to post here? Thanks.

By the way, what is a non-customer? I'm confused.

***************************************************

My edited comments: I'm guessing some inappropriate posts were made in this topic that were made by people who aren't actual clients of Corespace, and those posts were removed.

And anyone who isn't a client of Corespace is a non-customer? If so, I believe I understand.

By the way, thank goodness for this forum. Without it, I'm wondering if Corespace would have been as diligent as they seem to be in getting past whatever is really going on with their company.

It helps a lot to be able to communicate here when no other way to do this seems to be available; at least not presently. But thankfully with their phone service back up, clients can be apprised about what's going on. After all, how many frustrated clients would have known to come here to get answers?
Posted by Texaschwag, 07-15-2012, 05:15 PM

Quote:

Originally Posted by MP Admin

Have any non-customers posted here? I just presumed that anyone trying to get help from Corespace are clients like our company.

Can you please clarify why you felt compelled to post here? Thanks.

By the way, what is a non-customer? I'm confused.

***************************************************

My edited comments: I'm guessing some inappropriate posts were made in this topic that were made by people who aren't actual clients of Corespace, and those posts were removed.

And anyone who isn't a client of Corespace is a non-customer? If so, I believe I understand.

By the way, thank goodness for this forum. Without it, I'm wondering if Corespace would have been as diligent as they seem to be in getting past whatever is really going on with their company.

It helps a lot to be able to communicate here when no other way to do this seems to be available; at least not presently. But thankfully with their phone service back up, clients can be apprised about what's going on. After all, how many frustrated clients would have known to come here to get answers?

This is similar to what I said prior to my post being removed. Makes you wonder if webhostingtalk.com is in bed with Corespace??
Posted by Waylon2000, 07-15-2012, 05:17 PM

Quote:

Originally Posted by MP Admin

They'll either need to upgrade the server we're leasing with what's supposed to be provided, or they'll have to lower their price to what we're actually supposed to be paying. If not, then moving to another host will be the thing to do.

The only other dedicated server host I'm familiar with is Quadranet. I have one box there and so far, with the exception of some initial config issues) they've been pretty solid. YMMV.

Quote:

Originally Posted by MP Admin

In other words, what we were once getting with a $99 dedicated server we can get for $49. So any one of us can have our own redundancy for what we're already paying each month.

I was thinking more or less the same thing. Keep a server (maybe 2) on hot standby with nightly synchronization. DNS points to the live server and if (when) something like this happens, just bop the DNS to the standby box. They just have to be at completely different companies and at physically different data centers.

Quote:

Originally Posted by MP Admin

Time will tell whether or not Corespace is going to be able to resolve their problems. I hope for their sake, and ours, it will be sooner than later. After all, to be perfectly frank, I'm not in the frame of mind to be moving over to another server; at least not at this time.

Yeah, Id rather not move unless I really have to. Setting up a redundant box could be done with not too much work, recreate all the domain entries, then do transfer all the files via rsync or whatever you like (I'm not certain what Windows uses).
Posted by Waylon2000, 07-15-2012, 05:24 PM

Quote:

Originally Posted by writespeak

A reminder from the description for this forum:

Lois, I don't understand...the forum description I see says, "Web Hosting Talk : Web Hosting Main Forums : Industry Announcements : Providers and Network Outages and Updates"

I'd think this would be the place to discuss an outage by an ISP. Am I wrong, or...?
Posted by MP Admin, 07-15-2012, 05:47 PM

Quote:

Originally Posted by Texaschwag

This is similar to what I said prior to my post being removed. Makes you wonder if webhostingtalk.com is in bed with Corespace??

Well, it makes perfect sense for this forum to cater to those who pay them. But that's not necessarily a bad thing. As long as this forum isn't complicit in censoring our comments or deleting us as members simply to cater to their paid clients, then everything should be fine. Let's hope the moderation done here is above board. Time will tell.

************************************************************

Quote:

Originally Posted by Waylon2000

Yeah, Id rather not move unless I really have to.

Again, I feel the same way. We had this kind of outage numerous times with C I Host before they became Constellate, and now of course Corespace. However, I was lucky enough during those outages to have had a direct contact at their Los Angeles datacenter to help me personally (before they shuttered it). I might add that this number was given to me by a very understanding partner of C I Host who was obviously very supportive. So for me personally, the numerous outages that affected pretty much most of their clients didn't usually affect me so adversely.

I'm hoping that Corespace will make it through this mess sooner than later, and that this won't be the new normal for them. Time will tell.

Ironically, I'm looking just as bad to our clients as Corespace is looking to us, so in truth I'm just as bad as Corespace is as it's presently operating.

That irony is making me wonder how responsible I'm being to our clients. Thankfully so far, they've been understanding, and supportive.

I can't wait until all of this is behind us, and I suspect Corespace is feeling the same way. After all, just as we're relying on them to get things up to speed again, they're relying on others to make it happen.

It's six degrees of separation I guess.
Posted by writespeak, 07-15-2012, 06:01 PM

Quote:

Originally Posted by MP Admin

My edited comments: I'm guessing some inappropriate posts were made in this topic that were made by people who aren't actual clients of Corespace, and those posts were removed.

Right except I wouldn't necessarily call the posts inappropriate, just not right for this forum. We aim to keep this forum for providers to provide updates about outages and for clients to communicate with their providers during outages. Anything else gets in the way, and we have other forums for discussion.

Quote:

And anyone who isn't a client of Corespace is a non-customer? If so, I believe I understand.

Yes, that's what I meant.

Quote:

It helps a lot to be able to communicate here when no other way to do this seems to be available; at least not presently.

That's what this forum is here for.

Quote:

Originally Posted by Texaschwag

This is similar to what I said prior to my post being removed. Makes you wonder if webhostingtalk.com is in bed with Corespace??

Uh, nope. I did some thread cleanup because off-topic posts were reported in this thread. We do the same for any thread in the Outages forum, no matter who the provider is.

I don't pay attention to who advertises on WHT, so I have no idea whether the provider in question pays iNET. It isn't relevant when we do moderating.

Quote:

Originally Posted by Waylon2000

Lois, I don't understand...the forum description I see says, "Web Hosting Talk : Web Hosting Main Forums : Industry Announcements : Providers and Network Outages and Updates"

I'd think this would be the place to discuss an outage by an ISP. Am I wrong, or...?

Yup. We're trying to keep this thread to the topic of the outage.

If you have any other questions or concerns about the forum rules, please open a helpdesk ticket. Let's get this thread back to its original topic. To keep this thread useful for its purpose, posts about other topics will be removed. Thanks!
Posted by jcalhoun, 07-15-2012, 06:10 PM
Entire network seems to be down 100% now, no packets getting through. Hopefully they have taken everything offline in order to setup the Black Lotus services. Fingers crossed.
Posted by jcalhoun, 07-15-2012, 06:33 PM
All ports appear to be operating 100% normal now. Hope it lasts.
Posted by MP Admin, 07-15-2012, 06:35 PM

Quote:

Originally Posted by jcalhoun

Entire network seems to be down 100% now, no packets getting through. Hopefully they have taken everything offline in order to setup the Black Lotus services. Fingers crossed.

This is what I'm hoping has happened, so my fingers are being crossed.

Quote:

Originally Posted by writespeak

Yup. We're trying to keep this thread to the topic of the outage.

If you have any other questions or concerns about the forum rules, please open a - url redacted by this forum's notice - ticket. Let's get this thread back to its original topic. To keep this thread useful for its purpose, posts about other topics will be removed. Thanks!

Thanks for the clarification. I just hope you take into account that forum topics can take a turn when chatting it up. It's just human nature. Over-moderation can be a real killer to the conversation itself.

Just to remind you, this topic is about the continuing outage at Corespace that will soon be three straight days. So of course we'll continue to chat about it, and we'll also offer up solutions as needed because that's the appropriate thing to do.

But don't worry because some of your other paid clients may benefit from it.

By the way, and this should be a given that's common sense, you'd be completely wrong if you think for one moment that any of us having problems with Corespace would want to take the focus off of getting Corespace to fix these problems. In fact, if you see someone trolling on Corespace's behalf to get off the topic of their continuing outage, then please, please, please moderate any of those posts so that we can continue discussing this outage, and our solutions for it.

Additionally, if in fact it turns out that Corespace is unable to fix its problems, of course it will be appropriate for us clients to discuss other solutions. So at the very least, a clear link here in this topic addressing other options will not only be appropriate, but it would then be needed for those of us who want to keep our domains online.

Hopefully things won't go that far in this topic, because presumably, Corespace will become whole again, and we can remain its clients.
Posted by MP Admin, 07-15-2012, 06:40 PM

Quote:

Originally Posted by jcalhoun

All ports appear to be operating 100% normal now. Hope it lasts.

It's been up and down; up and down. But I applaud Corespace for doing whatever it takes to fix this outage once and for all.

This is one of the things that I liked about Constellate that has now become Corespace; that they always seem to make a really sincere effort to fix things.

Now if they can just work on their communication skills, that would be great. Acknowledgement for their participation here has been made, and again, I'm very grateful for it.

Good luck!
Posted by Waylon2000, 07-15-2012, 06:54 PM

Quote:

Originally Posted by MP Admin

It's been up and down; [I]up and down[/I

Yeah, but it's been up for almost 10 whole minutes now! lol
Posted by IRCCo Jeff, 07-15-2012, 07:05 PM

Quote:

Originally Posted by Waylon2000

Yeah, but it's been up for almost 10 whole minutes now! lol

I do not speak on behalf of CoreSpace, but I believe this fix is expected to be permanent.
Posted by holyearth, 07-15-2012, 07:11 PM
I'm still completely down!!
Posted by MP Admin, 07-15-2012, 07:27 PM

Quote:

Originally Posted by IRCCo Jeff

I do not speak on behalf of CoreSpace, but I believe this fix is expected to be permanent.

This would be great news. I surely hope this is correct. If so, hopefully Corespace will use what's happened as a learning experience to take the steps they need to be more secure. I also hope they'll step it up with their communication skills. What they didn't need in this forum was all the chatting that we felt compelled to do because there was no other place to go to for contact with Corespace.

And for me personally, I need to use this as a learning experience myself so that I won't be so reliant on a server hosting company. Their lenghtly downtime caught me off guard, and I wasn't as ready as I should have been.

I'm still not sure what I should be doing ... maybe follow my own advice regarding redundancy ... but I'll hopefully figure it all out so that I don't let my clients down as I did by simply waiting for things to improve.

I'm just grateful that Corespace got it together. For a while there I thought they had suddenly gone out of business. I've seen it happen before.

****************************************************************

Quote:

Originally Posted by holyearth

I'm still completely down!!

I hope you get back online soon; very soon! As of this post, our server is once again accessible, and each of our domains are operational.

Keeping my fingers crossed!
Posted by CoreSpace, 07-15-2012, 08:01 PM
Thank you Black Lotus ddos mitigation, we should be back to normal or near normal network connectivity. Thank you again and we will address all additional issues to avoid any any future frustrations, we also asked the zombie bot net not to attack us again Again all services should be back to normal, and thank you for your patience.
Posted by Waylon2000, 07-15-2012, 08:05 PM

Quote:

Originally Posted by MP Admin

And for me personally, I need to use this as a learning experience myself so that I won't be so reliant on a server hosting company. Their lenghtly downtime caught me off guard, and I wasn't as ready as I should have been.

Same here, I'm sorry to say...I think I'll have to bite the bullet and look into 1 or 2 cheap servers to act as hot-standby boxes. The fact that this happened over a weekend a bit of a saving grace for me as most of my traffic happens during the week...but this could have happened anytime.

Quote:

Originally Posted by MP Admin

I'm still not sure what I should be doing ... maybe follow my own advice regarding redundancy ... but I'll hopefully figure it all out so that I don't let my clients down as I did by simply waiting for things to improve.

The thing is that if all of our sites are on a single ISP and the ISP is hit, there's no solution, nothing we can do. And since it's all the same network co-location (within the same ISP) really wouldn't do much (if any) good.

I think the only way to work around this kind of thing is to have frequently-mirrored standby boxes ready to flick over and go live.
Posted by MP Admin, 07-15-2012, 08:39 PM

Quote:

Originally Posted by Waylon2000

The thing is that if all of our sites are on a single ISP and the ISP is hit, there's no solution, nothing we can do. And since it's all the same network co-location (within the same ISP) really wouldn't do much (if any) good.

I think the only way to work around this kind of thing is to have frequently-mirrored standby boxes ready to flick over and go live.

Right. Regarding mirrored servers, that was kind of my thinking as well, but of course with entirely different providers. For the way that we operate, that would mean updating our sites twice every time we upload updates; once for our chosen and active server, and once again for our backup server.

The only time I've ever done this was when we've switched servers, i.e., meaning that we kept our domains fully updated on both the old and the new servers until propagation had been completed.

But to be frank, especially since it's not particularly hard to activate a new server, I may continue to only keep local copies of our domains for "a quick switch" as needed. My reasoning for this is because I'm still of the mindset that Corespace is a good company. So whenever problems do arise, especially after what's happened now, my thinking is that they'll do whatever it takes to get us back online.

Obviously, my mindset about this will change depending on whether or not Corespace continues to have problems. My personal feelings are that they got caught off guard, and relied on a company that let them down. I know. It's not a great excuse, but it seems that's what happened. In other words, they thought they had a system in place to mitigate, and stop DDoS attacks, but when it came right down to it, that system failed.

I need to remind myself, too, that the real failing with any given web host is not the problems that come up, unless they're excessive, but it's in how they deal with those problems.

Time will tell how well Corespace takes care of its clients. I'm thinking the glass is half full.
Posted by Waylon2000, 07-15-2012, 09:20 PM

Quote:

Originally Posted by MP Admin

Right. Regarding mirrored servers, that was kind of my thinking as well, but of course with entirely different providers. For the way that we operate, that would mean updating our sites twice every time we upload updates; once for our chosen and active server, and once again for our backup server.

Here's how I'd figured on doing it, probably automated through some bash and php scripts:

1) Do the nightly backup of the server database.

2) Send a copy of the backup to the standby server and import it into the standby server db. Now both databases are in sync.

3) Grab any recently changed files, send them to the standby server and overwrite the current ones. I'd check this every night but for most of my sites changed files would be a fairly infrequent event. The db is what's constantly changing for the most part.

Should be easy to automate FTP to transfer the files. I may code up something tonight now that I can get to the servers.

Quote:

Originally Posted by MP Admin

Time will tell how well Corespace takes care of its clients. I'm thinking the glass is half full.

Yep. For the most part they've done right by me, so I won't switch away unless this kind of thing keeps happening.
Posted by georgia_tech_swagger, 07-16-2012, 08:55 AM
Things are still pretty jank.

Trying to scp a file to my server gets filter hammered by the time it hits 150 KB.

Trying to fetch any file in my linux package manager is hopeless.

Any php page where a user is adding content to my site is hopeless and times out.

I can't even wget any packages!

If your site is up right now ... with an unpatched vulnerability ... you're a sitting duck.

Posted by Waylon2000, 07-16-2012, 11:08 AM

Quote:

Originally Posted by georgia_tech_swagger

If your site is up right now ... with an unpatched vulnerability ... you're a sitting duck.

Is there a specific vulnerability that you're referring to?

Maybe I missed something, I was under the impression that this was a DDOS against the ISP and not related to any specific vulnerability. Could you please explain what you're referring to?
Posted by georgia_tech_swagger, 07-16-2012, 11:11 AM

Quote:

Originally Posted by Waylon2000

Is there a specific vulnerability that you're referring to?

Maybe I missed something, I was under the impression that this was a DDOS against the ISP and not related to any specific vulnerability. Could you please explain what you're referring to?

Right now all the sites are back up. But the filtering going on is severe. I can't scp anything to my server. I can't wget anything from my server. So ... if you're running something right now with a vulnerability ... you can't download the patch to it. So your option is shut it down ... manually patch files if that is available... or roll the dice.
Posted by Waylon2000, 07-16-2012, 11:15 AM

Quote:

Originally Posted by georgia_tech_swagger

if you're running something right now with a vulnerability ... you can't download the patch to it. So your option is shut it down ... manually patch files if that is available... or roll the dice.

Ahhh, okay, I understand.

FTP seems to be slow for me (can't use SFTP unfortunately) but otherwise things seem okay.

I'd like to hear some details about this attack from corespace, e.g. what domain or domains were targeted, etc.
Posted by IRCCo Jeff, 07-16-2012, 11:20 AM

Quote:

Originally Posted by georgia_tech_swagger

Right now all the sites are back up. But the filtering going on is severe. I can't scp anything to my server. I can't wget anything from my server. So ... if you're running something right now with a vulnerability ... you can't download the patch to it. So your option is shut it down ... manually patch files if that is available... or roll the dice.

Your IP space will only go into Black Lotus' diversion if directly under attack. All other traffic will pass uninhibited. You'll need to talk to CoreSpace about your specific situation, as they may have other security procedures in place not directly related to our DDoS mitigation.
Posted by georgia_tech_swagger, 07-16-2012, 11:20 AM

Quote:

Originally Posted by Waylon2000

Ahhh, okay, I understand.

FTP seems to be slow for me (can't use SFTP unfortunately) but otherwise things seem okay.

I'd like to hear some details about this attack from corespace, e.g. what domain or domains were targeted, etc.

They said previously in this thread their entire IP blocks were targeted.

Quote:

Originally Posted by IRCCo Jeff

Your IP space will only go into Black Lotus' diversion if directly under attack. All other traffic will pass uninhibited. You'll need to talk to CoreSpace about your specific situation, as they may have other security procedures in place not directly related to our DDoS mitigation.

See above. Due to the nature of the attack -- everybody is going through Black Lotus right now.
Posted by IRCCo Jeff, 07-16-2012, 11:31 AM

Quote:

Originally Posted by georgia_tech_swagger

They said previously in this thread their entire IP blocks were targeted.

See above. Due to the nature of the attack -- everybody is going through Black Lotus right now.

I am in management at Black Lotus and I can see everything that is under filter or not. Note that only traffic actively under attack is subject to our filtering. Please contact CoreSpace with your specific issue and see if they can help investigate the cause.
Posted by georgia_tech_swagger, 07-16-2012, 12:06 PM

Quote:

Originally Posted by IRCCo Jeff

I am in management at Black Lotus and I can see everything that is under filter or not. Note that only traffic actively under attack is subject to our filtering. Please contact CoreSpace with your specific issue and see if they can help investigate the cause.

Please send me an email just so I can see if we're in the trouble block.

georgia (dot) tech (dot) swagger (at) gmail

And CoreSpace is aware of my issues specifically -- the issues in general -- and has been for some time.
Posted by Waylon2000, 07-16-2012, 12:17 PM

Quote:

Originally Posted by IRCCo Jeff

I am in management at Black Lotus and I can see everything that is under filter or not. Note that only traffic actively under attack is subject to our filtering. Please contact CoreSpace with your specific issue and see if they can help investigate the cause.

1) Is the attack still in progress?

2) On my end http seems okay but FTP is nearly unusable...does this indicate that my IP range is probably being filtered?
Posted by georgia_tech_swagger, 07-16-2012, 12:21 PM
1) I'd have to guess yes
2) Yes ... you're having similar issues to me.

I don't know what kind of filtering they're doing ... but my best guess is packet size? I ssh tunneled directly to the server I'm trying to use ... literally using it as localhost. That worked fine ... but all the things that didn't work outside of the tunnel (posting) didn't work inside of it either.

So even highly encrypted tunneled traffic running outside of port 80 gets filter killed when larger packets start flowing. As I said earlier ... scp gets the hammer dropped quickly on it.
Posted by Waylon2000, 07-16-2012, 12:29 PM
You know, if they ever find the little *$^# who's responsible for this attack, I'd be perfectly okay with a 'mitigation team' visiting him and returning with his head in a Coleman cooler.
Posted by MP Admin, 07-16-2012, 12:51 PM

Quote:

Originally Posted by IRCCo Jeff

Your IP space will only go into Black Lotus' diversion if directly under attack. All other traffic will pass uninhibited. You'll need to talk to CoreSpace about your specific situation, as they may have other security procedures in place not directly related to our DDoS mitigation.

I didn't realize you were in management at Black Lotus. So for that I thank you since your company sure did come to the rescue. I'm very grateful for that, and I'm also grateful that Corespace utilized your resources.

For the record, our server seems to be fully accessible now, and in good working order.

The only bad thing that seems to have happened is that it appears we haven't received some email that was reportedly sent to us during the downtime. I did notice the late delivery of email, so maybe the reported missing emails will come to us at some point.

Our FTP access seems to be fine for each of the domains that I tested, and our WinSCP access seems to be working as it should.

For others as Jeff mentioned elsewhere, "You'll need to talk to CoreSpace about your specific situation, as they may have other security procedures in place not directly related to our DDoS mitigation."

I can tell you from previous experiences that other security measures have been applied to our server on occasion, which slowed our FTP access, but those measures were removed because they had been inadvertently applied. In other words, many times there are specific server issues versus issues with an entire network; well, sometimes anyway.

Not to make light of any problems that anyone is still having, but I am so glad not to have had to continually switch server hosts as I used to do in the earlier days of the Internet, circa 1996-2001. In general, server hosts are much more reliable these days. Thank goodness!
Posted by jcalhoun, 07-16-2012, 12:58 PM
Things are still not 100% for me either. I call a lot of web services over port 80 from my servers and almost all of the requests are timing out. Same for web browsing from my servers. Seems like outgoing port 80 for me is useless right now. Sent an email to the CoreSpace helpdesk but not sure I should expect much of a response.
Posted by georgia_tech_swagger, 07-16-2012, 01:05 PM
AHA! I just spoke to the CoreSpace Sr. Network Administrator ... he explained how the tunnel from Black Lotus added a 32 bit header ... so if you are using something that pushes a large amount of traffic and sets the no fragment bit on ... you hit the router deadlock of you have to fragment to pass it on but you set no fragment so it just drops the packet.

Gentlemen ... change your MTU to 1300 if, like me, your default was 1500.
Posted by time299, 07-16-2012, 02:35 PM
Black Lotus DDos mitigation, Thank you for helping corespace with this attack.
For me issues still continue today. I have had many clients call this morning each from different states saying that they can't get there sites or email or both to load. I see from my location that the same sites and email are working fine from my ISP, but not for them. They are getting "page can not be displayed" and email timeouts.

My questions are as follows.

1) Is the DDos over?

2) When will "Black Lotus" be removed and normal access restored.

3) Moving forward what is corespace doing for this type of issue in the future.

4) Do DDos attacks normally last 3 days?

I don't know what to tell my clients at this point.
Posted by CoreSpace, 07-16-2012, 04:12 PM
The Ddos is not over but our traffic is being filtered by Black Lotus, we have centralized the problem and the network should be back to normal any time now, we are just working out the last few kinks the issue was causing. Yes Ddos can last a day to weeks at a time, I believe the longest recorded was 80 days. All services should return to normal asap, we should have a final statement for all of the questions and postmortem of the event.

Thank you again
Posted by FISMgr, 07-16-2012, 04:24 PM
I saw several comment regards talking to CoreSpace but I have yet to find a valid phone line. I even tried the cell for the on site person at the Dallas facility that we use to exit the building, and got no response. Our Account rep responded to an email on the first day but have been otherwise unable to reach anyone. Our Servers are co-located and fortunately we have our DNS hosted elsewhere and I have pointed all of our client traffic to another off site backup server, but we are in the process of a quarterly data update and product build, and are severly happered by all of this. I realize it is out of your hands as to cause, but I would like to talk to someone to see if our servers are among those targeted and if there is anything we need to do from our end. How the heck do I reach anyone?
Posted by CoreSpace, 07-16-2012, 04:46 PM
All the phone systems and regular service should have returned to normal now as well as all of our numbers are working as well as the NOC phone, they may have been on the other line for the NOC phone when you called.