Hudson Valley Host has been down for 30 hours!

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Hudson Valley Host has been down for 30 hours!

Posted by emills01, 07-03-2013, 10:07 AM
Hudson Valley Host has been a pretty good host and I wrote a glowing review of them after they did an amazing job helping me migrate from my old host, but now they have been down for 30 hours after some botched server maintenance.

I know things go wrong some times but even with all of the bad hosts I've had - I've NEVER gone 30 hours (plus) downtime!

I'm concerned with the lack of communication as well. The email informing us of server maintenance said that it would begin at 3AM. I actually started sometime just after midnight. Once things went bad, they sent out an email saying it would take about 20 hours to fix, and it has been 30. After about 24 hours I tried to get some information via support chat and got nowhere.

This is frustrating to say the least, and very, very worrisome at best. A full day plus with no ad income, search engines getting server not found errors (what will that do to page rankings and search engine placement?!)...I'm losing money and audience.

And there's no end in sight!
Posted by Ethernet Servers, 07-03-2013, 10:13 AM
Hello,

The downtime in question is an unfortunate event and one that we are working on rectifying as quickly as possible.

We are hoping that all websites will be back online today as we have changed our approach in dealing with the situation.

In regards to communication, multiple emails have been sent out and when we have information, we're letting customers know. We cannot update clients with information we do not have.

Please PM me with your domain name and I will issue credit immediately, as will we will doing for all other customers.

I apologize for the inconvenience caused, and appreciate your patience.

Thank you.
Posted by CW Mike, 07-03-2013, 10:20 AM
Glad George has come in and helped you OP, but this can happen to the best of us. Something goes wrong.. you fix that then something else brakes, you fix that something else kicks a fuss etc. I'm not sure what's happened here but if I was you I'd work with George to sort the issue out.
Posted by stablehost, 07-03-2013, 10:21 AM
Every host is going to have a "massive" failure atleast once in their career, we're all human, except my secretary, she scares me sometimes...

Anyways, what's important when a provider has a failure, is how well they communicate it out to clients, actively respond to issues still and how they take steps to prevent it in the future.

Look at Rackspace for example, one of the best providers in the industry and they had a massive cloud failure a few weeks ago. It doesn't matter who you are, it's all about how you handle it and move forward with it.

Best of luck!
Posted by emills01, 07-03-2013, 10:30 AM
To be clear - this is not an angry rant telling people to stay away from HVH. I'm just a concerned customer looking for info.

Quote:

Originally Posted by HVH - George

Hello,

The downtime in question is an unfortunate event and one that we are working on rectifying as quickly as possible.

We are hoping that all websites will be back online today as we have changed our approach in dealing with the situation.

In regards to communication, multiple emails have been sent out and when we have information, we're letting customers know. We cannot update clients with information we do not have.

Please PM me with your domain name and I will issue credit immediately, as will we will doing for all other customers.

I apologize for the inconvenience caused, and appreciate your patience.

Thank you.

Thanks for responding so quickly.

For what it's worth, I've only received 2 emails - one notifying me that the downtime was about to take place, and another notifying me that something was wrong (the one saying it would be 20 hours).

I'm much, much more concerned about getting updates when things go past their expected wait times than getting a refund. You guys don't charge a lot - I'm not worried about that.

What I am worried about is that obviously something went very wrong, and now I don't know what's going on with my data. The email said no data was lost, but obviously things have changed for the worse.

I understand things go wrong, but when it wasn't fixed in the 20 hours it said it was going to take I would have hoped for another email telling me so and why. The "20 hours" email was very detailed and had the info I was looking for. But since then I haven't seen anything, and when I tried to reach out in support chat I was stonewalled. The guy wasn't rude, but it was like chatting with a bot.
Posted by Ethernet Servers, 07-03-2013, 10:36 AM
Hello,

We sent out 3 emails in total, perhaps one failed to reach you:

07/01/2013 17:45 Scheduled Maintenance - 7/2/13 @ 02:00AM EST
07/02/2013 03:47 Service Downtime Explanation
07/03/2013 04:33 Update on Server Downtime

As you can see from the above log, an email has been sent out every day since the issue first started occurring.

Since we perform daily backups, you may loose a few hours worth of data, but nothing more than that.

Thank you.
Posted by CD Burnt, 07-03-2013, 10:53 AM
moved to the outages section.

from now, posts by affected customers and providers only.
Posted by emills01, 07-03-2013, 12:57 PM
George - PLEASE post updates on the Network status page. The reason I'm not getting the emails is because with the server down they aren't being delivered (even though MX records on my domain point to an external email handler). The 4:33 update would be great.
Posted by Ethernet Servers, 07-03-2013, 01:00 PM
Will get that done ASAP.

Lots of websites are now back online. All websites should be back online within the next 10 hours. We have lots of accounts to restore in total as you can imagine
Posted by emills01, 07-03-2013, 10:17 PM
Unfortunately it's been 9 hours since that last update - none of my sites are up yet (I run 2 different shared accounts) and the Network Status page still has nothing in it.

No change, no updates, no email access - nothing. It's going on about 46 hours now.

I'm *extremely* disappointed right now, to say the least.

EDIT: I just saw the update on the Network Issues page, which appears to be different from the Network Status page for some reason. I guess that's better than nothing, but I'm still completely down. Even if I take a refund it will do nothing compared to the revenue I've lost and I don't even want to think about clients that are unable to reach me.
Posted by HVH-Ernie, 07-03-2013, 10:24 PM
The process is not very quick unfortunately. We have restored several hundred accounts. I am doing yours manually now. Ill PM you an update in a couple minutes.
Posted by emills01, 07-03-2013, 10:29 PM

Quote:

Originally Posted by HVH-Ernie

The process is not very quick unfortunately. We have restored several hundred accounts. I am doing yours manually now. Ill PM you an update in a couple minutes.

I don't want to give the impression that I expect special treatment - I don't. I appreciate that you guys are in a tough position trying to fix this.

At the time I made that last post I didn't know that there were separate pages for network status updates and I guess I was looking at the wrong one (I was using the one linked at the bottom of the main HVH site).

Thanks again for the update.
Posted by HVH-Ernie, 07-03-2013, 10:37 PM
Your account has been restored. Sorry for the delay.
Posted by Blyze, 07-04-2013, 04:16 AM
Going on 49hrs downtime on my account there.

If it isn't RamNode getting hacked, its my DNS provider nuking their filesystem. Down to <90.00% uptime over 30 days, my kingdom for a stable host T_T
Posted by nhmac, 07-04-2013, 06:23 AM
for me and my clients. I have a ticket open but all I keep hearing is that sites will be restored "soon."
Posted by Ethernet Servers, 07-04-2013, 06:49 AM
We are actively restoring all accounts one at a time. I can fetch backups if you wish to move to another hosting provider.
Posted by nhmac, 07-04-2013, 07:04 AM
since yesterday morning, with no result so far. In my ticket the response to my request was "the backups are incremental and not tared. We are actively restoring the backups."

That was yesterday.

If you can actually get them for me that would be great.
Posted by HVH-Ernie, 07-04-2013, 07:27 AM
Im working on restoring your manually now to speed up the process.
Posted by HVH-Ernie, 07-04-2013, 07:37 AM
nhmac the domains you outlined in your ticket have been restored. Ive also issued a 1 month credit to your account.

We are continuing to move the backups from the offsite servers and restoring them.
Posted by nhmac, 07-04-2013, 08:30 PM
the other 2/3 of the domains I am hosting have not yet been restored. I asked for one more, which the owner had sent out press releases about just before the sites went down, he's feeling rather desperate. No luck on that or the other less desperate ones yet.

I am sure I am not the only one having this kind of problem.
Posted by HVH-Ernie, 07-04-2013, 08:53 PM

Quote:

Originally Posted by nhmac

the other 2/3 of the domains I am hosting have not yet been restored. I asked for one more, which the owner had sent out press releases about just before the sites went down, he's feeling rather desperate. No luck on that or the other less desperate ones yet.

I am sure I am not the only one having this kind of problem.

Can you please PM me a list of usernames you need restored. Ill do them manually.
Posted by nhmac, 07-04-2013, 10:34 PM
Thank you, Ernie!
Posted by HVH-Ernie, 07-06-2013, 09:13 PM
Just an update all accounts on server1 have been restored. Server2 will be done by tomorrow. These are the only 2 effected.