HostDime UK offline for 10 hours + counting

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > HostDime UK offline for 10 hours + counting

Posted by xlguy, 03-20-2010, 06:18 AM
We first reported this about 8 hours ago. All of HostDime UK went offline, including their site www.hostdime.co.uk, while they did some maintenance in the UK datacenter.

However, the sites have now been offline for over 10 hours, which is a pretty long maintenance 'window'!

Talking to support just now and there is no ETA and they still haven't put anything on http://status.hostdime.com about what's going on. So they've taken all our sites offline for 10 hours, including during working UK hours and they can't tell us when we can expect them back online.

I thought a company as large as HostDime would have systems in place to ensure that a maintenance "window" couldn't suddenly become a "half day" - and possibly more. I can't believe after all this time they are still unable to give even a rough ETA of when we can expect normal service.

Very disappointed by this
Posted by net, 03-20-2010, 06:30 AM
That's a long downtime.

What kind of maintenance is it?
Posted by xlguy, 03-20-2010, 06:33 AM
They say they are moving servers around their UK DC to make more space. What's really frustrating is support right now are not able to give any further information and no ETA on everything coming back online, plus their status page on HostDime.com doesn't mention anything at all, so their communication to clients about this extended downtime is nill at present.
Posted by net, 03-20-2010, 06:37 AM
Arggg... moving servers is a huge headache.

It could even take up to 48+ hours :-( unless they are already prepared for this but things can happen you know.
Posted by xlguy, 03-20-2010, 06:40 AM
They are not changing the DNS or anything like that, just moving them around a DC as far as I know (there's no information available on their web site as mentioned). They planned this to happen during quiet hours in the UK but now it's the middle of the morning, so I guess something has gone wrong, they can't really take client sites offline for up to 48+ hours surely (we're at 10 so far)?!
Posted by eDedi, 03-20-2010, 08:17 AM
i thought it was the rain that's worth a good 2 days downtime
Posted by xlguy, 03-20-2010, 08:42 AM
Lol, thanks for trying to inject some British humour in to this

Just under 13 hours now and both our dedicated server and HostDime.co.uk are still offline. Still nothing on the official HostDime status page, I really hope the US team is going to investigate why a planned early hours move of servers has turned in to this (it's nearly 1pm UK time) and why the UK DC team can't give them an ETA.
Posted by DediShack, 03-20-2010, 09:20 AM
If they're moving to another facility they're probabally stuck in traffic/roadworks (sorry couldn't help myself!)

Have you tried contacting their tech dept. via phone? Pretty sure they could resend you the email outlining the maintenance window.
Posted by xlguy, 03-20-2010, 09:32 AM
Lol, all these jokes about good old England

Believe they are just moving within a DC rather than between. I've been on to their support this morning but they just keep apologising and saying that there is no specific ETA on us getting back online. The last guy I chatted to was helpful in telling me that something had gone wrong (nice to see some honesty) but what I need is the server back online and all I could get on that was "they're working on it". I guess there's going to be some angry emails between the US head office and the UK NOC team - at least there should be

Anyone know where / who runs the DC? I believe it's in Berkshire from Google
Posted by DediShack, 03-20-2010, 09:52 AM

Quote:

Originally Posted by xlguy

Anyone know where / who runs the DC? I believe it's in Berkshire from Google

Probabally Rapidswitch's DC or one of the Bluesquare facilities.
Posted by HostThree, 03-20-2010, 10:06 AM
It's Bluesquare they're at.
Posted by xlguy, 03-20-2010, 10:17 AM
I'm not sure how many HostDime UK customers are on WebHostingTalk, but for those who are interested it's still offline and the latest news from the tech support is that there was some kind of network issue after the move happened and still, unbelievably, no ETA - what's going on over there where a server move goes from a few hour window to 14 hours and no light at the end of the tunnel, I would have thought they could say "within 3 hours" or something similar.

What really bugs me is that you have to get through to technical support to get updates, over the last 14 hours there hasn't been a single note or update on their http://status.hostdime.com/ page, twitter account, or anything similar. Communication is key and if they published regular updates online it would free up their support team (who are friendly enough, but powerless as they're all in the US as far as I can tell).

Quote:

Originally Posted by HostThree

It's Bluesquare they're at.

Thanks, I was just curious
Posted by HostDime, 03-20-2010, 10:34 AM
We apologize for any prolonged delays, the migrations and move was completed hours ago, the main issue now relies with the bandwidth providers who seem to be updating their BGP routes. Some clients are still seeing down while others are not. So you understand what we really did was move servers within the facility and re route ips to our network gear and it is just a matter of the bandwidth providers updating routing tables which was done. However there has been issues and delays from that segement of things which prolonged things. Unfortunately it was a bit out of our control and we have been working on it none stop. We will be happy to issue you any credit for this prolonged downtime once it everything is 100% Ok. We have been in contact with the bandwidth ( upstream providers ) and everything should be routing properly very soon. The good news is this migration was done to help with expansion and it is a one time move and nothing like this should ever need to happen again.
Posted by xlguy, 03-20-2010, 11:38 AM
Thanks for the update, sites still offline. The www.hostdime.co.uk site itself also remains down so I'm guessing that is one of the sites involved in the current outage, as well as our dedicated server. Hopefully someone will be able to explain in layman's terms what the continued cause of this is? I read the post above a few times, but I'm not technically minded and have no idea about "BGP routes" and "segments". From a customer point of view (mine ) our sever has been offline for nearly 14 hours and with *no email updates*, *no status updates online* and *no ETA* (not even "6 hours max" etc.) on a fix, that's the bottom line... and one I hope HostDime will address. Waiting for clients to realise their servers have been down for over half a day and then telling them to open a ticket (the latest suggestion when I spoke to support) if they want status updates isn't an ideal way of doing things, IMO.
Posted by HostDime, 03-20-2010, 12:32 PM
What is your email please and phone #? I will call you please pm me
Posted by xlguy, 03-20-2010, 12:39 PM
I'll gladly drop you a PM, but it would be nice if you could address those issues in this thread too - I'm not the only HostDime UK dedicated customer and therefore I'm sure others would appreciate a bit of openness which you could demonstrate by tackling these criticisms here rather than on a phone. However, ultimately I return to this bottom line of the situation: "our sever has been offline for nearly 14(edit: 15 now) hours and with *no email updates*, *no status updates online* and *no ETA* (not even "6 hours max" etc.) on a fix". The only phone call I want right now is one telling me it's online again or exactly when it will be online.
Posted by HostDime, 03-20-2010, 12:43 PM
I explained above what the issue was, we are focusing on solving the issue right now honestly
Posted by xlguy, 03-20-2010, 12:49 PM
Fair enough, I just hope that you'll also agree to review the situation once everything is 100%. I wouldn't want this to happen again to other customers, the lack of communication (as I've pointed out several times) has been close to zero. Good luck getting it fixed and I hope to hear from someone at support soon to confirm we're back online.
Posted by erectvps, 03-20-2010, 12:53 PM

Quote:

Originally Posted by HostDime

I explained above what the issue was, we are focusing on solving the issue right now honestly

Would it be possible to drop certain providers just to get things working and bring them back online when BGP has updated? (Speaking as a person whos not that up to speed on this sorta tech)
Posted by HostDime, 03-20-2010, 01:43 PM
www.hostdime.co.uk can be loaded now right? seems things are stabilizing
Posted by HostDime, 03-20-2010, 01:44 PM
xlguy,

We just sent you an email via ticket please check it as soon as you can
Posted by Dustin Cisneros, 03-20-2010, 03:37 PM

Quote:

Originally Posted by HostDime

www.hostdime.co.uk can be loaded now right? seems things are stabilizing

Not here :/ in California.
Posted by Sean M, 03-20-2010, 08:40 PM
I can load http://www.hostdime.co.uk/ fine from Texas.
Posted by TBradley, 03-21-2010, 12:21 AM
That Amount of downtime is absolutely unacceptable. That is ridiculous that it would take that amount of time.
Posted by xlguy, 03-22-2010, 07:10 AM
They finally got the server back online 17 hours after taking it offline, and the reason it was offline (not that we ever got any warning, despite claims "at least 2" emails were sent out) was to make more space in the data center for new customers, which makes it even more frustrating for existing customers to be so negatively effected.

As I said earlier in this thread, I really hope the senior management of HostDime consider how badly this was handled, with no updates to the status page, no emails to customers explaining the prolonged delay and support staff not being able to give any kind of ETA even right up to 2 hours before it was fixed - it gave the impression they had no idea of the scope or nature of the problem.