Hostv Down For 120 hours!!! Hostv Screw Me up?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Hostv Down For 120 hours!!! Hostv Screw Me up?

Posted by desirez28, 02-08-2010, 11:37 PM
This is the worst thing happen in my life. 5 days down, 120 hours gone. seo ranking drop, loss thousand of dollar because I trust hostv VPS hosting. They have entertain me for 4 days since their node 16 was down. You can check it here: http://twitter.com/hostv and you will know what is happen.

Here is the date it's happen, 4th Feb 2009 is down. I had sent an email and request for support. They mention that's maintenance progress. So I wait for 24 hours and I sent an email again, what's happening? They said now is the server problem, cannot work. Now already 72 hours, I beg and request for help since then they said will restore back in 10 hours time, here is the quote:

https://mail.google.com/mail/images/cleardot.gifHostV Support

to me
show details Feb 7 (2 days ago)

Hi,

Unfortunately the insitial speed of restore was a little slow. Around 100 GB has been restored and we expect 500 GB more remaining. We think this will complete in 10 hours from now as the speed has increased. We appreciate your patience and cooperation.

So I still keep waiting, but I think you guys can guess what's next:

HostV Support

to me
show details 11:54 AM (23 hours ago)

Hello,

Yeah I agree that's its been a long delay and we have been having a bad time now. But in this rate we expect this to be over within 20 hours max.

Thank you

So what's the next episode? delay max another 1 month? Or what? Somebody who respond on this matter please clarify it, I don't like to be treated like fool, that's why I post this if anyone have the same problem with me.

BTW: They are very polite and nice people there, just the thing is not work so I really frustrated and tired of my clients and my income decease drastically.

I hope they can give me special price to convert to dedicated server or convert to VPS server which are no problem. At least, what I think you can do to reduce the pain I feel so far. Hope somebody understand my situation.
Posted by Cirtex, 02-08-2010, 11:45 PM
Hi Desirez

I understand your frustration, the downtime is only affecting one of our nodes (NODE-16) out of many but we've been working our best with R1SOFT to restore all services from backup, from an unexpected hardware failure which corrupted main server raid.

The R1Soft backup restore is taking longer than expected and we've ran into few bottlenecks which the r1soft team have been trying to address, we'll keep everyone updated and work to restore service asap.

Please also note we will credit or refund for this downtime accordingly, please do contact billing once service is restored.

We truly apologize for this inconvenience, this is very rare situation for us. Thank you for your understanding and patience.

Cheers
Posted by desirez28, 02-09-2010, 01:28 AM
I understand this already happen. But previously you guys promise, 10 hours, 20 hours max, how long should it take to get it back? I need clients, no clients no money. This is online business so please do take this seriously. What's the time I can get back all my websites online?
Posted by Cirtex, 02-09-2010, 01:38 AM
Hi

Please understand we're trying our best, the delay is due to few issues while restoring via our backup server. We're working with the software developers at R1Soft to resolve these bottlenecks and things are smoothly restoring now atm and hopefully we'll have everything up and running by tomorrow morning!

Again we apologize for the inconvenience caused!

Cheers
Posted by desirez28, 02-09-2010, 01:40 AM
Can you please move my account to another server? I really need it in the best way!!
Posted by Cirtex, 02-09-2010, 01:50 AM
Hi

Once it is restored, we can migrate you if you wish however I don't think it will be necessary as we had re-installed entire server and replaced the faulty hardware that had caused the data corruption.

However, do feel free to contact us once everything is up and running again with any requests!

Cheers
Posted by desirez28, 02-09-2010, 04:08 AM
I HOPE THIS WILL BE SOLVE TOMORROW. I WILL REPORT BACK THE STATUS HERE UNTIL I CAN GET EVERYTHING DONE. REALLY SCREW UP WHEN THIS HAPPEN, I NEED A BETTER HOSTING THAN THIS ONE, REALLY.
Posted by hostingvince, 02-09-2010, 10:53 PM
I am having a similar issue with one of my VPS, not hosted with Hostv BTW

The response I got was that it is the R1 system that is good, but very very slow.

In my case, a 30GB restore is on schedule to complete in approx 30 hours - yep, 1GB per hour!

I am told that it's pushing around 2-3mbit of data
All I know is that my home PC can transfer data quicker than that.

OK, so R1 does all the encrypting stuff at the same time, and many files are small which is a bummer as it only transfers 1 single file at a time. Why can't they transfer a complete tar image for the host to uncompress at their end?

Is there no better alternative to R1 for hosts?

- Vince
Posted by venkatam, 02-10-2010, 12:49 AM

Quote:

Originally Posted by hostingvince

In my case, a 30GB restore is on schedule to complete in approx 30 hours - yep, 1GB per hour!

Is r1 soft always this slow or is this just a bug in their software that needs to be fixed?
Posted by calamine, 02-10-2010, 03:23 AM
looks like HostV is getting bad publicity..HostV (Cirtex) node16 failure has appeared in theregister.co.uk now..
http://www.theregister.co.uk/2010/02/10/hostv_outage/

Quote:

"As for an ETA on the restore, we don't have one,"

Quote:

Update

Citrex CEO John Xie says that the node in questions had a hardware RAID failure after it was rebooted for a server patch. "It caused so much corruption from the single RAID Card failure that we had to restore from backup," he says.

But the company is also having problems with its backup system, from R1soft. "The restore process is still going on, and the main issue is the speed of restore from encrypted and protected files, "We're closely working with R1soft developers...to bring this one particular node back online," he continues.

"There are no excuses for this but we have already offered a refund and migration for all clients on this server after this restore has been completed."

Posted by Cirtex, 02-10-2010, 02:12 PM
Hi

R1SOFT has identified some corrupted data backed up, therefore the explanation was the BAREMETAL Restore (which is fast) failed after many tries.

So we are manually restoring from R1SOFT Backup and it is taking much longer than expected.

We're almost done. R1SOFT is continuing to help us in this matter and said the issue should be addressed in the new patch for the buagent.

Again we apologize to everyone for the downtime.
Posted by desirez28, 02-11-2010, 01:07 AM
Thank you for your reply, I hope you can really fix it asap and make it up asap.
Posted by hostingvince, 02-11-2010, 07:46 AM

Quote:

Originally Posted by venkatam

Is r1 soft always this slow or is this just a bug in their software that needs to be fixed?

Bug or no bug, it is always slow.
Can't believe my VPS restore is still not completed after 45 hours now. :-(

The R1soft.com solution can't even give you any progress report or even tell you how much data they store so you could compare against what has been transferred so you can get a reasonable estimate of how long to complete.

There must be a better way?

- Vince
Posted by desirez28, 02-12-2010, 11:01 AM
Still not up yet la... I wonder how many hours still need to get it done...
Posted by desirez28, 02-12-2010, 03:34 PM
Now going to finish another day, they said will get done in few hours but the now going to be the next 24 hours.... Anyone understand my feeling?
Posted by Cirtex, 02-12-2010, 04:05 PM

HI

We're updating customers on our forums on progress regularly.

Thank you
Posted by desirez28, 02-13-2010, 02:04 AM
Now already pass 24 hours, still not up yet, today is the day 9 of server down. I can't see any shadow of my websites yet. 24 hrs + 24hrs + 24hrs + 24hrs + 24hrs + 24hrs + 24hrs + 24hrs + 24hrs + ? = ????

How many hours you need? Please ask yourself. Really bad time with hostv now...
Posted by Cirtex, 02-13-2010, 02:32 AM
Hi

Please let me know your ticket id. I will investigate. We're restoring service for many VPS already from node16 to new node.

Private message me the TICKET ID & Account info, that way I get email notification of new private messages and can respond faster

Cheers
Posted by desirez28, 02-13-2010, 07:58 AM
I have sent u the pm, please reply me asap, thanks!!
Posted by Cirtex, 02-14-2010, 12:56 AM
Hi everyone,

We've made full updates on hostv forums status section in regards to this entire node-16 situation.

Please feel free to private message me here if you have any concerns and ticket id needing me to investigate.

Cheers
Posted by desirez28, 02-14-2010, 05:21 AM

Quote:

Originally Posted by Cirtex

Hi everyone,

We've made full updates on hostv forums status section in regards to this entire node-16 situation.

Please feel free to private message me here if you have any concerns and ticket id needing me to investigate.

Cheers

I already pm you mr. john, but no reply from your sites at all. all my websites only sure cpanel. what the heck is happening? I need HELP!! ASAP!!!
Posted by Cirtex, 02-14-2010, 06:58 PM
Hi

We're investigating, sorry for the delay.

Cheers
Posted by desirez28, 02-15-2010, 06:06 AM
Now alrady 11 days, all my websites are still down. what is happening??????????
Posted by desirez28, 02-15-2010, 03:18 PM
no more respond from their sites after12 hours, right now what should I do?
Posted by venkatam, 02-15-2010, 03:37 PM

Quote:

Originally Posted by desirez28

no more respond from their sites after12 hours, right now what should I do?

Call up their general inquiries and ask them how they could possibly make this up to you. And of course before you ask them that, ask them when your server will be back online.

http://www.hostv.com/contact.shtml
Posted by Cirtex, 02-15-2010, 03:41 PM
Hi

Please contact us directly or on our forums for fastest response.
We're working on all tickest atm and due to the issues there is a bit of delay but things are settling down now.

Cheers
Posted by Nonchalant, 02-16-2010, 05:21 AM
I am thinking about signing up as a new customer to hostv but... 11 days? I would have freaked out

Now I feel stuck and I'm not even a customer
Posted by Cirtex, 02-16-2010, 05:31 AM
Hi

This is a first for us and really we did not expect the downtime to be so long. There was a very very rare hardware failure w/ the raid that never happened before, and then the backup restore of image/baremetal restore also had issues and we ended up having to do manual restore and trying to recover as much data as possible.

Rest assured, we're working very hard currently to make sure, address and stress testing our backup system so that this does not happen again.

Cheers

Quote:

Originally Posted by Nonchalant

I am thinking about signing up as a new customer to hostv but... 11 days? I would have freaked out

Now I feel stuck and I'm not even a customer

Posted by Henrik, 02-16-2010, 06:47 AM

Quote:

Originally Posted by Nonchalant

Now I feel stuck and I'm not even a customer

How come you feel stuck? You only have HostV as an alternative for your VPS needs?

Cirtex is pretty customer oriented though, as I understand them, and wouldn't leave you hanging in the wind if the worst would happen.
Posted by calamine, 02-16-2010, 09:09 AM
definitely cirtex learnt some hard lessons in this episode.. since this was a first for them , hopefully they would publicly disclose the changes if at all they have made to their setup so as to not repeat such incidence and to boost customer confidence.

" A man has no more character than he can command in a time of crisis. "
Posted by oliviakitty, 02-16-2010, 09:26 AM
Wow, 11 days of downtime. Anyone who stays is a real trooper.
Posted by desirez28, 02-16-2010, 10:59 PM
Damn hostv... All my sites are still not online... They just recover the server..... They really screw me up... Everything gone!!
Posted by calamine, 02-17-2010, 01:55 PM
SOS node16 customers - is this some kinda of world record that hostv is trying to achieve?? the world's lousiest webhost and that too with a "amazing" downtime of 16days on a vps server??
Posted by Cirtex, 02-17-2010, 03:43 PM

Quote:

Originally Posted by oliviakitty

Wow, 11 days of downtime. Anyone who stays is a real trooper.

There was only node-16 affected, and it wasn't down whole 11 days, most of those days the restore from our external r1soft backup was going on that was taking so long.

Quote:

Originally Posted by desirez28

Damn hostv... All my sites are still not online... They just recover the server..... They really screw me up... Everything gone!!

Again, posting here is not going to help, we already said due to the backup restore failure, some customer data was not fully recovered, please contact us to try and see if we can restore anything else, we're doing the best we can and will refund as well if requested.

Quote:

Originally Posted by calamine

definitely cirtex learnt some hard lessons in this episode.. since this was a first for them , hopefully they would publicly disclose the changes if at all they have made to their setup so as to not repeat such incidence and to boost customer confidence.

" A man has no more character than he can command in a time of crisis. "

Agreed, we have disclosed all information in this incident and really did the best we can to restore 100% of files after the hardware failure

Quote:

Originally Posted by calamine

SOS node16 customers - is this some kinda of world record that hostv is trying to achieve?? the world's lousiest webhost and that too with a "amazing" downtime of 16days on a vps server??

The downtime was not 16 days, everything is back online in the node level, but there was data loss for some clients which caused downtime for some of their websites.

Quote:

Originally Posted by sheba59

Wow. That is a long time. Well, I am happy I am no longer with them. I went from them to Liquidweb then to Knownhost where I have been for 2+ years. I feel for you people.

TIA

I'm glad you're doing well w/ knownhost, best of luck with them in the future.

Cheers
Posted by desirez28, 02-18-2010, 12:50 PM
Honestly until now all my sites are still down, but what could I do? Hostv already did their parts and I hope they can restore all my sites again, but at the moment contact with hostv support but it's quite slow, don't know why sometime take more than 12 hours to get a reply. Might be they are busy.
Posted by Mekhu, 02-18-2010, 02:33 PM

Quote:

Originally Posted by desirez28

Honestly until now all my sites are still down, but what could I do? Hostv already did their parts and I hope they can restore all my sites again, but at the moment contact with hostv support but it's quite slow, don't know why sometime take more than 12 hours to get a reply. Might be they are busy.

12 hours? If that's true then please do yourself a favor and go find a new Provider. In this industry you shouldn't have to wait longer than 15-30 minutes for a response.
Posted by Cirtex, 02-19-2010, 07:58 AM

Quote:

Originally Posted by desirez28

Honestly until now all my sites are still down, but what could I do? Hostv already did their parts and I hope they can restore all my sites again, but at the moment contact with hostv support but it's quite slow, don't know why sometime take more than 12 hours to get a reply. Might be they are busy.

Everything is back up and online long long time ago, the reason for downtime as we know is due to data loss for few clients affected by this NODE-16.

Please contact us again and we'll try our best to recover any other data possible, but if not you'd have to restore your own backup.

Again as stated, we can provide full refund, please email billing.

We apologize for this situation.
Posted by jweeb, 02-19-2010, 10:15 AM
An honest host.
Posted by desirez28, 02-19-2010, 11:19 PM
Cirtex, today I just notice my cpanel never have fantastico installed. Can I get it now to create and restore back my websites? I will contact billing department now...
Posted by Cirtex, 02-19-2010, 11:25 PM

Quote:

Originally Posted by desirez28

Cirtex, today I just notice my cpanel never have fantastico installed. Can I get it now to create and restore back my websites? I will contact billing department now...

We can install fantastico no problem, please contact support.

Cheers
Posted by burnwire, 02-17-2011, 01:13 AM

Quote:

Originally Posted by desirez28

Cirtex, today I just notice my cpanel never have fantastico installed. Can I get it now to create and restore back my websites? I will contact billing department now...

Have cirtex/hostv recovered your site from there backup or your site was totally lost?, im having the same problem now. and it really really screw me up.
Posted by Anthony, 02-17-2011, 02:23 PM
Good luck to everyone involved. Is this the first time data loss has occurred?
Posted by burnwire, 02-19-2011, 10:42 AM

Quote:

Originally Posted by Anthony

Good luck to everyone involved. Is this the first time data loss has occurred?

I lost site with them last year, they dont have backups. now my server with hostV is offline for 5 days and assuming what happened last year will happen again.