Ultrahoster Downtime - 5AM GMT - 23 March 2011 - Knowledgebase

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Ultrahoster Downtime - 5AM GMT - 23 March 2011

Ultrahoster Downtime - 5AM GMT - 23 March 2011

Posted by ultrahoster, 03-23-2011, 01:33 AM
Hello Guys,

Unfortunately, we've had a server disk failure, hence the servers may be down for your service.

Don't worry, your data is intact!

We're working hard to get everything back online as soon as possible!

Please stay updated on this page!

Regards,
Ultrahoster
Posted by ultrahoster, 03-23-2011, 02:38 AM
Update #1 - 6.30AM GMT

The datacenter technicians are just digging up the latest nightly backups of all accounts.

More updates to follow...
Posted by ultrahoster, 03-23-2011, 07:30 AM
Update #2 - 11.15AM GMT

Backups are being sent over to our network from remote servers. This may take a couple of hours as we are transferring terabytes of data from an offsite location. However we have given priority to our 1GBPS network so the transfer is done as quick as it can.

Updates to follow...
Posted by wllhung, 03-23-2011, 10:23 AM
Hople it can come back asap.
Posted by ultrahoster, 03-23-2011, 11:33 AM
We are working very hard, actually we're ahead of schedule as we're waiting for the machines (servers & network) to do their job. It is an unfortunate issue, but inevitable as hard disks do fail without much notice.

Should have an update in about half an hour.
Posted by Karl_CLOOK, 03-23-2011, 11:47 AM
What RAID configuration was the server running?
Was it one HDD failing or a whole array?
Posted by Net-Buster, 03-23-2011, 11:56 AM

Quote:

Originally Posted by ultrahoster

Hello Guys,

Unfortunately, we've had a server disk failure, hence the servers may be down for your service.

Don't worry, your data is intact!

We're working hard to get everything back online as soon as possible!

Please stay updated on this page!

Regards,
Ultrahoster

So nice and responsible Act.

Best of luck Guys
Posted by ultrahoster, 03-23-2011, 12:04 PM
Update #3 - 4.00PM GMT

Everything is looking good, all data is intact and fine. We're just restoring accounts so websites should be up live. If you can't access your website, please wait a few hours for DNS to resolve. The Ultrahoster website shall be up in the next hour.

Please feel free to contact us here for any specific questions. Unfortunatley, we can't give a accurate time for everything to be up and working as we're literally waiting for the servers to finish the data transfer and rebuild the accounts which takes quite a bit of processing power, even with our Quad Core servers. However we do hope most of everything is sorted, before the end of tonight, GMT time.
Posted by ultrahoster, 03-23-2011, 12:10 PM

Quote:

Originally Posted by The_Cow

What RAID configuration was the server running?
Was it one HDD failing or a whole array?

I don't have full details for you yet, apart from what's already been posted. The server techs have all the details, but we're letting them do their job. We will be emailing clients with details later on today (once we're back up).
Posted by ultrahoster, 03-23-2011, 08:04 PM
Update #4 - 11.00PM GMT

We're still waiting for the disks to rebuild themselves. Some accounts are already working fine. Shouldn't be long now for the rest.
Posted by hayley2005, 03-24-2011, 12:05 PM
all my sites down hosted with u have been down at least 30hrs now
Posted by ultrahoster, 03-24-2011, 12:43 PM

Quote:

Originally Posted by hayley2005

all my sites down hosted with u have been down at least 30hrs now

Hi, I understand your frustration but we really are working as fast as we can. Having high amounts of data to transfer has really slowed things down.

I do sincerely apologize on behalf of the company.
Posted by hayley2005, 03-24-2011, 01:32 PM
Thanks i will keep an eye on my emails for when your online. Sorry to hear ur having problems.
Posted by tomazg, 03-24-2011, 05:05 PM
Websites stil not working. I think, that you don't have backups! Websites don't working at least 48 hours.
Posted by ultrahoster, 03-24-2011, 06:21 PM

Quote:

Originally Posted by tomazg

Websites stil not working. I think, that you don't have backups! Websites don't working at least 48 hours.

We do have backups on remote servers, but they seem to be taking longer than expected to be fully transferred over. I assure you, that your data is completely safe.

Also, it has not been been 48 hours downtime at the time of posting this message.
Posted by Net-Buster, 03-24-2011, 06:27 PM

Quote:

Originally Posted by tomazg

Websites stil not working. I think, that you don't have backups! Websites don't working at least 48 hours.

Hardware failure can happen to any host !!

When data is copied to another hard drive, it is copied alphabetically i.e. A to Z which means, if your username is "yomato" then your account will be available on new disk at the end however, your client accounts could have become already available whose username's are starting from A,B, or C.

I suggest that you have to be patient as working with machine takes time.
Posted by wllhung, 03-24-2011, 07:29 PM
come on. long downtime.
Posted by rchm, 03-24-2011, 08:16 PM
My customers need fast use the services
Posted by tomazg, 03-25-2011, 01:06 AM
I do not understand why it is necessary to wait that long. If you archive the data to be up and running after a few hours.

Please tell me when websites will be online?
Posted by Tilen, 03-25-2011, 08:05 AM
My websites still don't work. This is been going on for the last 48+ hours.

Do you have any more info of when this could start working? My WHM username starts with letter "G".

Regards,

Tilen
Posted by markpain, 03-25-2011, 09:07 AM
I can't log into my cpanels at all at least give us some type of backup for our customers. This has been the most stressful company I have worked with. Just when you think the going is good (BAM) were hit with something else come on guys lets get this running smoothly already.
Posted by bubblymunkey, 03-25-2011, 12:41 PM
My site has also been down for 48+ hours. I understand it's a big task to get everything back this time but there always seems to be problems
Posted by egcd, 03-25-2011, 01:25 PM
This is madness. waiting to see my customers with torches!
Posted by ultrahoster, 03-25-2011, 02:32 PM
Update #5 - 6.00PM GMT

Guys,

We sincerely apologize for the delays. It is madness here as our datacenter is having other hardware issues with the server, and we have no choice but to get new servers. This has taken far longer than expected, and we are ashamed of ourselves. We have been working endlessly with the datacenter to get things sorted quick.

Please bear with us.
Posted by tomazg, 03-25-2011, 03:51 PM
Please tell us when websites will be online!
Posted by ultrahoster, 03-25-2011, 08:49 PM
Update #6 - 12.00AM GMT

Hello everyone. We have good news! We will have all websites working up and live in the next 3-6 hours of this message.

ShoutCAST and other services may take slightly longer as we have to get the software vendor to restore accounts. Its not long now. I promise!!!

Thanks so much for holding on...
Posted by Tilen, 03-26-2011, 04:14 AM
Yes, websites are working now. Thanks for fixing that.
However, the cPanel option of creating a "full backup" isn't working yet.
Posted by hayley2005, 03-26-2011, 05:39 AM
My sites still arnt online.. insane amount of downtime.
Posted by bubblymunkey, 03-26-2011, 06:04 AM
Mine is back now thankfully, but it's taking a little longer than usual to load
Posted by Tilen, 03-26-2011, 06:12 AM
Yes, the loading of the websites is slower than usual, in my case also.
Posted by Henrik, 03-26-2011, 06:27 AM

Quote:

Originally Posted by bubblymunkey

Mine is back now thankfully, but it's taking a little longer than usual to load

Probably because of the restoration work on the node that you are hosted at.
Posted by lioneye, 03-26-2011, 08:48 AM

Quote:

Originally Posted by egcd

This is madness. waiting to see my customers with torches!

Lol !!! I think this host's resources are confined to 1 server only. If I were in your situation then I would have left the unstable ship as soon as possible...but that is just me
Posted by ultrahoster, 03-26-2011, 08:52 AM
Update #7 - 12.00PM GMT

Hello everyone. Everyone should now be online. There's only around 2 accounts left which are still restoring as they are large in size.

Yes your site may load slowly, but that is temporary as the 1GBPS network was being fully utilized for backup retrieving from the remote servers, and the server load was and is still high as the accounts restore.

All should speed up in shortly.
Posted by ultrahoster, 03-26-2011, 08:53 AM

Quote:

Originally Posted by lioneye

Lol !!! I think this host's resources are confined to 1 server only. If I were in your situation then I would have left the unstable ship as soon as possible...but that is just me

Thanks for your input, but we actually have a few servers from which we operate, which is why not all customers were affected by this issue.
Posted by Tilen, 03-26-2011, 09:46 AM

Quote:

Originally Posted by ultrahoster

Thanks for your input, but we actually have a few servers from which we operate, which is why not all customers were affected by this issue.

Hello.

Could you please check out the option of creating backups from own cPanel? It doesn't let me do one. I can only download "Home Directory" directly, but I can't create "FULL SITE BACKUP" - instead of showing "2011.03.26_username_xyz-backup.tar.gz [IN PROGRESS]" I only have "There are no available backups to download".

Regards,

Tilen
Posted by rchm, 03-26-2011, 10:51 AM
Please, I need access to my account of reseller hosting and reseller shoutcast, but I don't have the passwords, the passwords is in the information of my account in your portal of customers. Thanks.
Posted by hayley2005, 03-26-2011, 01:25 PM

Quote:

Originally Posted by ultrahoster

Update #7 - 12.00PM GMT

Hello everyone. Everyone should now be online. There's only around 2 accounts left which are still restoring as they are large in size.

Yes your site may load slowly, but that is temporary as the 1GBPS network was being fully utilized for backup retrieving from the remote servers, and the server load was and is still high as the accounts restore.

All should speed up in shortly.

The two sites u mention i think are my mates whatafeeling.org & thesaturdaysdaily.org ? as they are still not online.

All my sites are online so thanks.
Posted by hayley2005, 03-26-2011, 02:14 PM
Cant edit my post sorry but i forgot to say that something isnt installed as the Coppermine Gallery script isnt working properly on all sites :/

Quote:

PHP running on your server does not support the GD image library, check with your webhost if ImageMagick is installed

Posted by 2009plus, 03-26-2011, 02:26 PM
So, what has been the total amount of downtime on this node?
Posted by ultrahoster, 03-26-2011, 04:03 PM

Quote:

Originally Posted by Tilen

Hello.

Could you please check out the option of creating backups from own cPanel? It doesn't let me do one. I can only download "Home Directory" directly, but I can't create "FULL SITE BACKUP" - instead of showing "2011.03.26_username_xyz-backup.tar.gz [IN PROGRESS]" I only have "There are no available backups to download".

Regards,

Tilen

Please PM me, and give me your account name and i will check this up. It's probably saying "In Progress" as the server has been constantly restoring accounts for the past 17 hours.

Quote:

Originally Posted by rchm

Please, I need access to my account of reseller hosting and reseller shoutcast, but I don't have the passwords, the passwords is in the information of my account in your portal of customers. Thanks.

We're still having the Ultrahoster site restore itself. Its taking some time as it's huge. Just waiting for it to restore. It's happening as we speak, and it's the only account left.

Quote:

Originally Posted by hayley2005

Cant edit my post sorry but i forgot to say that something isnt installed as the Coppermine Gallery script isnt working properly on all sites :/

Seems that Apache doesn;t have all modules installed. I will have a technician recompile Apache with the required modules. I will update you here.

Quote:

Originally Posted by 2009plus

So, what has been the total amount of downtime on this node?

Approximately, 68 hours from our records. This does vary between client accounts as the account restorations have been happening at various times.

If anyone has an issue, please PM me here.
Posted by ultrahoster, 03-26-2011, 04:14 PM

Quote:

Originally Posted by hayley2005

Cant edit my post sorry but i forgot to say that something isnt installed as the Coppermine Gallery script isnt working properly on all sites :/

I've been told that Apache is being recompiled as we speak with GD, so Coppermine should work fine in the next 30 minutes or so.
Posted by ultrahoster, 03-26-2011, 04:20 PM
Update #8 - 8.00PM GMT

All client accounts have been successfully restored. Only the Ultrahoster website is being restored which is huge. You can visit the domain and you may see the website, but it wont work too well until it's fully restored.

If your a client and your website is not accessible, please wait as it's the domains propagating. You can visit your Cpanel or WHM from the following IP 64.120.180.179.

Server performance may be slightly slow, it's due to the restoration currently undergoing.

ShoutCAST customers, everything is done from our side, however we're waiting for WHMSonic to restore accounts. We've been nagging them for quite some time, but haven't had a recent response. It's unfortunate as only they can restore ShoutCAST accounts themselves, otherwise we may have done the restorations ourselves.

Could we kindly ask clients not to perform backups as of yet, as multiple running backups have been it's slowing down the whole restoration process. Don't worry, we have your data backed up already. Feel free to backup to your hearts content after 8 hours of this message.

Please feel free to PM me, i will keep you all updated here.
Posted by ultrahoster, 03-26-2011, 07:27 PM
Update #9 - 11.00PM GMT

We're happy to be back now. The website is back. Some DNS resolving issues here and there, but they will fix themselves over the next few hours.

ShoutCAST clients, we're still on WHMSonic's back, so your service should be resumed soon.
Posted by rchm, 03-27-2011, 02:56 PM
I have an account hosting that I can't see her information
Posted by Tilen, 03-28-2011, 01:50 AM

Quote:

Originally Posted by ultrahoster

Please PM me, and give me your account name and i will check this up. It's probably saying "In Progress" as the server has been constantly restoring accounts for the past 17 hours.

That's been solved over the weekend. Everything runs normal with my sites at your server now, thanks.

T.
Posted by sx1seba, 03-28-2011, 04:26 AM
Please, I need access to my account of shoutcast reseller, But today is the day: 28 March 2011 and my account is still offline.
I think that is too much offline...
Posted by JackDavies768, 03-29-2011, 10:59 AM
@Ultrahoster - Just sending a quick message to say well done for handling this so well through WHT and all the things people have been saying. Sure it was a slow transition time but well done for handling it so well! Best of luck in the future!
Posted by tomazg, 07-28-2011, 10:37 AM
All web sites not working.
Posted by hayley2005, 07-28-2011, 11:14 AM
up and down for about a week now tbh :/ Email says they migrated from last night, moving accounts to new servers, but no email since last night whats happening.
Posted by target, 07-28-2011, 11:30 AM

Quote:

Originally Posted by hayley2005

up and down for about a week now tbh :/ Email says they migrated from last night, moving accounts to new servers, but no email since last night whats happening.

Look into this thread: http://www.webhostingtalk.com/showthread.php?t=1068858