WiredTree Servers Down Again

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > WiredTree Servers Down Again

Posted by Serverji, 12-10-2009, 04:52 PM
One/Some of WiredTree Servers are Down Again Today.

This is the latest we have heard from them in 4 hours

"Late this morning, we got an alert from our monitoring that the load averages on the hardware node in question were above our acceptable limits. When we tried to log into the server via SSH, our attempts were unsuccessful. We immediately dispatched our data-center techs to check out the hardware node. Upon plugging in a console to the hardware node, we found that it was unresponsive to keyboard interaction and the screen was blank. We rebooted the server and watched it POST and load the RAID card BIOS. Just as it started booting it locked up again. We then took the server off the rack and started troubleshooting the hardware issue. After swapping out the RAID card and cables, the system is still not not able to boot. Unfortunately, our data center techs are seeing the RAID array on the hardware node in such a failed state that recovering any data off of the array is very slim at this point. With that said, we are going to be restoring your VPS to different from our nightly backups to get your server back online as fast as possible.

While this kind of event is not very pleasant or fun to endure, we assure you that we are working as fast as possible to get your VPS back online. The backups that we are restoring from are fresh from last night. We will be looking into what caused this issue and working with our hardware vendors for a solution. Thank you for your patience and understanding."

Is there anyone facing downtime at this moment ?
Posted by httpEasy, 12-10-2009, 05:00 PM
Be prepared to have this thread merged as previous ones. Best of luck.
Posted by WiredTree Zac, 12-10-2009, 05:13 PM
We have one VPS node which has suffered a hardware failure we are in the process of restoring now. From first reports it has suffered a simultaneous disk failure. Some VPS are already back up and all are in progress currently. We deeply apologize for the inconvenience.
Posted by AlanTan, 12-24-2009, 12:47 AM
Server already down more then 6hrs , that really hurt my business.
Posted by frki1, 12-24-2009, 01:06 AM
My also..6 + hours and..and i dont see when it will be up...
Posted by AlanTan, 12-24-2009, 01:22 AM
almost every month server down this time is the worst ...
Posted by frki1, 12-24-2009, 01:27 AM
Well it will be almost 7 hours now...i vrealy think that 7h is tooo much..
Posted by AlanTan, 12-25-2009, 01:23 AM
down so many hrs coz me loose USD$600++
Posted by badboyx, 12-26-2009, 06:25 PM
i think WiredTree should study this case well and find a solution to decrease the downtime from 6+ hours to 1 hour or less
Posted by WiredTree Zac, 12-26-2009, 07:51 PM
The downtime was caused because Virtuozzo kernel panicked and when the server was rebooted, it forced a filesystem check (fsck). Unfortunately when that happens there is not anything we can do to speed things up in any way - you cannot skip the check when it is forced (and even if you could it would be a bad idea as you would be risking major data loss/corruption). We were on top of the situation from the start and if there was any way possible we could speed the process up, we definitely would.
Posted by AlanTan, 12-26-2009, 10:51 PM
that is 24 DEC close to Christmas , do you know how many sales can make at that day .
Posted by AlanTan, 12-26-2009, 11:09 PM
They even no send any email inform me that server down , if they inform me early i can stop my campaign at Pay Per Click search engine.
Posted by badboyx, 12-27-2009, 05:58 AM
using SSD is the best solution for speed-up the fsck
i know its expensive but you can make a special vps's plans that provide SSD
even a 10GB of disk space for 50$ as its going to be much faster than the normal drives