Saturday Scheduled Power Outage 6am-6pm Datashack/Joes/Wholesale

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Saturday Scheduled Power Outage 6am-6pm Datashack/Joes/Wholesale

Posted by Techno, 02-20-2012, 06:33 PM
"Dear DataShack Customer, On Saturday, February 25th, Oak Tower will be performing maintenance on the main switch gear feeding the building's power. Our Dedicated servers located in Oak Tower are with two facilities. Wholesale Internet and Joe's Datacenter. The vast majority, located at Wholesale, should be unaffected by this maintainence as Wholesale will run on Generator for the duration. We do have a couple hundred servers, all Dell and HP units, that are colod with Joe's. Joe's Datacenter will be without power for the duration of the maintenance. Power will be unavailable starting at 6:00 central US time and will return by 18:00 central US time. The building could complete the work earlier. These times are worst case. This event will not affect any server at the Cable Building in North Kansas City. If you have questions about where your server is located please let us know. If you have anything other than a Xeon or Opteron you will not be affected." Last edited by Techno; 02-20-2012 at 06:36 PM.
Posted by mcdl, 02-20-2012, 06:39 PM
Great, this + their power outage the other day. I never got an email stating all this. (Just noticed their announcement but they could have emailed for some that do not check their site often. Last edited by mcdl; 02-20-2012 at 06:45 PM.
Posted by MrRadic, 02-20-2012, 07:44 PM
I don't believe that Joe's has any sort of backup power. In reality when you're paying such low prices you shouldn't complain. They don't offer any sort of uptime SLA. If this is effecting your business, raise your budget 25% and move.
Posted by Chris_H, 02-21-2012, 07:08 AM
According to Datashack, they should still be online using generator power. Joe's customers might not be so lucky. Just remember to backup any sensitive data just in case to prevent file corruption.
Posted by pubcrawler, 02-21-2012, 08:41 PM
I like the trio of popular interrelated providers in KC (WII, Joe's, Datashack). Joe needs to do something about power backup though. It's just as affordable to be with the other two companies, actually perhaps less depending.
Posted by DeltaAnime, 02-21-2012, 08:49 PM
Isn't datashack with WSI anyways? Or do they have their own space now? Francisco
Posted by techjr, 02-21-2012, 09:40 PM
I Believe DataShack originally used WSI and now Aaron is helping them open a facility of their own. Not sure of the location in the area but I'm hopefully going to visit 1102 Grand soon so hopefully I will find out and take a tour of datashacks facility if I have the time too.
Posted by DataShack, 02-21-2012, 10:29 PM
None of the three of us is at 1102. We don't expect to have any downtime due to the power outage except for our servers located at Joe's which is a couple hundred boxes at most. A very small subset of customers. Aaron and I are partners in a realestate group outside of our respective hosting companies. We are currently building out a new data center in a property we purchased in North Kansas City. When we are finished both Wholesale and DataShack will move all dedciated servers there. Wholesale will keep colo customers at Oak Tower and DataShack will keep colo customers at The Cable Building. Aaron hopes to be done sometime in April.
Posted by techjr, 02-21-2012, 11:46 PM
Thanks for the info. I noticed it being mentioned a few times that both company's were working together but from what I heard it was more of assistance from Aaron who has done facilities before, this is even better news in my opinion. I didn't word it correctly, I meant to say I will be going there and hopefully be checking out your facility while I'm traveling around (With permission of course etc)
Posted by chasebug, 02-22-2012, 01:15 AM
How long downtime are we looking at for this planned move and you say it is going to be in April of this year?
Posted by fancypocket, 02-22-2012, 01:38 PM
I am interested to know about this moving downtime too before getting more servers from them.
Posted by cnnd, 02-23-2012, 12:20 AM
I am intersted
Posted by WII-Aaron, 02-23-2012, 11:42 AM
Your servers will be moved on a schedule convenient to you. Total downtime will be about 30 minutes while the server is moved. We will begin moving in April. Since we will be scheduling each customer individualy we expect not to be fully consolidated for 6 months.
Posted by Patrick, 02-23-2012, 11:43 AM
^-- This is why I love you guys!
Posted by mcdl, 02-23-2012, 02:50 PM
That will be a hell of a task.
Posted by WII-Aaron, 02-23-2012, 03:20 PM
Since we're not shutting down any current facilities or moving any current infrastructure we can take our time and schedule each server move whenever the cusotmer wants it. I really think it'll be painless. Plus our larger colo customers like 1and1 and Virpus are staying at Oak.
Posted by dmorley, 02-25-2012, 04:02 PM
I don't understand why Datashack has nothing posted anywhere about this, I did not get the email so being offine right now is a shock. They have a website, twitter, facebook but yet no mention... Come on Datashack!
Posted by Techno, 02-25-2012, 04:17 PM
Maybe it went to spam. I received an email on the 20th. But you are right - nothing on Twitter or Facebook.
Posted by WII-Aaron, 02-25-2012, 04:50 PM
We sent the announcement Monday via e-mail in addition to posting it in our support system. DataShack did the same. The issue at about 1pm was that the Generator, which had been running since 6am had a loose over voltage pot. The vibrations had caused it to move down and when it hit 480 the generator sensed it was over voltage and shut itself down. One of our techs went out and cleared the faults which is why it came back on but then went down again in a few minutes. Our generator guy dialed it back up to the correct over voltage setting and applied nail polish to keep it there. He is also checking it every 30 minutes to make sure there are no other issues. As an added safety, we've moved the HE and Cogent building feeds onto a smaller, secondary generator so if we experience any other issues the other building tenants that still have power will be unaffected. There were several issues this morning when the building tried to shut power down that delayed their maintainence. We aren't sure if utility power will actually be restored by 6pm. I can't stress enough: PLEASE do not come to the building. The elevators are down, the lights are out and the emergency lights in the stairway are out. It is not a safe situation for people unfamiliar with the building layout.
Posted by UMSN, 02-25-2012, 05:22 PM
I am seriously thinking about switching to WII from Joe's.... No backup power? And the prices are the same... seriously? And like you said, there was a delay, do you have any progress updates? I need my machines back on...
Posted by totalgeek, 02-25-2012, 05:48 PM
I had this issue with Datashack just recently. Someone said a family member passed but as a business all I care about is service that I ordered commnication is key. Datashack seems to really have bad customer relation or communication skills. Btw I seem to be down again for Datashack.
Posted by pubcrawler, 02-25-2012, 05:50 PM
Down too with Datashack. Looks like Wholesale is offline. Virpus' website is throwing errors again too. Maybe that 2nd generator for core routing equipment went offline?
Posted by ramnet, 02-25-2012, 06:09 PM
Nope. Was everything from the looks of things. Back up now.
Posted by UMSN, 02-25-2012, 07:26 PM
Does anyone have a status update on the building in general? I'm hosted at Joe's, and am still down. Very frustrating.
Posted by chasebug, 02-25-2012, 07:32 PM
You really should leave Joe's. Last year there was a maintenance like this and my server at Joe's had over 10 hours downtime while other server at WI had no downtime. This year a little different, 2 outages at WI lasting about 30 minutes each.
Posted by UMSN, 02-25-2012, 07:33 PM
Well it's also a great frustration to deal with my IP's. My block is owned by Joe's, maybe I'll host my new server with them.
Posted by JOEsDC, 02-25-2012, 07:38 PM
Building power is scheduled to be back on at the 6:00 mark. That being said there have been numerous issues during the maintenance that caused the power to be down the entire maintenance window. And until its back up I cannot guarantee a time. As for Joe's Customers even when the power is back on we have to power everything back on and it may take a little bit to get going. We have staff on site ready to power back up so most should be back on right away. Last edited by JOEsDC; 02-25-2012 at 07:39 PM. Reason: clarification
Posted by Postbox, 02-25-2012, 08:29 PM
It's a pity you didn't notice that Techno posted it here several days ago. Maybe cut Datashack a little slack right now? http://www.webhostingtalk.com/showpo...1&postcount=10
Posted by WII-Aaron, 02-25-2012, 08:49 PM
Building power has been restored. We've cut back over from generator. If you are still having issues please send in a ticket. We have several additional techs in to handle the cleanup.
Posted by MichaelMcC, 02-26-2012, 10:33 PM
The server I have with Aaron came back fine after a fsck. The server I have with Data Shack somehow got corrupted when power failed and I can't get the sites up again. I can get to them via IP only but not the by domains. Scripts will not run and I can't get into DA or FTP. Scratching my head on this one.
Posted by Aquamarina, 02-27-2012, 06:46 AM
Ok, I'm a little bit freaked out here. The power restored but my sites never. I'm 2 days down now. Now at joes they say hdd is corrupted. How on earth can this happen?
Posted by MattHouston, 02-27-2012, 06:48 AM
It probably got corrupted due to how the server was shut down. Did they run an FSCK?
Posted by Aquamarina, 02-27-2012, 06:58 AM
They say they did everything. Now they are going to setup a new hdd and give me access to the other one as slave. I hope I can recover the data, otherwise I should vanish from earth. My clients have already started screaming...
Posted by chasebug, 02-27-2012, 06:59 AM
My server at datashack never lost power, only lost internet connectivity. Are you sure your server lost power?
Posted by Server Management, 02-27-2012, 07:00 AM
Thats why you should have off site backups...
Posted by Aquamarina, 02-27-2012, 07:03 AM
I have off site backups but not all dbs. Dbs are always the problem...
Posted by MattHouston, 02-27-2012, 07:03 AM
+1 - totally agree with this.
Posted by MichaelMcC, 02-27-2012, 09:36 AM
Both went down at one point. I've had servers with WII since 09 but the server I have 1 with DS going on a year. The one thing people need to understand is that an unmanaged server pretty much means if anything happens, as it did with the power outage, you are pretty much on your own. WII did reboot my server and had it fsck to repair itself. Not so lucky with DS, I SSH'd into it and rebooted/fsck and have updated everything but it's beyond me what the problem is. Not all services are running so I can't ftp to it or get into DA. I suppose I will hire someone with more experience to rebuild the server. To answer someone elses question up the thread about them working together. The person who was more than willing to help me out and get my WII server going is the exact same person who told me I was on my own with DA. So, support tickets are handled by the same people, at least yesterday. Ok, now to find help.
Posted by JOEsDC, 02-27-2012, 01:49 PM
OMG I don't normally get into these kind of threads but I'm so excited that I must. I listened to the drive as the machine powered on and noticed it was not spinning up and not even making the normal dreaded click click click. I then tried a kind of unorthodox fix on the drive and it seemed to work! I hooked power to the drive without the sata cable plugged in and tapped where the motor is until I heard the drive spin up (was not doing it before) then hooked the sata cable in booted the machine without removing power from the drive. I then put a hard drive seek delay in the bios as well as turning off smart. After doing all this I was able to get it to boot to the drive! I already installed a new 2tb drive as a slave and put a clean partition already ntfs formatted so data can immediately start being backed up to the new drive.
Posted by Aquamarina, 02-27-2012, 02:05 PM
OMG! I love Joe! )))))) He really saved my company -at least till the crisis drops us all dead :p. I had started searching for excuses and how I will tell those few clients that we are out of business. Now I can continue building my two new sites -after I backup, backup and... backup. Thanks Joe, you worth every penny!
Posted by ItsChrisG, 02-27-2012, 02:32 PM
and how about getting RAID?? Even a RAID1... and MONITOR it to make sure you catch a drive that drops out of the raid array before the 2nd dies.
Posted by VN-Ken, 02-27-2012, 03:02 PM
Hopefully this will teach you to take backups... amazes me "saved my company".
Posted by ramnet, 02-27-2012, 03:43 PM
Joe that's a great story :-) All's well that ends well.
Posted by kyay, 02-27-2012, 04:45 PM
Love that story!
Posted by p0go, 03-01-2012, 10:13 PM
Great job, WSI, DS, and Joe's... Good to see every thing back online and running smooth. Some great providers right here... Good luck for the upcoming planned work in April also, I can't wait too see the new facility and hopefully it's a easy ride for you guy's to complete that whole mission, as you said it should be painless, but, that's when you get the most problems in this business, when it should be painless, it becomes the most problems happen, haha. -p0go.