Tailormadeservers / Colo4Dallas down?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Tailormadeservers / Colo4Dallas down?

Posted by xnpu, 08-10-2011, 12:22 PM
Hi, Anyone else having trouble reaching tailormadeserver or colo4dallas? Thanks.
Posted by xnpu, 08-10-2011, 12:23 PM
Noticed the Knownhost thread also discusses this just now. Maybe this one should be closed.
Posted by avizzle, 08-10-2011, 12:27 PM
Colo4 is experiencing a power plant issue causing the facility to go down. Vendors are working to restore power.
Posted by daviolinist12, 08-10-2011, 12:27 PM
Yes, TMS and Colo4 are down as we have servers in the datacenter as well. All of our servers are down. Apparently this is due to a power issue at the DC which is affecting everyone.
Posted by andryus, 08-10-2011, 12:32 PM
Many of my servers with tms are down.
Posted by Tarak, 08-10-2011, 12:35 PM
Strange, datacenter like Colo4 is having power issue.
Posted by cpservr, 08-10-2011, 12:36 PM
yeah, i can confirm that too.. and my browser keep loading after clicking submit ticket button.
Posted by dexx, 08-10-2011, 12:45 PM
some of our servers are down too. no news yet?? Colo4 is one of the worst experiences, and seems to be down with DOS attacks, and now for the power issue??
Posted by ASIMichael, 08-10-2011, 12:49 PM
We are also down and seems the colo4.com website also. Phones are all busy. I think they have voip so if its power or network related thats why the phones are down. Gives thought to having backup #s with cell phones. Hang in folks..might be a long day. I myself might have to drive down just for an update
Posted by KevinJ5, 08-10-2011, 12:49 PM
Same with Colo4.com Anyone have a better solution? Back when it was Colo4Dallas.com I don't remember having a single bit of downtime.
Posted by Hamid Reza, 08-10-2011, 01:09 PM
Anyone can confirm its a power failure?
Posted by Hamid Reza, 08-10-2011, 01:10 PM
I should note that all servers didn't go down at the same time, Thats why I wonder whether its a Power Failure (which would take all servers down at the same time)
Posted by Patrick, 08-10-2011, 01:11 PM
Confirmed by Jose.
Posted by Speckz, 08-10-2011, 01:11 PM
I was told this came from Colo4:
Posted by TMS - JoseQ, 08-10-2011, 01:17 PM
What was posted above, is all the info we have as well... we're awaiting a resolution.
Posted by xnpu, 08-10-2011, 01:20 PM
Thanks for that. Good to know you're aware and standby.
Posted by linux-image, 08-10-2011, 01:21 PM
Any update ?
Posted by ! Unique !, 08-10-2011, 01:21 PM
Hello ! we are lost a lot of money ! what can we do now !! if u can't solve this problem please give me my server backup and recharge back my money !
Posted by andryus, 08-10-2011, 01:23 PM
This problem is not from TMS , is from their datacenter , Colo4.
Posted by Patrick, 08-10-2011, 01:23 PM
I've been with TMS since late 2006 or early 2007 and this is the first time something like this has happened. Colo4 is considered one of the best data centers in the world, sometimes things just go bad and redundancies fail... sh*t just happens? If you're new to TMS, don't let this outage ruin the fun. It's definitely one of those one in a million freak incidents and it could happen to anyone.
Posted by Mark L, 08-10-2011, 01:24 PM
Colo4 has posted an update here - http://www.webhostingtalk.com/showthread.php?t=1072692
Posted by VPSForge-Ray, 08-10-2011, 01:24 PM
New thread http://www.webhostingtalk.com/showthread.php?t=1072692 Edit: Mark beat me by seconds
Posted by MikeTrike, 08-10-2011, 01:26 PM
I can also confirm a colo4 power issue, ColocateUSA and ApertureHost are also impacted. My web sites be down to ongoing colo4 power issues.
Posted by elmister, 08-10-2011, 01:27 PM
Stay calm, server will be back sooner or later, if you are loosing "a lot" then you should consider a multiple locations setup. Failures happen sometimes, i've been with TMS for 3 years and this is an unusual issue, i'm not worried, i trust the people behind TMS and Colo4, we will be up again soon
Posted by ! Unique !, 08-10-2011, 01:31 PM
I'm afraid if i lose backup lose all thing !
Posted by Patrick, 08-10-2011, 01:34 PM
Relax, your server is OK. You're not going to lose anything. Take a deep breathe, that's all you can do at the moment. No point in freaking out when you can't control the outcome. Colo4 is working as fast as humanely possible to get everything back up, and then any lingering issues related to TMS will be taken care of by Jose and his crew. You're in good hands, relax.
Posted by ! Unique !, 08-10-2011, 01:35 PM
ok .. well done .. i am wait .. Am Relax now
Posted by elmister, 08-10-2011, 01:36 PM
Always keep local backups, external USB hard drives are cheap enough, download them twice a month with rsync and you'll sleep fine ;-) Don't worry, probably all our servers are safe.
Posted by dexx, 08-10-2011, 01:47 PM
Jose, Any ETA?
Posted by colnector, 08-10-2011, 01:54 PM
Our TMS server is also down. I created a ticket with the highest priority that, as written, should be answered within 15 minutes. I got no response although an hour passed. There's no indication on http://www.tailormadeservers.com that something is wrong and without WHT I would be out in the dark. Why is that? Any idea when power should be back up?
Posted by Patrick, 08-10-2011, 01:58 PM
You're probably being left in the dark because they have hundreds of other open tickets regarding the same matter. There is nothing for them to report. The data center has just stated: I don't have an ETA on this plant returning. The contractors are going through the systems now to determine what caused the outage. This must be done before restoring a 2000 amp ATS (that's 2000 amps at 480 volt). Monitor this thread for updates from Colo4: http://www.webhostingtalk.com/showthread.php?t=1072692
Posted by cpservr, 08-10-2011, 01:58 PM
i close my priority 1 ticket after seeing this thread. hope he'll answer you shortly, at least my ticket not in queue list..
Posted by EGXHosting, 08-10-2011, 02:04 PM
This explains why my sites are down now. Thanks everyone for the information.
Posted by jackpx, 08-10-2011, 02:11 PM
I have 2 servers in Tailor ...1 Online, 1 Offline
Posted by uberhostNET, 08-10-2011, 02:15 PM
As if things weren't bad enough, we have been getting calls from Colo4's customers because they can't answer them, lol. On my tracking screen right now I see several or more people on our site because they searched for "Colo4Dallas" or "Colo4" using Google. I guess we come up fairly high in the search results so they call us on our toll-free number to get answers. How many ways do we have to pay for this outage, lol? Cheers all, Rick
Posted by colnector, 08-10-2011, 02:24 PM
and I'm happy this thread exists so at least I can find out what the problem is. I guess hosting providers should have some form of automatically informing all their customers of critical errors (power failure being one). I do get emails when a planned hardware upgrade is suspected to create a shortage so why can't I get an email that an unexpected shortage happens? I hope to soon see the servers up and running. Crossing my fingers. Thanks
Posted by solidhostingph, 08-10-2011, 02:25 PM
I just received an email from Jose confirming power issue at Colo4.
Posted by TMS - JoseQ, 08-10-2011, 02:28 PM
I'm sorry I can't update the helpdesk... the server is using resolvers that are down due to the power issue. On top of that, the server only allows ssh connections from certain hosts, all of which are down, so I can't fix the resolver issue either. Unfortunately I'm getting my updates just as you are, via Colo4's WHT thread. Just like us, they're probably being flodded with requests for updates, and they're mostly focused on getting things resolved rather than delay the fix to provide updates to the hundreds of people asking. Rest assured the problem will be fixed as soon as it is humanly possible.
Posted by colnector, 08-10-2011, 02:41 PM
Thanks Jose for your response
Posted by elmister, 08-10-2011, 04:23 PM
Let me suggest you to add those ips to other server meanwhile your dns servers are down
Posted by albatroz, 08-10-2011, 06:44 PM
Everything is still down? I can´t believe that it is almost a 12 hours outage.
Posted by Speckz, 08-10-2011, 06:48 PM
It started around 12:05 PM EST. It's just past 6 hours now.
Posted by Patrick, 08-10-2011, 06:49 PM
Colo4 is starting the process of restoring power... they have given an ETA of 18:00 -> 19:00 CDT for everything but that could change.
Posted by elmister, 08-10-2011, 07:40 PM
2 of my servers are up again!
Posted by TMS - JoseQ, 08-10-2011, 07:43 PM
Switches and services are coming back. Please note not all servers will be coming online on their own, so they will need manual assistance. We will be doing our best to get everyone back online as soon as possible. Since not all switches are yet online, please refrain from opening tickets just yet. We'll update here and twitter once all of the network is ready to go.
Posted by elmister, 08-10-2011, 07:44 PM
Just want to add TMS twitter -> http://twitter.com/#!/tailoredservers Last edited by elmister; 08-10-2011 at 07:48 PM.
Posted by jackpx, 08-10-2011, 08:02 PM
I hope that my server will be online soon.
Posted by VPSForge-Ray, 08-10-2011, 08:14 PM
My server is still offline
Posted by sodapopinski, 08-10-2011, 08:17 PM
One server is up and one server is still down. The problem is the server which is up is remote backup server and the helpdesk system while the other one which is live server for clients still down
Posted by Speckz, 08-10-2011, 08:39 PM
From https://twitter.com/#!/tailoredserve...50969962921984
Posted by Patrick, 08-10-2011, 08:55 PM
From their Twitter feed: Several switches and PDUs are still offline. Please be patient. I see a ton of tickets, but I can't respond until we're online. Stop submitting tickets people until Jose gives the OK.
Posted by solidhostingph, 08-10-2011, 08:58 PM
Patience is a virtue! :-)
Posted by forumtalk, 08-10-2011, 09:06 PM
good I am almost dead
Posted by elmister, 08-10-2011, 09:24 PM
4 servers recovered 9 more and i can go to sleep (3:24am here)
Posted by solidhostingph, 08-10-2011, 09:32 PM
How long is the wait for these 9 servers?
Posted by solidhostingph, 08-10-2011, 09:36 PM
@elmister are you from TMS?
Posted by uberhostNET, 08-10-2011, 09:39 PM
From the Colo4 thread: http://www.webhostingtalk.com/showpo...&postcount=317
Posted by forumtalk, 08-10-2011, 09:42 PM
" Current Update: As of 6:30PM CDT all PDUs are online supplying power to our customers. " then why still down
Posted by Patrick, 08-10-2011, 09:44 PM
Yeah, I saw that. Who knows..
Posted by elmister, 08-10-2011, 09:45 PM
I'm a customer. Last server was powered on 30 minutes ago. Colo4 may be giving power to customers since 2 hours ago, but each customer, (and TMS is colo4 customer) may have their own PDU that may need to be switched on individually In other server i noticed the server got power 1 hour before it got network access, server was powered on but switch was off
Posted by elmister, 08-10-2011, 10:06 PM
Another 7 servers went up, only 2 to go ;-)
Posted by Patrick, 08-10-2011, 10:08 PM
@elmister: Were your servers rebooted or did they maintain their uptimes?
Posted by elmister, 08-10-2011, 10:24 PM
at this exact time i'm looking at 2 servers with 3h 6 min uptime, but they started to respond to ping about 25 minutes ago. So i think all servers were powered on, but switches weren't or maybe switches needed some kind of configuration
Posted by elmister, 08-10-2011, 10:26 PM
Of the 2 remaining servers one is online but having big packet loss, i'll have to open a ticket, i'm afraid Jose might have thousands of tickets and it will take a lot to get assistance on this
Posted by VPSForge-Ray, 08-10-2011, 10:43 PM
This is taking too long. My server is still down and ticket still unanswered
Posted by jackpx, 08-10-2011, 10:50 PM
yes,... my server online.
Posted by AWH - Kalpesh, 08-10-2011, 10:55 PM
Still waiting for our server to kick back into life as well. Hopefully it won't be too much longer after hearing some of yours are slowly responding.
Posted by VPSForge-Ray, 08-10-2011, 10:57 PM
Still waiting for mine...
Posted by elmister, 08-10-2011, 11:00 PM
I got answer to one of my tickets, be patient
Posted by daviolinist12, 08-10-2011, 11:00 PM
Stay patient everyone. Jose is working very hard to get everyone's servers back online - just spoke with him on MSN.
Posted by elmister, 08-10-2011, 11:07 PM
i think we shouldn't speak with him on msn/icq/etc, it's not a good time for speaking with him. Having multiple message windows increases stress
Posted by daviolinist12, 08-10-2011, 11:08 PM
In order for my servers to get online, he needed to manually get them up. - that's why.
Posted by elmister, 08-10-2011, 11:25 PM
last 2 servers fixed. Dont worry, JoseQ will answer your tickets, he is fast. He fixed them but didn't answer the ticket, just check your server from time to time, maybe he fixed them while fixing other servers at the same rack.
Posted by AWH - Kalpesh, 08-10-2011, 11:33 PM
I haven't submitted a ticket to let them know our server isn't responding. Should I let him know or wait? I've been quite patient thus far and I don't want to disturb him.. I'm just not sure if they are checking ALL servers or just the ones there is a ticket for.
Posted by Patrick, 08-10-2011, 11:36 PM
Yes, put a ticket in.
Posted by forumtalk, 08-10-2011, 11:48 PM
still down here 3 more to go
Posted by Hamid Reza, 08-10-2011, 11:48 PM
I have some affected servers without uptime change. That means the switch/router behind the server were affected by this power failure?
Posted by AWH - Kalpesh, 08-10-2011, 11:56 PM
We are up! Thanks to Jose and the rest of the team.
Posted by forumtalk, 08-11-2011, 01:41 AM
one more to go
Posted by TMS - JoseQ, 08-11-2011, 03:36 AM
Sorry I couldn't keep checking here once we got things going... As Murphy's Laws would have it, it wasn't a simple flip of a switch. A bunch of our Remote Reboot PDUs decided to power on with all of their outlets turned off, and no response on their respective IPs. A tech had to manually go through each of them, reset the config and then set it to power on all of the outlets. Some of these PDUs had more than one switch attached to them, so many servers came up but had no connectivity due to this issue. Likewise, some servers were up through the entire ordeal, but the switches they were connected to had no power. Most of the hardware is back to normal. There were some server casualties, some of which are still being dealt with. If your server is still offline, please don't hestitate to contact me on IM or open a ticket. I offer my apologies for this event. Unfortunately it's one of those things that although rarely, they can happen. I appreciate your patience and understanding during this whole ordeal.