Ubiquity Server's Dallas - Offline - Knowledgebase

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Ubiquity Server's Dallas - Offline

Ubiquity Server's Dallas - Offline

Posted by Patrick, 07-25-2011, 01:50 AM
Ubiquity moved their entire Dallas operation on Saturday resulting in almost 20 hours of downtime for us, despite their original announcement saying no more than 2 to 3 hours. No real updates as to what happened, outside of a brief notice about 6 hours into the outage that the work would be another 8 hours and that's the last I heard from them. Fast forward to about 40 minutes ago and they are offline again. No notice on their announcements page. A brief update to a ticket that they are aware of an issue. Ubersmith is showing the switches status as down when I check the MRTG graphs. Absolutely nothing to report to our clients who are already nothing short of enraged about what happened on Saturday... Don't get me wrong, the guys over at Ubiquity have been great to us over the last year but this whole weekend has been nothing short of a mess due to the terrible communication. I'm honestly saddened by this, Ubiquity were one of the few providers I truly believed in but this whole experience has left me with a real bad taste. Sigh.
Posted by Patrick, 07-25-2011, 02:14 AM
... and we're back. There better be one good RFO and decent SLA credit in store here. I am not happy with Ubiquity at the moment, and I never thought I would say that. Sigh.
Posted by jayglate, 07-25-2011, 02:28 AM
Where did they move to and from?
Posted by UbiquityAllen, 07-25-2011, 08:44 AM
Hello Patrick, The full announcement and updates were found here: https://support.ubiquityhosting.com/...ed-maintenance We posted this on July 1st to give enough time for everyone to prepare for it. We initially stated in the article: We estimate that the process will take approximately 6 to 8 hours to complete, although additional time may be required. --I do know we updated this once stating we need to extend the window by an additional 8 hours. Although that had to be extended for some other reasons. For exact reasoning however(I was not actually at the data center performing the move and so forth), feel free to submit a ticket to our support center here, https://support.ubiquityhosting.com/, anytime, and we can have it reviewed for you as well as an SLA request if you have not already done so. Last night we experienced a large scale DDoS attack that was mitigated so I apologize for the issues, but the data center move should be completed at this time, and we're working on getting all machines fully operational that are still having issues, which is minimal. XO Communications to CoreXchange Thanks!
Posted by Patrick, 07-25-2011, 09:05 AM
I know you guys stated 6 to 8 hours, but I assumed that was for the entire operation from start to finish and I quote your announcement: "The total amount of downtime experienced by each server will vary based on its location in our current facility, but we expect that most systems will be down for no more than 2 to 3 hours." Did you mean that the servers would be powered off for 2 to 3 hours but then the network would take another 8+ hours to come back online which kind of makes that 2 to 3 hours window pointless? If that's the case, then that's a rather disingenuous way of wording the announcement. We were down for almost 20 hours, give or take an hour. I forget how long exactly, but it was definitely above 16 hours. I felt bad starting this thread, you guys are an exceptional provider when things are going great. Tickets answered in minutes, helpful answers at that, non of that canned BS replies some other data centres use. However, I still feel like you guys dropped the ball big time on this one in the communications department. In a 18 or 20 hour outage, one update that only says it's going to be another 8 hours isn't very helpful or reassuring to clients. I'm sure you guys were running around trying to get everyone back online, but a quick update of why things are running late would have been nice.
Posted by Sir Stu, 08-11-2011, 05:19 AM
I wasn't too impressed with this either. I don't think they ever actually emailed me about it - just posted it on their support portal. I only go there to file tickets, so if nothing is wrong and I don't need to order anything, I don't go there. I was therefore unaware it was going to happen until just a couple of days before. You would think it would occur to them to send out emails...? Service at this location has been crappy as well. Their sales guys are awesome, and their techs do a good job... sort of. There have been performance issues with Dallas for months. The old location was marginal. The new location is better, but still nowhere close to acceptable. Our server seems to drop off the net completely for ten minutes every morning, for no apparent reason, and there is often terrible latency. Ubiquity's techs seem generally unaware, and they don't seem to be getting the hint that there are real, serious problems with that location. Is this because the network infrastructure in Dallas is crap? Or maybe the local talent pool is on the shallow side? I don't know for sure. I have heard stories of bad performance from people who were hosted in other data centers in that city, and they seem to have problems like these on a regular basis. Whatever it is, I'm getting well and truly sick of it. So are my customers. I get two or three tickets every single week about lag and complete dropouts, and MRTG shows that the server sometimes just vanishes off the internet for five or ten minutes at a time. I also have servers from Ubiquity in two other cities (New York and Seattle) and we have never, ever got any complaints about latency in either place - not ONE! So what's with Dallas? Well, maybe someone cares, but that someone is no longer me. I'm pulling out by the end of the month. Ubiquity has otherwise done a reasonably good job. I like working with them. It's a shame that Dallas is such a failure vortex.
Posted by UbiquityAllen, 08-11-2011, 05:39 AM
We post the news and you would get the email if you subscribe to support.ubiquityhosting.com which we recommend doing. That's where we keep all of our news, maintenance, and outage reports at. As of right now, there are no issues outstanding. In the event that issues are brought to our attention, and networking issues, we resolve the issues as quickly as possible. We do monitor our network 24/7 as well so if there are issues, our networking team is normally already working on them. If you have ticket IDs or anything we can look into, forward them over to us and management and we'll be glad to take a look. I am sorry to hear you're pulling your stuff out however by the end of the month as well.
Posted by Sir Stu, 08-11-2011, 06:09 AM
I'll include some ticket numbers in the cancellation ticket. To be fair, I think a lot of this is just who your upstream providers are peering with. Take Global Exchange (GLBX) for example. They have a router that likes to drop traffic like it's gone out of style, but they are too busy being generic corporate tools to listen to anyone who isn't their direct customer. I wish none of my customers routed through them, but unfortunately a lot of them do. Again - I like you guys a lot. Your service in NY and Seattle is outstanding, and for all the tickets we get about Dallas every week, we get none at all about the other cities. But whether it's your fault or not, Dallas has been a pig for the last month and a half, and we're done asking our customers for pathpings.