Ubiquity Chicago down?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Ubiquity Chicago down?

Posted by adamnp, 05-13-2013, 05:39 PM
Curious if their website/support panel, and networks is down for others?

I've been unable to reach anything now for 55 minutes, and had issues 4 hours ago also.

Anyone from ubiquity able to comment?
Posted by RS|John, 05-13-2013, 05:43 PM
I have been down since 4:48 eastern. Almost an hour now. Their website seems down as well. No idea what is going on. This doesn't seem good/normal at all.
Posted by adamnp, 05-13-2013, 05:48 PM
Same -- I was hoping I was mistaking... But after checking from a bunch of other peers, everything looks dead in Chicago atm.

Appears they are posting on Twitter:

https://twitter.com/ubiquityservers/
Posted by Zimple, 05-13-2013, 05:51 PM
on the twitter they said,

"We are aware of the networking issues at our Chicago data centers and we are working to resolve these issues as quickly as possible."
Posted by hungoverfurball, 05-13-2013, 06:20 PM
Also have several servers down, Sadly as a GSP downtime and latency issues are a bit of a bad thing. Already have had several refund requests and a few paypal disputes in this short amount of time :/
Posted by Zimple, 05-13-2013, 06:30 PM
Seems like it will take little more time to come up.

"Please bare with us, there seem to be some significant networking issues that are beyond Ubiquity's Chicago data center."
Posted by John Seena, 05-13-2013, 08:21 PM
Been down for almost 3 hours now. So not good for anyone.
Posted by RS|John, 05-13-2013, 08:22 PM
What could possibly be the cause of a 3 hour outage?
Posted by adamnp, 05-13-2013, 08:53 PM
Appears it is coming back now
Posted by RS|John, 05-13-2013, 09:12 PM
Our servers are back online
Posted by 24x7group, 05-14-2013, 03:36 AM
Does anyone already know an exact cause of this outage/downtime?
Posted by leckley, 05-14-2013, 09:13 AM

Quote:

Originally Posted by 24x7group

Does anyone already know an exact cause of this outage/downtime?

You will have a better chance getting blood from a stone than a real cause of the outage from Ubiquity.
Posted by Zimple, 05-14-2013, 12:18 PM

Quote:

Originally Posted by 24x7group

Does anyone already know an exact cause of this outage/downtime?

According to them

-----------------

Ubiquity Chicago experienced a limited service interruption at our Chicago 1 facility this afternoon. We have determined that the issue was caused by a problem with our point to point transport providers. The issue has since been resolved and at this time all services should be restored. If you experience any further issues, please contact us immediately by creating a support ticket at https://motion.ubiquityservers.com or calling 1-866-438-8247 or +1-480-212-1710 (International).

-------------------
Posted by leckley, 05-14-2013, 12:47 PM

Quote:

Originally Posted by mobilk

According to them

-----------------

Ubiquity Chicago experienced a limited service interruption at our Chicago 1 facility this afternoon. We have determined that the issue was caused by a problem with our point to point transport providers. The issue has since been resolved and at this time all services should be restored. If you experience any further issues, please contact us immediately by creating a support ticket at https://motion.ubiquityservers.com or calling 1-866-438-8247 or +1-480-212-1710 (International).

-------------------

Exactly my point...nothing like an RFO with no information.
Posted by fraghost, 05-14-2013, 02:47 PM
If you haven't already, request your SLA credit.

I was down for exactly 3 hours, so I imagine everyone else's downtime would be similar.
Posted by Smitheson, 05-14-2013, 04:10 PM
I also got my SLA credit.

Quote:

Originally Posted by fraghost

If you haven't already, request your SLA credit.

I was down for exactly 3 hours, so I imagine everyone else's downtime would be similar.

Posted by mousepad99, 06-15-2013, 08:33 AM
Ubiquity has got to be the worst dedicated service provider we have ever used.

At the moment, we've servers with the following providers:

Ubiquity
Voxel
Softlayer
Iweb
yes up host
Steadfast
Quadranet
and a few more I forgot.

None of these providers, joined together can come close in YEARS of services the issues we had with ubiquity in a few months.

One of our server in their Chicago facility experiences network issues every week, sometimes twice a week. What good is a SLA REFUND of a few dollars when you lose business?

We're slowly migrating our services aways. We have servers in three of their facilities, we spend 12,000-15,000 USD a year with them and we get a ****** service.

Right now my ticket is still unanswered and one of our production server is experiencing over 50% of packet loss.

Terrible network, terrible support, no RFO, Scheduled maintenance without email warnings etc..

I once had to argue for 10 hours straight with a tech that would constantly tell me that our servers were self managed, so he could do nothing (The server hard drive failed). DEspite my explanations that I could not go on site to change the hard drive myself, because they are THEIR hardware, not colo, it took over 48 hours to get the hard drive replaced.

If you want, can also post my log of Nagios monitoring our chicago servers from Softlayer and Voxel.

On May 31th, they blamed their stability issues on their "appeal" instead of their lack of planning, from their Twitter:
"its fair to say we underestimated our product appeal & our infrastructure needs to expand to correct these growing pains. "

STAY OUT OF UBIQUITY
Posted by TravisJeffries, 06-17-2013, 01:03 AM
mousepad99 I'd like to see some logs.

We're at our wits end with Ubiquity. I've been talking to Chris S (Who has been a bunch of help by the way, he's one of the primary reasons we've stuck around) about these issues and he has said that the issues should be resolved with their upcoming upgrades, but it certainly hasn't been easy for the past month and I'm having serious doubts if the problems will be resolved.

We've been having serious packet loss every single day, multiple times a day. Support is way less than helpful. Just take this ticket for example:

http://i.imgur.com/W0k1MyQ.png

Here are some screenshots from pingplotter, for fun:
http://i.imgur.com/pl7Fcn9.png
http://i.imgur.com/q9GqxDU.png
http://i.imgur.com/em5wHt7.png

I felt like I was slightly rude in that ticket, but keep in mind I've had to hold my composure throughout several instances of this. This is an ongoing problem with them. If the upgrades that are happening in their LA and CHI datacenters don't resolve the issues, we're done.
Posted by mousepad99, 06-17-2013, 01:18 AM
Travis,

We just had another network issue about 30 minutes before you posted this.

Yesterday we had over 50% packet loss for over 5 hours.

Today we lost the network about 5-6 times, average 10 minutes each downtime. We run voice services, you can imagine the nightmare it causes to us. We've already started to migrate away.

If at least there was a little bit of communication, there's nothing.

Yesterday (Approx):
Heavy packet loss from 4:30 AM to 10:30 AM (5-6 hours down time, they blamed it on a widespread DoS affecting various of their data centers)

Today, 5 to 15t minutes failures:
7h52 AM (5-10 Minutes)
8h20 AM (5-10 minutes)
10h38 AM (5-10 minutes)
6H52 PM (5-10 minutes)
11h18 PM (30 minutes)

Also, our seattle equipment failed at 4h50 am for about 5-10 minutes as well.

AND this is nagios, this not a ping tool, nagios will report only after at least 3 failures to check a service on a specific port.

All of these outages, have been double checked by our staff and we ping from various locations when we get an alert.

My sysadmins are going insane, and our reputation been affected badly with our chicago customer.

We're moving our Chicago stuff to steadfast, our seattle stuff to Softlayer and oru Atlanta stuff to GNAX / Net Depot. All of these 3 locations we have them with Ubiquity right now. The worst experience is with Chicago, which is a nightmare, followed by Seattle. Atlanta rarely had issues but I guess not many of their customers are choosing that data center.
Posted by TravisJeffries, 06-17-2013, 01:23 AM
I am very sorry to hear that mousepad99 and understand your frustration completely. While our game servers that we provide may not be on the level of importance as your voice servers, we certainly are impacted in a big way when the network drops out or big chunks of packet loss hits. We lose all of our players on the servers that we work all day to fill, instantly. Tonight for example we are not even able to refill all of our servers.

It's making our community and community servers look really bad. They have one last shot to rectify this with their upgrades, of which they've tweeted a few photos of. If these issues still remain then we'll be done with them.

We planned to setup nagios to monitor our servers as well, which is why I was curious about those logs.

Here's a snapshot of our gmail with all of the downchecks going into it.

http://i.imgur.com/AdW5civ.png

EDIT: And JUST as I type this post up... here we go:

http://i.imgur.com/C8q8jyJ.png

Just long enough to completely clear out a few of the servers we were able to recover after they were cleared out before.
Posted by cooldude919, 06-17-2013, 08:48 AM

Quote:

Originally Posted by mousepad99

Travis,

We just had another network issue about 30 minutes before you posted this.

Yesterday we had over 50% packet loss for over 5 hours.

Today we lost the network about 5-6 times, average 10 minutes each downtime. We run voice services, you can imagine the nightmare it causes to us. We've already started to migrate away.

If at least there was a little bit of communication, there's nothing.

Yesterday (Approx):
Heavy packet loss from 4:30 AM to 10:30 AM (5-6 hours down time, they blamed it on a widespread DoS affecting various of their data centers)

Today, 5 to 15t minutes failures:
7h52 AM (5-10 Minutes)
8h20 AM (5-10 minutes)
10h38 AM (5-10 minutes)
6H52 PM (5-10 minutes)
11h18 PM (30 minutes)

Also, our seattle equipment failed at 4h50 am for about 5-10 minutes as well.

AND this is nagios, this not a ping tool, nagios will report only after at least 3 failures to check a service on a specific port.

All of these outages, have been double checked by our staff and we ping from various locations when we get an alert.

My sysadmins are going insane, and our reputation been affected badly with our chicago customer.

We're moving our Chicago stuff to steadfast, our seattle stuff to Softlayer and oru Atlanta stuff to GNAX / Net Depot. All of these 3 locations we have them with Ubiquity right now. The worst experience is with Chicago, which is a nightmare, followed by Seattle. Atlanta rarely had issues but I guess not many of their customers are choosing that data center.

Small world, but it would seem I am actually one of your customers. Caused me some issues, even after the notes said it was fixed my inbound DID wouldn't work even though I reset everything. Ended up moving to Dallas and everything started working again. Latency is only a little higher to Dallas so seems I am best off staying there until your move is complete.
Posted by John Seena, 06-25-2013, 07:51 AM
Yet another outage with their chicago data center. We have been completely down for for 7+ hours. All ubiquity says is 'they will update us shorty'. 7 hours later and they have no more details?!

But hey, good news, they have a few deals posted in the advert section.
Posted by oldsmoboi, 06-25-2013, 08:56 AM
Is their Chicago data center in Lombard IL?
Posted by NYCServers-Nick, 06-25-2013, 09:00 AM

Quote:

Originally Posted by oldsmoboi

Is their Chicago data center in Lombard IL?

No, according to their site it's located at 1808 Swift Drive in Oakbrook, IL.

https://www.ubiquityservers.com/chicago-data-center
Posted by oldsmoboi, 06-25-2013, 09:03 AM
That is so close to Lombard IL (often stay in Lombard when working in Oakbrook) that it is possible they were hit by the same power outage that hit a Lombard Data center where my Cloudive host is?

I've been out since about 7pm eastern yesterday.
Posted by SethPaynee, 06-25-2013, 09:07 AM
intereting
Posted by John Seena, 06-25-2013, 09:17 AM
We are still down. Nothing but generic 'cut and paste' replies with no end in site. No reason, no estimated time, no nothing.
Posted by John Seena, 06-25-2013, 09:24 AM

Quote:

Originally Posted by oldsmoboi

That is so close to Lombard IL (often stay in Lombard when working in Oakbrook) that it is possible they were hit by the same power outage that hit a Lombard Data center where my Cloudive host is?

I've been out since about 7pm eastern yesterday.

Yikes, 14 hours downtime?? And here i am furious with 8 hours of no websites. What are they telling you about what they are doing to fix it??
Posted by oldsmoboi, 06-25-2013, 09:27 AM
My only updates come from my "Did Cloudive Evaporate?" thread in this forum. I'd link you, but I only have 4 posts.
Posted by TravisJeffries, 06-25-2013, 11:40 AM
5 straight minutes of 30%+ packet loss.

http://i.imgur.com/6IOFQxU.png

I would normally be a little sympathetic because of the storm that supposedly knocked out power? It hasn't been storming for a month straight though.
Posted by PersonalJ, 06-25-2013, 11:51 AM

Quote:

Originally Posted by NYCServers-Nick

No, according to their site it's located at 1808 Swift Drive in Oakbrook, IL.

https://www.ubiquityservers.com/chicago-data-center

Their hardware has been moved to Lombard into Continuum's facility. Hardware that was in their other facilities have been migrated physically.
Posted by oldsmoboi, 06-25-2013, 11:56 AM
Ubiquity is in the same DC that I'm in then.
Posted by phinsup, 06-25-2013, 12:02 PM
Must have been some major issues in Chicago, we received emails from two providers that they were down (gigenet cloud and backupsy) in chicago.
Posted by oldsmoboi, 06-25-2013, 12:03 PM
Backupsy is the same company as Cloudive.
Posted by smithc, 06-25-2013, 12:35 PM
Hi all,

To clarify any misunderstanding – between now and through the remainder of this month, we are undergoing a scheduled maintenance and migration in Chicago. Equipment is being moved into the Continuum facility, however Ubiquity is operating and maintaining its own independent network (completely separate from Continuum’s).

Initial notice was provided at the start of this month and updates have been provided as well. All affected clients were notified and provided an estimated amount of time whereby services would be temporarily unavailable, as well as an opportunity to coordinate any special considerations with our migration team. The migration is well underway and all necessary measures are being taken to limit the extent of this impact and restore services to all clients as quickly as possible.

Efforts were made to contact all affected clients; any clients who believe they did not receive notification (or have any questions or concerns) are encouraged to contact our support team either by opening a ticket from their client control panel, or by emailing support@ubiquityservers.com

Thank you!

Christopher Smith
Ubiquity Servers
Posted by cyansmoker, 06-25-2013, 12:46 PM

Quote:

Originally Posted by oldsmoboi

My only updates come from my "Did Cloudive Evaporate?" thread in this forum. I'd link you, but I only have 4 posts.

http://www.webhostingtalk.com/showthread.php?t=1278876
Posted by mousepad99, 06-26-2013, 01:36 AM

Quote:

Originally Posted by smithc

Hi all,

To clarify any misunderstanding between now and through the remainder of this month, we are undergoing a scheduled maintenance and migration in Chicago. Equipment is being moved into the Continuum facility, however Ubiquity is operating and maintaining its own independent network (completely separate from Continuums).

Initial notice was provided at the start of this month and updates have been provided as well. All affected clients were notified and provided an estimated amount of time whereby services would be temporarily unavailable, as well as an opportunity to coordinate any special considerations with our migration team. The migration is well underway and all necessary measures are being taken to limit the extent of this impact and restore services to all clients as quickly as possible.

Efforts were made to contact all affected clients; any clients who believe they did not receive notification (or have any questions or concerns) are encouraged to contact our support team either by opening a ticket from their client control panel, or by emailing support@ubiquityservers.com

Thank you!

Christopher Smith
Ubiquity Servers

This doesn't explain how the downtimes.

We personally never received any emails about any updates, and I doubt you moved our servers more than dozen times.

Tongith again, we have a server down, it's been 1 hour an a half so far. One of your tech said he would investigate, never replied again.

I logged in IPMI to find the power was turned off. We turned it back on, it went off again 15 minutes later, and this time the IPMI isn't reachable.

No updates on ticket either, nothing as usual.

I get it that you've lost control of your Chicago facilities but theres no excuse for the terrible level of support currently being offered.

Hopefully, we have ordered a bunch of servers at Steadfast (Over $1,200 USD per month), and we'll be migrating out soon.

You never post your issues publicly, never send any RFO or emails regarding downtimes, always have sporadic issues, including SEattle and Atlanta facilities as well.

Sorry but there's no "confusion", the only confusion is coming from inside your own company.
Posted by smithc, 06-26-2013, 09:05 AM
Hi mousepad99,

I appreciate the feedback and I'd like to investigate this. Please feel free to PM me a ticket number to reference, alternatively you can email me directly at chris.smith[at]ubiquityservers.com

Thank you

Christopher Smith
Ubiquity Servers
Posted by TravisJeffries, 06-26-2013, 11:19 AM
I give Chris (smithc) a lot of props because I know from his end he can only do so much to correct the issues. Like I've said before he is pretty much the only reason why we've stuck around this long.

This morning we've gotten hit with 3 huge bouts of packet loss. The most recent one as of a few minutes ago:

http://i.imgur.com/crgfkha.png

I must say, I'd have never thought service would be this bad with Ubiquity. We used to be hosted out of their NY datacenter and very rarely had issues. Chicago is just issue after issue after issue. We've also pretty much given up on tickets because after we provide the requested pathpings/mtr we're usually just told the issue is resolved with no further info.

Jay is quite bad about this, actually.

http://i.imgur.com/I0ez2Lq.png

Please keep in mind that we've been very cordial throughout most of this but it has gotten to the point where it just feels like we're running around in circles. As this week nears an end and the upgrades along with it I really wonder if these packet loss issues we deal with on a daily basis are going to persist. Because we really do not see an improvement so far.
Posted by TravisJeffries, 06-26-2013, 02:09 PM
A few minutes of 100% inaccessibility. (100% packet loss)

http://i.imgur.com/qLntIjf.png

And for fun, here are all the downchecks that have come in from the ubiquity locations to our Chicago system using their panel ping monitoring service.

http://i.imgur.com/GlQAAEj.png

Check the top right. 104 alerts. That's in the span of about a month. So we're talking a bit over 3 downtimes per day. To be fair, this check is setup at 10% packet loss but the majority of the time it is much higher than that. The average loss is about 30% and most of the times it's higher, 70% to 100%. We just now lost 200 active clients on all of our servers.

http://i.imgur.com/NXlQoje.png
Posted by liston13, 06-26-2013, 02:32 PM
Yes, indeed they are. I have been down since 1am Tuesday morning and they are no longer responding to our emails or requests.
They have also repeatedly refused to provide me with a supervisors email or contact information.

I have contacted the State Attorney General's, Lisa Madigan, office as well as Gary Shapiro the US Attorney of Northern Illinois.
Posted by EvolutionCrazy, 06-26-2013, 02:54 PM
my server is down, they are not responding to enquires to me either.

what's wrong with them?
Posted by smithc, 06-26-2013, 03:27 PM

Quote:

Originally Posted by liston13

Yes, indeed they are. I have been down since 1am Tuesday morning and they are no longer responding to our emails or requests.
They have also repeatedly refused to provide me with a supervisors email or contact information.

I have contacted the State Attorney General's, Lisa Madigan, office as well as Gary Shapiro the US Attorney of Northern Illinois.

liston13, please PM or email me (chris.smith[at]ubiquityservers.com) a ticket number, I will address this promptly.

Quote:

Originally Posted by EvolutionCrazy

my server is down, they are not responding to enquires to me either.

what's wrong with them?

EvolutionCrazy, also please PM or email me (chris.smith[at]ubiquityservers.com) a ticket number, I will address this promptly.

Thank you both

Christopher Smith
Ubiquity Servers
Posted by EvolutionCrazy, 06-26-2013, 03:42 PM

Quote:

Originally Posted by smithc

EvolutionCrazy, also please PM or email me (chris.smith[at]ubiquityservers.com) a ticket number, I will address this promptly.

Thank you both

Christopher Smith
Ubiquity Servers

Thanks Christopher for sorting this out
Server is up again and working fine now.
Posted by TravisJeffries, 06-27-2013, 12:07 PM
Another day, another round of extreme packet loss. Not incredibly long this time, but 70-100% packet loss is enough to wipe all of the active connections we have out.

http://i.imgur.com/0eREw4t.png

Our servers were cleared of ~150 players. I'd also like to mention that our community is having a huge dent put into it by the unreliability of our servers on a daily basis.
Posted by TravisJeffries, 06-28-2013, 12:09 AM
Wiping out our servers right as we speak.

http://i.imgur.com/MD9yqMx.png

So much for upgrades. I feared they wouldn't solve the problem.
Posted by inteltechs, 06-29-2013, 01:31 AM
one of our servers is still down....
Posted by smithc, 06-29-2013, 03:30 PM
Hi inteltechs - please PM or email me (chris.smith[at]ubiquityservers.com) a ticket number, I will address this promptly.

Thank you!

Christopher Smith
Ubiquity Servers
Posted by inteltechs, 06-30-2013, 01:57 PM
what is the plan Chris? Our server is still down and I have no word from you guys.
Posted by smithc, 07-01-2013, 01:59 PM
Hi all,

There were definitely some unforeseen obstacles with this Chicago facility migration and maintenance. Fortunately, it has been completed and services have been restored (excluding a small handful of clients we are still working with). All clients are now in our new facility and on our improved Chicago network.

Any clients who are experiencing any issues or have any questions are encouraged to contact our support team by opening a ticket, or emailing support (support[at]ubiquityservers.com).

Additionally, please feel free to reach out to me directly (either send me a PM, or email me directly: chris.smith[at]ubiquityservers.com) if you have a ticket open or any feedback.

Thank you

Christopher Smith
Ubiquity Servers