Continuum DC Unplanned Maintenance?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Continuum DC Unplanned Maintenance?

Posted by fraghost, 01-18-2011, 10:03 PM
Email warning us about power failure during unplanned maintenance sent at 7:46PM CST.

Lost power to my services around 7:50PM CST. Couldn't have given us more time to shut down my services?

Very annoying and inconvenient, but I still love Continuum. I'll be extremely upset if there is any damage to my system.
Posted by Packhead, 01-18-2011, 10:08 PM
Well what's super annoying is I didn't receive any email. I was working in my vSphere client and suddenly VMs started shutting themselves down. Then the host went offline. Opened a 911 ticket and almost immediately got a response back about power issues:

---------
support@continuumdatacenters.com to me
show details 7:53 PM (13 minutes ago)

Dear Customer,

We are currently experiencing a power interruption at the data center. You may
experience an interruption of service. We are working as fast as we can to
ensure that we do not experience hard downtime. A problem with the generator
which is currently being worked on may cause us to lose power. In order to
protect your data, we suggest that you power your server(s) off until we notify
you that there is no longer a threat to power loss. If you have any questions
please feel free to give us a call at ----.

Regards,
Justin Chatfield
Continuum Data Centers LLC

--------------------

I appreciate that they (apparently) pressed the power button on my server to have it soft shut down everything, but I really would've liked to notify my customers before their servers went down.

Needless to say, this is the first outage Continuum has had in a looooong time. Although, this is like their third generator issue. They need to get somebody in there to fix this thing for good. Why don't they have a transfer switch to get them back onto another UPS or something while they do maintenance? Oh well -- as mentioned above, still very VERY happy with Continuum -- moving more equipment there soon. They just need to work on their power issues, and stat!
Posted by Datacenter1, 01-18-2011, 10:16 PM
I still up, but I already shutdown all servers except a couple
Posted by fraghost, 01-18-2011, 10:17 PM
I would have also liked some more time to warn my clients. Hopefully this doesn't last much longer. I agree they need a better plan to avoid such an issue... doesn't make much sense.

Continuum is a great company, we'll see how they handle this issue. =)

Looks like it would be a good idea to move my client panel, email and main site to a Ubiquity VPS.
Posted by scooby2, 01-18-2011, 10:42 PM
Why would a problem with the generator cause the whole datacenter to lose power? I could see a problem with building power or the transfer switch causing problems if the generator is broke, but the generator should not be in line unless there is a power outage, correct?
Posted by tnhomestead, 01-18-2011, 11:06 PM
Thats what I dont understand -- why were they on generator power in the first place? On the other hand, great service so far -- will be staying.
Posted by Packhead, 01-18-2011, 11:14 PM
My servers are back online. I'm waiting for confirmation from Continuum that it's safe to deem them ready to go back to production, though. Don't want to ramp everything up only to have to take it down again.
Posted by fraghost, 01-18-2011, 11:20 PM
Mine are still offline.
Posted by fraghost, 01-18-2011, 11:24 PM
Back online. Have you got a response back from them about stability?
Posted by drmorley, 01-18-2011, 11:26 PM
I just got this email:

The generator is currently stable and it is safe to return power to your machines. Let me know if you need any power buttons pressed.

Dylan
Posted by scooby2, 01-18-2011, 11:34 PM
Still offline here.
Posted by fraghost, 01-18-2011, 11:42 PM
Just got a confirmation email saying the generators are back online and that they will keep the backup gen on standby. If you're still offline, open a ticket. You probably need your power button pressed.
Posted by Ceetoe, 01-18-2011, 11:44 PM
Just got the all client e-mail as well. I'm back in service.

Dear client,

We have begun powering clients back on. The loss of phase has now been fully resolved and generator is operating normally. We will continue to keep backup power generator available until the issue has been cleared by our on site engineers. Please submit a support ticket if you are still unable to access your devices, or need them powered on. Due to a high volume of calls, the best way to reach the support desk at this time is by opening a ticket, or sending an e-mail to support@continuumdatacenters.com.
Posted by tnhomestead, 01-19-2011, 12:23 AM
Im fully up and running!
Posted by MACscr, 01-19-2011, 02:08 AM
So again, whats the story with the generators being needed in the first place? Do they even have UPS's in place, if so, what is their expected lasting time? The way they word things, its almost as if they dont normally have a generator onsite and ready to be switched on at any moment. As already mentioned, this isnt their first power issues where their generators failed. Do they not do any testing?

Yes, I have been happy with their service, but these type of outages are starting to make their planning and infrastructure look a bit unprofessional.
Posted by cyberhouse, 01-19-2011, 06:23 AM
the whole datacenter was not down my racks have been up all day i have seen no downtime.
Posted by proclanh, 01-19-2011, 02:01 PM
The issue was they lost power from the electric company to the building. This made the DC run on battery backup for about 10-15 minutes until the Generator kicked in. Once that started they realized the Generator was not producing enough power so they asked some clients including myself to shut down any unnecessary services. They ordered a backup Generator and we were able to restore power our servers within 2 hours. Not sure about the power loss from outside or with the Generator but it does seem like an issue that needs to be addressed. 2 hour downtime is not good even if we shut our servers down manually. Other than that no complaints here.
Posted by MACscr, 01-19-2011, 03:47 PM

Quote:

Originally Posted by proclanh

The issue was they lost power from the electric company to the building. This made the DC run on battery backup for about 10-15 minutes until the Generator kicked in. Once that started they realized the Generator was not producing enough power so they asked some clients including myself to shut down any unnecessary services. They ordered a backup Generator and we were able to restore power our servers within 2 hours. Not sure about the power loss from outside or with the Generator but it does seem like an issue that needs to be addressed. 2 hour downtime is not good even if we shut our servers down manually. Other than that no complaints here.

The generator not being able to produce enough power (its not like they dont know how much power their infrastructure is using at all times) just comes down to poor planning. Plus they should have redundant generators anyway that could be used to product the extra power if needed.
Posted by Continuum, 01-19-2011, 04:19 PM
All,

This wasn't a load issue (e.g. not able to produce enough power) or poor load planning.

ComEd had a widespread loss of phase in our area last night. Loss of phase is considered a loss of line power and we fail over to UPS. We ride UPS power for approximately 8 seconds while the generator starts and comes up to frequency. At no time did we have power loss to any equipment from a UPS or generator failure. Both the generator and UPS were providing power (with more than adequate capacity) to the facility and all critical loads. Although the generator was providing power to the facility, it was not being handled within optimal frequency, this causes the UPS to transfer load back and forth from UPS to generator (while this happens battery capacity is diminished thus continuing to reduce battery run-time). We shut certain breakers down manually, or requested some customers to shed load to non-critical equipment voluntarily, only as a precaution, while we were troubleshooting the issue.

A second generator is on-site as a backup and available to handle a full building load if necessary.

Again, I apologize for the inconvenience.

-Tom
Posted by MACscr, 01-19-2011, 04:44 PM

Quote:

Originally Posted by Continuum

All,

This wasn't a load issue (e.g. not able to produce enough power) or poor load planning.

ComEd had a widespread loss of phase in our area last night. Loss of phase is considered a loss of line power and we fail over to UPS. We ride UPS power for approximately 8 seconds while the generator starts and comes up to frequency. At no time did we have power loss to any equipment from a UPS or generator failure. Both the generator and UPS were providing power (with more than adequate capacity) to the facility and all critical loads. Although the generator was providing power to the facility, it was not being handled within optimal frequency, this causes the UPS to transfer load back and forth from UPS to generator (while this happens battery capacity is diminished thus continuing to reduce battery run-time). We shut certain breakers down manually, or requested some customers to shed load to non-critical equipment voluntarily, only as a precaution, while we were troubleshooting the issue.

A second generator is on-site as a backup and available to handle a full building load if necessary.

Again, I apologize for the inconvenience.

-Tom

Ok, I am confused by that statement. If my servers lost power without any action on my part, how can you consider that your backup power process worked? Especially when this is the 3rd time in the last year, with close to the same type of scenario as well.

Believe me, overall I am a happy Continuum customer and would still recommend you to people, but that doesn't mean I don't see room for improvement, nor not feel the need to call things as I see them.
Posted by fraghost, 01-19-2011, 04:44 PM
Has this issue been resolved to prevent future issues?

It would have been nice if I could have shut my own server down, and properly. From the time I got the email to the time I attempted to connect to my services, it was already shutdown. Only 4 minutes went by before I could act. I wanted to send an email to my clients to let them know and to shut down myself.

I accept your apology, of course, because Continuum is awesome, but hopefully things get handled differently next time. I know that my main site and email server will be hosted elsewhere to prevent this from happening. Stupid on my part I guess.
Posted by Continuum, 01-19-2011, 04:51 PM
Feel free to contact me directly at tom@continuumdatacenters.com if you have any further questions, concerns or suggestions.