Netriplex Power Outage

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Netriplex Power Outage

Posted by WebManagerNY, 05-12-2011, 06:45 PM
Netriplex Power Outage @ 5/12/2011 5:36 PM.

Power appears to have been restored.

As a result of this power outage, I now have database corruption on primary database powering my ecommerce site. It is going to a late night...
Posted by WebManagerNY, 05-12-2011, 08:38 PM
Just received the following updates:

Thu, May 12, 2011 at 8:01 PM

Please be advised that the Netriplex AVL01 data center facility team has reported that the UPS on Power Bus 1 momentarily dropped the critical load during a utility power blip impacting some customers in our AVL01 datacenter.

Facilities management is currently investigating this issue and has called our UPS service & support vendor to come onsite to investigate this issue with our electrical team. Further information is not known at this time.

Thu, May 12, 2011 at 8:10 PM

This update is to inform you that our UPS emergency service vendor is onsite investigating the issue at this time. An estimated time of resolution is still unknown.

====================

So let's assess the current situation:

Power to my entire 1/2 rack takes a hit
I have data corruption galore on my primary DB server
According to the most recent notification from Netriplex, ETA for resolution to the onsite investigation into this power outage is unknown

Yet the Netriplex Current Network / Systems Status Page located at the following URL, http://www.netriplex.com/support/index.aspx, reads as follows:

As of 5/12/2011 8:20:40 PM Eastern, there are no known issues.
Posted by RyanD, 05-12-2011, 09:24 PM
thats pretty impressive, it took them 9 minutes to get a service technician on site? I guess they are using transporters these days, that or the timestamps are off
Posted by gordonrp, 05-12-2011, 09:33 PM
Were you affected by the power outage a couple of months ago too? 2 outages in 2 or 3 months isn't good. Fortunately we weren't affected either time it happened.
Posted by Mike12King, 05-12-2011, 09:39 PM
We're still having problems with this outage...

Is it really all online?
Posted by WebManagerNY, 05-13-2011, 10:20 AM

Quote:

Originally Posted by gordonrp

Were you affected by the power outage a couple of months ago too?

Yes, unfortunately my rack was affected by all of the following recent Netriplex outages:

Power Failure on May 12, 2011
Power Failure on Feb 24th, 2011
Internet Outage December 12th, 2010
Internet Outage October 4th, 2010

Quote:

Originally Posted by gordonrp

2 outages in 2 or 3 months isn't good.

Yes, I would agree. However, the whopping $39.67 SLA credit I received for the Feb 24th Power Failure more than makes up for the inconvenience of the power being pulled on my entire half rack of Dell 1955 Blades Servers & EMC Clariion Disk Array --- WOULDN'T YOU SAY?

Yet other power outage last night... this time massive data corruption...
Posted by boomertsfx, 05-13-2011, 10:23 AM
I've had a 1U colo with them for 3 years now -- never had any issues until this year with the 2 UPS outages, and an occasional routing issue. Other than that, rock solid. Of course, it's just my personal server, but it's never been down for that long (maybe an hour or two in 3 years)...
Posted by Jay Suds, 05-13-2011, 01:13 PM

Quote:

Originally Posted by WebManagerNY

Yet other power outage last night... this time massive data corruption...

A+B power FTW ...
Posted by WebManagerNY, 05-13-2011, 01:37 PM

Quote:

Originally Posted by Jay Suds

A+B power FTW ...

20/20 hindsight
Posted by ishan, 05-14-2011, 03:26 PM

Quote:

Originally Posted by RyanD

thats pretty impressive, it took them 9 minutes to get a service technician on site? I guess they are using transporters these days, that or the timestamps are off

They later emailed saying the email timestamps were incorrect or something similar, I was too sleepy to read that part carefully.

Quote:

Originally Posted by gordonrp

Were you affected by the power outage a couple of months ago too? 2 outages in 2 or 3 months isn't good. Fortunately we weren't affected either time it happened.

We were affected by it too Twice in 5 months is too much. Fortunately we did not suffer any corruption/data loss.
Posted by CGotzmann, 05-14-2011, 11:45 PM
As mentioned above, this is why A+B power exists. 1 feed will never have 100% uptime, just like single homing to any 1 network provider.
I'm sure they also hate that they have had these downtimes, but sometimes accidents just happen in a timeframe that is close to each other.
Posted by Dougy, 05-15-2011, 11:11 AM

Quote:

Originally Posted by WebManagerNY

20/20 hindsight

if you have big important servers, why on earth would you just have A?
Posted by UH-Bobby, 05-15-2011, 12:50 PM
We were unaffected by this. I was unaware there was even a problem. Are you on UPS backed power? If not, why don't you invest in an APC UPS?
Posted by gordonrp, 05-15-2011, 03:47 PM
Why do people bash the OP's purchasing choices? Obviously he/she knows about dual feeds. However even on a single feed multiple failures of the same piece of equipment within a few months is not to be expected if/when the first outage is correctly analyzed. Obviously a failure here beyond equipment.

Yes, yes, dual power is the best. No one is arguing that.
Posted by WebManagerNY, 05-15-2011, 04:31 PM

Quote:

Originally Posted by Dougy

if you have big important servers, why on earth would you just have A?

Yes, I agree I am at fault here. I relied too much on the Netriplex marketing material which reads, "100% Uptime - Netriplex AVL 01 is designed for one hundred percent uptime of power, cooling and internet connectivity, something that's very hard in the industry where four or five nines is the standard".

I need to plan another trip down to asheville and move to a A + B setup.
Posted by ishan, 05-15-2011, 04:32 PM

Quote:

Originally Posted by WebManagerNY

Yes, I agree I am at fault here. I relied too much on the Netriplex marketing material which reads, "100% Uptime - Netriplex AVL 01 is designed for one hundred percent uptime of power, cooling and internet connectivity, something that's very hard in the industry where four or five nines is the standard".

I need to plan another trip down to asheville and move to a A + B setup.

Down we go again !
Posted by WebManagerNY, 05-15-2011, 04:35 PM

Quote:

Originally Posted by UH-Bobby

We were unaffected by this. I was unaware there was even a problem. Are you on UPS backed power? If not, why don't you invest in an APC UPS?

Because I'm paying Netriplex to provide a reliable UPS service for my 208v blade servers and EMC Clariion disk array. Further, I do not want to add another point of failure by putting in a cheap APC unit -- the Netriplex UPS solution is billed as being "industrial strength".
Posted by UH-Bobby, 05-15-2011, 04:51 PM

Quote:

Originally Posted by WebManagerNY

Yes, I agree I am at fault here. I relied too much on the Netriplex marketing material which reads, "100% Uptime - Netriplex AVL 01 is designed for one hundred percent uptime of power, cooling and internet connectivity, something that's very hard in the industry where four or five nines is the standard".

I need to plan another trip down to asheville and move to a A + B setup.

I think in that marketing material it says that 100% is only designed if you use A and B.
Posted by UH-Bobby, 05-15-2011, 04:52 PM

Quote:

Originally Posted by WebManagerNY

Because I'm paying Netriplex to provide a reliable UPS service for my 208v blade servers and EMC Clariion disk array. Further, I do not want to add another point of failure by putting in a cheap APC unit -- the Netriplex UPS solution is billed as being "industrial strength".

Fair point, but APC can be far from cheap. I've seen full racks of APC equipment that isn't cheap, but works awesome.