Major Paetec Outage in Conshohocken

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Major Paetec Outage in Conshohocken

Posted by devsys22, 08-16-2010, 01:43 PM
I have customers that I do hands-on work for (reboots, installs, etc.,.). I got a call at 4am this past Friday from one customer claiming that many servers were down. I went to the Paetec facility in Conshohocken, PA, and it was PACKED with customers fixing their servers.

Paetec has denied everything, and even lied to customers about the power failure. Once customer of mine just thought it was his server, but then he noticed the uptime results on other servers and realized that every single server was rebooted/lost power at the same time. Eventually, Paetec admitted that they did "maintenance" on the power equipment. Thats kind of funny actually.

In addition to being a power failure, there must have been an electrical spike of some kind, because several servers had their power supplies or memory destroyed. While I was there I noticed several other people with similar problems of mysteriously DEAD servers. Out of 2 full racks, we lost 2 machines.

As a SAS70 facility, they clearly did not meet the requirements of transparency as they did everything they could to cover up the issue.

There were alot of angry people there that Friday....
Posted by devsys22, 08-23-2010, 02:02 PM
Update... Attached is a PDF of Paetec's Service Disruption Notice.

Three days following the outage, they finally admitted there was a problem. Though in the hours following the outage, they did not admit to anything - causing me to troubleshoot my issue as if it were a problem with my server, only later did I find out it was an outage when all 40 of my servers had the same uptime, and when I went to facility, there were like 20 other customers there working on dead systems.

Paetec doesn't say much in the notice, sort of blaming it on a "utility brown-out". That still doesn't explain why the UPS failed! Or why it fried so many systems.

Besides, I know someone that works in the building next door, and they did not lose power, if it was a utility brown out, the whole neighborhood would have gone down.
Posted by zubrcom, 08-23-2010, 04:34 PM
Maybe they subscribed to the "cut your electric costs by accepting brown-outs" campaign from Peco?
Posted by CGotzmann, 08-23-2010, 06:04 PM
While power outages will happen at all datacenters, no matter the level of redundancies, the fact that they were denying anything was wrong during the critical hours of troubleshooting is ridiculous.
Posted by fastdeploy, 08-23-2010, 06:27 PM

Quote:

Originally Posted by devsys22

Paetec doesn't say much in the notice, sort of blaming it on a "utility brown-out". That still doesn't explain why the UPS failed! Or why it fried so many systems.

I've been through this before at another data center - a place that had never had any power outage in 5+ years of being in business - but they owned up to the issue right away. An electrician had done some UPS maintenance and apparently didn't bypass things properly and the entire DC lost power briefly.

I frankly can't comprehend the point of denying an outage - nothing could be more obvious, especially when a bunch of customers begin to materialize at the door all at about the same to fix all the machines that don't come up properly.
Posted by woods01, 08-23-2010, 10:27 PM
This seems to be pretty common now-a-days with datacenters having internal power fluctuation problems due to their internal systems not being properly regulated assuming nothing will go wrong on the 'inside'.

This happened recently with another company but I can't recall which one.

What more you expect out of PA hosting though, pennsylvania isn't exactly a major internet hub