Pacificrack/OC3 gone?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Pacificrack/OC3 gone?

Posted by andren, 01-27-2011, 06:14 AM
We have our servers at OC3/Pacificrack down for about 10 minutes. Anyone else?
Posted by Mikie4648, 01-27-2011, 06:18 AM
Yep, i just posted about it. Me too
Posted by anthoniech, 01-27-2011, 06:21 AM
Me too, all my sites are down, but https://manage.quadranet.com/ back up now.
Posted by cakehouse, 01-27-2011, 06:24 AM
Down since 1:59am PST for me.
Posted by ItsChrisG, 01-27-2011, 06:25 AM
Their is a power outage occurring right now from the LADWP utility feed. We are running on generator power. One of the UPS units in the LAX9 datacenter is down for currently unknown reasons and is being investigated by on-site technicians as we speak. Only circuits/racks on this UPS are affected. Circuits on the other UPS's in LAX9 are unaffected. All other datacenters on LAX3 (1x), LAX6 (1x), LAX8 (3x), LAX13 (2x) are unaffected and currently online with no interruption and running on generator power. Our customer management portal is not located in the affected area and has not gone down. Our website server is hosted on the affected UPS unit and is currently inaccessible.
Posted by cycomhk, 01-27-2011, 06:32 AM
Isn't this a little bit exaggerated? 10mn of downtime and a host can be suspected to be gone?
Posted by ItsChrisG, 01-27-2011, 06:32 AM
The affected UPS unit has been back online for about 10 minutes now. Any affected circuits should already be back online. Servers will be coming back on their own. If your server is still down in 10 minutes, please open or re-open your support ticket and be specific about which server is down noting the hostname and/or the device ID.
Posted by Mikie4648, 01-27-2011, 06:33 AM
I thought generators are tested everyone month!!!!
Posted by Mikie4648, 01-27-2011, 06:37 AM
It went down at 1;59am PST and came back up more than 20min later. More than 10min isnt it?
Posted by Ed-Freethought, 01-27-2011, 06:44 AM
Their generators worked fine by the sounds of it, but generators don't start instantly, they need 30 seconds or so to come up to speed and start outputting a stable sine wave which will then cause the ATS to switch from utility to generator power. The UPS is needed to provide ride through power whilst the generator comes up to speed, so if the UPS fails then there is an interruption in power. Additionally, if a UPS fails then it is an in-line device between the utility power or generators (via an Automatic Transfer Switch) and the internal power distribution infrastructure. You can run UPS units in N+1 configurations or switch a UPS into bypass mode to try and mitigate such failure, but no setup is perfect and there are still potentials for outages depending on all sorts of scenarios. It takes time to investigate why a UPS isn't working and you have to be careful - these units contain could easily kill you if you don't know what you are doing. You also don't want to switch the UPS into bypass mode if there is a problem on the raw feed side and end up dumping unstable power surge into into your servers, which could do more damage.
Posted by Mikie4648, 01-27-2011, 06:46 AM
Thanks for the technical overview. Its back up thats what important. Hopfully this blackout doesnt continue into tomorrow.
Posted by Ed-Freethought, 01-27-2011, 06:52 AM
No problem, happy to help