Apparent widespread Power Failure at TATA/VSNL London

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Apparent widespread Power Failure at TATA/VSNL London

Posted by chrisbyrd, 11-26-2009, 01:41 PM
Seems to be affecting some of our equipment, C4L, ServerCity, Coreix and others...
Posted by IbnKuldun, 11-26-2009, 01:50 PM
Definitely affecting coreix, my machines are down and they are not answering support calls, probably because they are scrambling to fix this.
Posted by chrisbyrd, 11-26-2009, 02:08 PM
There is some suggestion of a wider power issue affecting the Stratford area of East London

(Yes, there are of course generators and UPS at this facility, but appear to have failed)
Posted by Mekhu, 11-26-2009, 02:09 PM
We've got servers over at OpenHosting in London and we have some online, some offline.

What sucks is I reported issues over the past few days now. Our monitoring has been picking up minor outages all week. And now.. BAM!
Posted by gd1970, 11-26-2009, 02:10 PM

Quote:

(Yes, there are of course generators and UPS at this facility, but appear to have failed)

Thats pretty damn shoddy if thats the case.

My rack (at coreix) is out.

How did you hear about the power outage (source ?)
Posted by Ascendo, 11-26-2009, 02:14 PM
Ironically, we chose to host there due to the flawless record ITO power outages.
Posted by IbnKuldun, 11-26-2009, 02:16 PM
Any idea why UPS failed too?
Posted by Mekhu, 11-26-2009, 02:17 PM

Quote:

Originally Posted by chrisbyrd

Seems to be affecting some of our equipment, C4L, ServerCity, Coreix and others...

Any word on an ETA or any further update? We're getting about 3-4 tickets per minute and we'd love to update them.
Posted by chrisbyrd, 11-26-2009, 02:20 PM

Quote:

Originally Posted by gd1970

How did you hear about the power outage (source ?)

Nothing concrete on the wider stratford problem - just a few twitter posts from people in the area

We have several passive optical circuits between the TATA/VSNL site and various other datacentres, all of which are presently down (ie, means power loss to our gear on the other end of it) - even though nobody has confirmed this officially yet.
Posted by Rus Foster, 11-26-2009, 02:25 PM
Looks like webhostchat (the UK equiv of WHT) has wandered off as well.
Posted by Mekhu, 11-26-2009, 02:25 PM

Quote:

Originally Posted by chrisbyrd

Nothing concrete on the wider stratford problem - just a few twitter posts from people in the area

We have several passive optical circuits between the TATA/VSNL site and various other datacentres, all of which are presently down (ie, means power loss to our gear on the other end of it) - even though nobody has confirmed this officially yet.

Thanks Chris. I really appreciate the updates.

Edit, for those of you who aren't big Twitter fans, here's a search of the word #outage. Shows a lot of discussion about this.

http://twitter.com/#search?q=%23outage
Posted by chrisbyrd, 11-26-2009, 02:36 PM
TATA Account Manager has confirmed power outage at the facility
Posted by Ascendo, 11-26-2009, 02:39 PM

Quote:

Originally Posted by chrisbyrd

TATA Account Manager has confirmed power outage at the facility

Thanks for the update...would appreciate it if you could keep updating as I'm getting nothing out of my ISP.

Do you know if the entire facility is out, or just parts of it?
Posted by geoffgarside, 11-26-2009, 02:41 PM
From what we've seen its affecting Stratford as a whole, so the whole facility is out
Posted by IbnKuldun, 11-26-2009, 02:41 PM
Ascendo, apparently its affecting the whole of stratford and quite a few data centers. Which can only mean that those huge petrol powered generators outside the data centers are a big con.
Posted by chrisbyrd, 11-26-2009, 02:44 PM
TATA just called me back - presently estimating 30-60 minutes for restorating
Posted by Ascendo, 11-26-2009, 02:44 PM

Quote:

Originally Posted by IbnKuldun

Ascendo, apparently its affecting the whole of stratford and quite a few data centers. Which can only mean that those huge petrol powered generators outside the data centers are a big con.

Petrol powered generators would indeed be a con...they're powered by diesel!

One would assume that TATA would have more than one generator for a facility that size - I was hoping that perhaps not everything was down.
Posted by geoffgarside, 11-26-2009, 02:45 PM
As I recall from the tour I think they've got two or three generators in there
Posted by AdamSmith, 11-26-2009, 02:46 PM

Quote:

Originally Posted by IbnKuldun

Ascendo, apparently its affecting the whole of stratford and quite a few data centers. Which can only mean that those huge petrol powered generators outside the data centers are a big con.

Lots of providers might call Tata Stratford "their" datacentre but it's just the one building offline.

Most of the time generator backup works perfectly but of course generators are internal combustion engines, a design well over 100 years old, so they need to be tested and maintained just like a car/truck/train.

Something has obviously gone very wrong here with the generators or the ATS. We don't have any equipment there but I hope it gets resolved soon for you guys .
Posted by Mekhu, 11-26-2009, 02:51 PM

Quote:

Originally Posted by chrisbyrd

TATA just called me back - presently estimating 30-60 minutes for restorating

Thanks Chris.
Posted by Milovan, 11-26-2009, 02:55 PM

Quote:

Originally Posted by geoffgarside

As I recall from the tour I think they've got two or three generators in there

Power has been out for a while and the facility was running on UPSs until they ran out of juice, apparently they could not (and still cannot) get the generator(s) started.
Posted by geoffgarside, 11-26-2009, 03:01 PM
damn, I don't envy the guys who were trying to get the generators going while the UPS juice was draining
Posted by Milovan, 11-26-2009, 03:05 PM
We have two engineers on site and we will update as soon as we hear anything. Should be helpful as it has been rather impossible to get TATA on the phone.
Posted by chrisbyrd, 11-26-2009, 03:07 PM

Quote:

Originally Posted by Milovan

We have two engineers on site and we will update as soon as we hear anything. Should be helpful as it has been rather impossible to get TATA on the phone.

Can you DM me your email/phone? I have a contact at TATA who is responding
Posted by geoffgarside, 11-26-2009, 03:07 PM
Grand, wonder how quiet it is in there tonight
Posted by geoffgarside, 11-26-2009, 03:20 PM
power is apparently back on now, heres hoping things come back up nicely
Posted by Ascendo, 11-26-2009, 03:23 PM
Still dead for us.
Posted by geoffgarside, 11-26-2009, 03:25 PM

Quote:

Originally Posted by Ascendo

Still dead for us.

Equipment might still be offline or booting, but apparently the power is coming back online now, or at least some of it. They might be restoring it to the building in stages.
Posted by Milovan, 11-26-2009, 03:36 PM

Quote:

Originally Posted by geoffgarside

Equipment might still be offline or booting, but apparently the power is coming back online now, or at least some of it. They might be restoring it to the building in stages.

I can confirm this is correct.
Posted by geoffgarside, 11-26-2009, 03:37 PM

Quote:

Originally Posted by Milovan

I can confirm this is correct.

Do you know which colo rooms are powered yet?
Posted by mwright, 11-26-2009, 03:37 PM
got this posted by a guy in another thread:

TimHayes View Beta Profile
New Member

Join Date: Nov 2009
Posts: 1
Managed to contact Alan Dean - they have had major problems...

November 26th, Power Outage at TATA Stratford Facility

As of 4:48pm grid power has been lost to the facility, the UPS systems took over as expected however the generators did not come online, a power engineer has been on site and resolved this issue, we are currently working to bring all racks and servers back online.

POWER RETURNED, RACKS BEING POWERED UP IN PHASES

The above estimate has come from the power engineer on site, please note that once power has been restored to the site we will begin a phased start up of all servers starting with core infrastructure, we expect to complete the initial power up within 30 minutes of the above times however support will be slow to react for several hours afterwards.

Incident History
4:48pm - Power to site lost - running on UPS.
5:15pm - UPS systems depleted, generators failed to start.
5:30pm - Generators failovers will not funciton, power engineer dispatched.
6:32pm - Power Engineer arrived on site.
6:54pm - Power Engineer estimates 30-45 minutes to return power to site.
7:16pm - Power returned to site, process of booting up each rack commenced
Posted by Ascendo, 11-26-2009, 03:40 PM
Back up now.
Posted by geoffgarside, 11-26-2009, 03:52 PM
wooooo, our equipment is coming back online now
Posted by geoffgarside, 11-26-2009, 04:34 PM
Apparently there will be a planned outage to fix the temporary fix they used to get the generators online. I hope its not going to be tonight.

Can anyone in the TATA building confirm this?
Posted by nhdarryl, 11-26-2009, 04:56 PM
This is the reply we have had from Coreix on this:

Quote:

Originally Posted by Coreix

The TATA Stratford facility from which Coreix operates suffered a power failure today, while the site has worked through several outages in the past (including one yesterday) the generator systems on this occasion did not start correctly.

An engineer has been on site and resolved this issue and power has been restored, in most cases service will have been resumed and no further action will be required by yourself.

If you still have an ongoing issue please respond to this e-mail and we will look into the issue further, if you do not respond we will assume the issue has been resolved.

Unfortunately the fix that has been put in place is a temporary bypass and will require us to operate a controlled power reset to bring the facility back onto UPS and Generator power, we are awaiting confirmation of when this will take place.

We apologise for any inconvenience you have been caused and will contact all clients with more detailed information within the next 24 hours. We further apologise for the unavailability of the status page during this issue due to an unanticipated DNS fail over issue.

Also just this second got another update after asking when this would be:

Quote:

Originally Posted by Coreix

At this point, I honestly don't know, we are awaiting further information from the power engineer.

In terms of the actual power down and up this process should be managed so should take no more than a few minutes, I would expect it's basically a case of turning off all the breakers, swapping the switch fabric across and then turning the breakers back on again.

Our core issue is likely to be that a lot of servers will end up being hard reboot, we'll be mass mailing every support ticket we have open on the issue before this happens as a minimum. At this point we are awaiting more information and we will let you know as soon as we know more.

Posted by Milovan, 11-26-2009, 04:56 PM
This has not been confirmed yet and is not official, but yes, there are rumors of a planned outage.
Posted by geoffgarside, 11-26-2009, 04:58 PM
Is their generator power not run through their UPS system?
Posted by nhdarryl, 11-26-2009, 05:19 PM
Another update, no outage

Quote:

Originally Posted by Coreix

I've spoken with the power engineer and they have confirmed that they will not need to turn power off, the only issue we have is that it will take about 8 hours for the batteries to fully recharge (ie if there is another power failure in this time frame the UPS may not hold the load long enough for the gen's to kick in).