Easynet air conditioning down

Portal Home > Knowledgebase > Articles Database > Easynet air conditioning down

Posted by EuroVPS/Sam, 10-26-2009, 08:34 PM
Dear WHT , The leaseweb air conditioning seems to be down , due to which we have downtimes on all our servers Currently the A/C in the Leaseweb rack is OFFLINE and many of our servers have automatically shut down via their security systems. For preventive measures, we have shut down ALL of our servers to prevent server burn ups and data loss from failing drives.We have already driven attention of leaseweb support to the scenario. Anyone else experiencing troubles ?? Last edited by EuroVPS/Sam; 10-26-2009 at 08:40 PM.
Posted by MikeDVB, 10-26-2009, 08:38 PM
That's pretty extreme, has LeaseWeb released any information?
Posted by rustelekom, 10-26-2009, 08:40 PM
As far as i could see our servers is fine (yet ?). May be it is isolated issue with particular room ?
Posted by EuroVPS/Sam, 10-26-2009, 08:40 PM
nothing yet ...
Posted by PersianImmortal, 10-26-2009, 08:48 PM
Thanks for the info Sam. Here is the Leaseweb announcement on this issue: http://noc.leaseweb.com/status.php?i=418 ----------------------------------------------- Dear customer, At the moment we experience some troubles in (part of) our network. We are doing our utmost to solve this issue a.s.a.p. As soon as we have more information available, we will publish it here. Please be patient while we are working on this problem. Regards, LeaseWeb BV
Posted by EuroVPS/Sam, 10-26-2009, 08:54 PM
http://noc.leaseweb.com/status.php?i=418
Posted by Bihira, 10-26-2009, 09:31 PM
Ouch that sucks; though I've been hearing of more and more datacenters allowing hotter than normal tempature and getting roughly the same failure rate... Though I guess that's for another discussion.
Posted by IfHost, 10-26-2009, 09:38 PM
Wow, that sucks... Hmmm I would move to a new company. In this cut throat industry any down time is unacceptable in my opinion.
Posted by Everyday, 10-26-2009, 09:51 PM
Is there only one AC unit or did something happen to take down all the units? Whoever built their place, or where ever they colo, obviously didn't build in any redundancy. Not exactly a place you would want to be in. That being said, stuff happens but this is pretty severe.
Posted by squirrelhost, 10-26-2009, 09:52 PM
Looks like swiftway.pl (and swiftway.nl, and swiftway.co.uk) are all down. Far as I know they have all their servers in Easynet, but leasweb only have some older kit there. Anything I've got from leaseweb over last 2 year is in their own Evoswitch datacenter.
Posted by AdmoNet, 10-26-2009, 09:57 PM
I agree. Most places build in redundancy with both DX (standard CRAC units) and chilled water loops. Most places have redundant chillers with N+1 to assure if one unit fails they will still have plenty of capacity. It would be interesting to see what failed exactly to cause this large of an issue.
Posted by eurovps_jeff, 10-26-2009, 10:18 PM
Airco failure location EASYNET room 2A/2F [Update] Tue 27 Oct 2009 00:30 Expected Downtime: Unknown Location: Easynet room 2A Affected IP ranges: Unknown Reason: Airco failure ******************************* Dear customer, At the moment we experience some troubles in (part of) our network. We are doing our utmost to solve this issue a.s.a.p. As soon as we have more information available, we will publish it here. Please be patient while we are working on this problem. Regards, LeaseWeb BV [UPDATE 02:07] Unfortunately our routers in room 2A reached their thresholds for normal operating mode. They will shutdown within a few minutes to avoid hardware failure. Due to this all network connectivity will be affected. As soon as the airco failure has been repaired we will power up our routers again. [UPDATE 03:09] Since approx. 00h30 EasyNet is suffering a cooling failure on the second floor.At this moment EasyNet is working on solving this issue, however an ETA has not been given yet. In case customers want to have their servers shutdown, please contact support@leaseweb.com. We will try to shutdown the servers cleanly, however we cannot guarantee this possibility due to access restrictions within the DC at this moment.
Posted by eurovps_jeff, 10-26-2009, 10:27 PM
Greetings; We just got word that cold air is flowing again. We will start powering up servers in a short while. Update soon to follow. Thanks, Jeff D. Accounts Director EuroVPS, Inc
Posted by eurovps_jeff, 10-26-2009, 10:33 PM
From noc.leaseweb.com [UPDATE 03:33] At this moment EasyNet is overriding systems and systems are starting up again. Temperature will be dropping slowly and it is expected to be at normal temperature within 2hours again. We are currently waiting for acceptable operation temperature to start our routers again. Thanks, Jeff D. Accounts Director EuroVPS, Inc
Posted by harajme, 10-26-2009, 10:34 PM
Hi, My server with swiftway is also down since one hour ago.
Posted by swiftnoc, 10-26-2009, 10:56 PM
The airco system of the whole second floor was down. Our engineers are onsite to restart all servers that are affected, once the rooms are back on normal temperature. Our core routers are located in Easynet, so we had to shut them down as well. As far as i understand it had something to do with the coolers on the roof, but nothing has been confirmed yet.
Posted by fastdeploy, 10-26-2009, 11:14 PM
And yet from the biggest to the smallest providers downtime is an unfortunate reality of running this kind of business. Of course every provider strives to design and build a fault-tolerant N+1 infrastructure in an N+1 (or +2) facility but many of us (most?) rely on someone else - a data center - to provide a key piece of that redundancy. Just in my region every major DC provider has had at least 1 power outage - in spite of N+1 gensets, large battery plant, automatic transfer switches, and fully engineered electrical infrastructure - so no one is perfect. Even when the entire facility is owned and built out by the owner (Rackspace, Liquidweb, etc) they've been known to have outages too. Now chronic or regular outages or downtime due to poor management or maintenance of the network and/or server infrastructure of that provider - or a DC provider that is just incompetent - is a different story.
Posted by AdmoNet, 10-26-2009, 11:19 PM
I agree. It's almost impossible to eliminate all points of failure. ThePlanet had a huge failure a while back (maybe quite a while back) in one of their power rooms. I believe a transformer exploded. Proper maintenance can usually avoid if not almost completely eliminate issues with cooling and power. Good luck!!
Posted by MikeDVB, 10-27-2009, 12:12 AM
Explosion followed by around 52+ hours of downtime which did affect me at the time. I left them over that but it wasn't because of the downtime but because of the level of information they didn't distribute to their client base at the time. I am curious as to what caused the failure... /subscribes
Posted by swiftnoc, 10-27-2009, 01:15 AM
Our Network infrastructure is back online, we switched it on active again once room temperatures allowed it.
Posted by harajme, 10-27-2009, 01:16 AM
Hi, My server is a live now thanks Swiftway thanks GOD
Posted by Mashii, 10-27-2009, 05:58 PM
This is acctally what I like providers updating the clients with what's happening and why it happend ect rather then leaving the clients in the dark. So +1 for swiftnoc and eurovps and leaseweb
Posted by yourwebhostereu, 10-28-2009, 01:16 PM
The room was 70+ degrees.... the should have shut down the power (both feeds) but they didn't...
Posted by swiftnoc, 10-28-2009, 01:42 PM
Shutdown the power is not always feasible. First of all, it was not one room. It was the whole floor (~ 1400 m2) with multiple datafloors, suites and cages. If you shutdown the power - a lot of disks will be corrupted instantly, data loss occurs, etc. Swiftway uses only brand hardware and no white label or self build. We found that this is a huge advantage. Our equiptment all suspended the power by itself when it noticed that the air intake was higher then 50 degrees celcius. As a result, we had no hardware failures when the issue was resolved. Even better, all out Supermicro and Dell servers unsuspended themself when the air intake dropped. We only had to manually start our IBM and HP servers. So: Dell, Supermicro servers went in suspended mode, not powering down but not active. Unsuspended themself automatically when temperature dropped. HP and IBM also noticed the temperature increase, but auto shutdown themself. They needed manual intervention to turn them back on. We are again very gratefull that we decided to choose quality brand hardware. --------
Posted by yourwebhostereu, 10-28-2009, 02:07 PM
Ok, let's describe it in other words: shut down every server. Note which servers were already down and start the servers when the problem is over. Now they have claims form defect hardware and the down time, else they only would have to pay the down time. Anyway, I heart that EUnetworks puts air conditioning containers to prevent such high temps, perhaps a good solution for this DC.
Posted by swiftnoc, 10-28-2009, 02:13 PM
Let me cut in for you. Shutdown every server - 21 000 servers - while the temperature is rising by the minute. How would you organise that? who would do it? I am sure a company cannot make their engineers do it while its +50 celsius and rising.
Posted by yourwebhostereu, 10-28-2009, 02:23 PM
Even with 21000 servers you can do this. But the only thing you need is a good plan for this. Okay, you can't shut down every server, that is not realistic. But they can at least start with shutting down the servers in every room, this will at least reduce the amount of defect hardware. Anyway, you've got a good point. The racks are locked so it is impossible to access them to shut down those servers unless they have a main access code for it. But they should have e-mailed every client with a colo so the client knows what the problem was (or if they didn't know that there is a problem) with the possibility to shut down the server. But no, they only listed it on the noc website. Perhaps some big clients are notified about this but no. You are right that they don't have much options, but they should have informed the clients properly. In my opinion there should be a mailing list for every client who has a dedicated server or colo in the DC, including the clients from clients (rackspace sellers for example) so they know what is going on and what options they have. And they should have a backup system like EUnetworks has, that is cheaper for them and there wouldn't be any down time.
Posted by swiftnoc, 10-28-2009, 02:36 PM
Its confusing i know, but i am not talking about Leaseweb. But about Easynet. I know we kinda hijacked this thread but all top floor Easynet was affected, not just Leaseweb. Security have masterkey to the racks, they have normal locks. What you mean with that? Easynet is a Tier-4 datacenter, EUnetworks (NL) a tier-3. Easynet is setup more redundant, then EU networks. Easynet also has a stronger financial base (IMHO) being owned by BskyB. Financial base is quite important these days. The issue was not with the airco units itself. Let this be clear. Its far more complex.
Posted by yourwebhostereu, 10-28-2009, 02:43 PM
I've read what the problems were, I know it was the DC I'm also not writing about Leasweb. I've heart (I didn't check this) that EUnetworks has Air conditioner containers. When the air conditioning in the rooms is defect they put those containers by the rooms and activate them. Then the temperatures won't raise up to 70-80 degrees. With the backup system I mean the air conditioner containers, not the redundancy I know that the system which controls the air flow was defect, the cooling system was working fine but there was no airflow in the rooms because the defect system locked the circulation. They have opened this manually after a few hours. Correct me if I'm wrong Edit: last days are heavy for webhosters i3D and Easynet, there are a lot cancellation requests already... (not here, you probably alread knew that we are in EU networks).
Posted by swiftnoc, 10-28-2009, 03:02 PM
How would that air conditioning containers remove and cycle the air when the airflow conducts are closed?
Posted by yourwebhostereu, 10-28-2009, 03:04 PM
I will search the internet, but you can move containers against such rooms. But then you need some space in the wall for this.
Posted by swiftnoc, 10-28-2009, 03:06 PM
The space in wall probably utilize the same airflow conducts. So with such a defect, they would not be useable as well. [offtopic] Did EUnetworks finish their new floor already ?
Posted by ItsChrisG, 10-28-2009, 03:12 PM
To cool a room of 1400m2 you would need a LOT of portable a/c's, so much that its not possible for this to be a solution.
Posted by swiftnoc, 10-28-2009, 03:23 PM
They are multiple suites, not one room of 1400 m2. The whole 2nd floor was without airco. I think the floor is divided in ~ 5 suites or so. That makes it even more complicated altough, many portable a/c units needed. And since the airflow ducts where the issue, i do not think that this would have been a solution.
Posted by yourwebhostereu, 10-28-2009, 03:30 PM
Well they have spare a/c, anyway last time they had a power error they added generators in containers next to the canteen. The person who said that they would add containers with a/c is not sure if they really do that, so I don't know what they would do I'm sorry about that. But I do know that EUnetworks has from two sides air input. Here is everything about them: link No, it will probably not finish this year... Anyway, Easynet is a Tier 1 or 2 DC, isn't it? Anyway, seems to that I wrote some things before I was thinking or was sure about that, I'm sorry. Heavy day for me...
Posted by swiftnoc, 10-28-2009, 03:45 PM
No, EUnetworks is tier 2 or 3, Easynet is Tier 4. Just for you comparison of financial strenght. 'euNetworks Group Limited has reported a 32% improvement in revenue to €14.3 million for the six months ended 30 June 2009 (“1H09”), compared to €10.8 million achieved in the same period a year ago (“1H08”)." Eunetworks Group Limited is the owner of Eunetwork datacenter your located Easynet owner BSkyB: Revenue £4,952 million (2008) And BSkyB is Easynets biggest customer and user of the DC as well. The Easynet group alone had a total Revenue of £185m (2008)