Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Hurricane Sandy - Outages NY/NJ/CT
Hurricane Sandy - Outages NY/NJ/CT
| Posted by voip_expert, 10-29-2012, 11:09 PM |
75 Broad Street - Going Down
The flooding has submerged and destroyed the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level. The available fuel reserves on the mezzanine level are estimated to support customer loads for approximately 5-7 hours. Once this fuel supply has been exhausted the generator will no longer be able to sustain operation and critical customer power loads will be lost. |
| Posted by SolaDrive, 10-29-2012, 11:25 PM |
|
Geez, really astonishing what this storm is doing to those cities. How is flooding to the building itself? |
| Posted by voip_expert, 10-29-2012, 11:26 PM |
|
33 Whitehall - Cogent is Down |
| Posted by voip_expert, 10-29-2012, 11:29 PM |
|
nLayer link is down at 882 3rd Ave. |
| Posted by JonBiloh, 10-29-2012, 11:46 PM |
|
Voxel/Internap are down at 111 8th Avenue |
| Posted by mctDarren, 10-29-2012, 11:46 PM |
Quote:
|
Originally Posted by voip_expert
nLayer link is down at 882 3rd Ave.
|
ezzi.net is also in that facility, I believe. Anyone heard their situation there? I saw some boxes there come back a short time ago, then they dropped off again after about 20-30 minutes. |
| Posted by voip_expert, 10-29-2012, 11:48 PM |
Quote:
|
Originally Posted by JonBiloh
Voxel/Internap are down at 111 8th Avenue
|
Our uplink still up with them and performing very good! |
| Posted by voip_expert, 10-29-2012, 11:49 PM |
Quote:
|
Originally Posted by serversphere
ezzi.net is also in that facility, I believe. Anyone heard their situation there? I saw some boxes there come back a short time ago, then they dropped off again after about 20-30 minutes.
|
Facility it self is up, and all operational so far XO and nLayer only have they circuits down there, Cogent, Verizon are just fine. |
| Posted by R1CH, 10-29-2012, 11:53 PM |
Quote:
|
Originally Posted by JonBiloh
Voxel/Internap are down at 111 8th Avenue
|
Is this the LGA6 facility? |
| Posted by JonBiloh, 10-29-2012, 11:54 PM |
Quote:
|
Originally Posted by R1CH
Is this the LGA6 facility?
|
Yes, LGA6. I believe it is fed by LGA11. |
| Posted by Lancia, 10-29-2012, 11:56 PM |
|
I can confirm my server with Voxel/Internap is down at LGA6 is offline. That's at 111 8th Avenue. |
| Posted by R1CH, 10-29-2012, 11:56 PM |
|
Ah, guess that explains it. Got the warning emails about LGA11, but our box is at LGA6. |
| Posted by Lancia, 10-29-2012, 11:58 PM |
I did too and was thinking "wow, glad I'm not in LGA11!" And then...mine went down! Good to see I'm not the only one, thanks for posting here guys.
Hopefully we hear from Internap soon. I'm not going to bother them because I'm sure they know there's a problem! |
| Posted by voip_expert, 10-30-2012, 12:00 AM |
|
Voxel in 7 Teleport operational just fine. |
| Posted by IcEWoLF, 10-30-2012, 12:02 AM |
Here is more insight information about Internap NY:
Quote:
|
InterNAP is reporting that utility power is no longer available to our NYC location and it has been successfully transferred to generators, keeping all services online.
|
Quote:
|
InterNAP should have enough on-site fuel for multiple days, and refueling contracts to provide more as necessary.
|
Quote:
|
InterNAP reports that the facility itself has 60+ hours of fuel but that the basement is now flooded, preventing it from being pumped up to the generators. The site can sustain power for another 4-6 hours on the fuel currently available, they say. It is possible that waters will recede by then. We have posted an event to all NYC customers with all the details available.
|
Quote:
|
o real whoops so far -- InterNAP seems to be handling things as well as they can and I haven't seen them making mistakes with this in the last 24 hours. I just hope that the flood waters abate quickly in order to minimize downtime. Power outages aren't the end of the world as long as there's minimal corruption -- mostly a nuisance.
|
This is from NFoServers FB page. |
| Posted by voip_expert, 10-30-2012, 12:11 AM |
|
Cogent have gone down at 12:03:42 AM at 882 3rd Ave. |
| Posted by VortexServers, 10-30-2012, 12:15 AM |
Quote:
|
Originally Posted by JonBiloh
Voxel/Internap are down at 111 8th Avenue
|
 Good thing I went with my home town vs NYC  |
| Posted by r00ter, 10-30-2012, 12:17 AM |
|
We're in InterNAP NYM008 (111 8th - 10th Floor), all our gear is still up at this time. |
| Posted by inerail-chris, 10-30-2012, 12:21 AM |
|
The Voxel issues are being caused by a power issue in XO's suite in 111 8th. |
| Posted by bqinternet, 10-30-2012, 12:37 AM |
Quote:
|
Originally Posted by voip_expert
Our uplink still up with them and performing very good!
|
Our Voxel link has been up the whole time as well. We go through their LGA3 gear at 111 8th. |
| Posted by mripguru, 10-30-2012, 12:39 AM |
|
Datagram is down too (33 Whitehall - not Cogent) |
| Posted by inerail-chris, 10-30-2012, 12:53 AM |
|
121 Varick has lost power - generator exhaust backing up. |
| Posted by rhanthony, 10-30-2012, 01:13 AM |
|
Could we get a location code for Voxel sites tacked into the top of this thread where the addresses are showing? And is anyone else unable to access their ubersmith control panels right now? |
| Posted by jbfrompg, 10-30-2012, 01:23 AM |
I got a hold of someone at Internap/Voxel. They said there was an electrical panel issue at the LGA8/LGA9 facility. It should be up within hours. LGA11 does not have a good forecast at this time. Ubersmith is in LGA8/LGA9 and they are trying to relocate it.
Edit: Voxel/Internap at 111 8th Avenue is LGA8/LGA9 |
| Posted by hostvirtual, 10-30-2012, 01:34 AM |
Quote:
|
The building itself is being evacuated and no remote hands support will be available to assist in any equipment shutdown. Life safety is our number one priority and we are making plans to completely exit the facility. No customer access to the building is possible at this time either. Due to the evacuation, Internap will not be able to provide any exact updates on when the fuel will be exhausted and critical customers loads lost, but as noted, we believe it will take place in approximately 5-7 hours from now.
|
http://pastebin.com/9Fr2eW6U - pretty chilling stuff. |
| Posted by jdownj, 10-30-2012, 01:48 AM |
|
Ouch... Some of my stuff is not far away, but not in the city itself, and all I've dealt with so far is some chucklehead deciding that NOW would be a great time to do "Scheduled Maintenance" on networking gear and not of course let me know that it was scheduled. I honestly am surprised that there hasn't been connectivity issues yet... I'm staying awake for a few more hours to see what happens. |
| Posted by Kett2004, 10-30-2012, 01:52 AM |
Here's the latest update from Steadfast on their nyc16 facility at 121 Varick St:
We have just been informed that due to an auxiliary electrical failure, power systems have failed at our nyc16 facility at 121 Varick St in New York City. All New York servers and services are now offline and we have no ETA for service restoration. We will provide updates as we have them.
Most systems should be configured to power on when power service resumes and we can assist via ticket as needed when power becomes available. We will, of course, honor full SLA credits for the period(s) in which power has been unavailable in the facility. We apologize for any inconvenience this has caused you or your business.
The following information was provided by our facilities vendor at 121 Varick St, known upstream as LGA4:
"The auxiliary power for our LGA4 Network Operations Center has gone out. LGA4 data center is will be going down. NOC responses will be delayed as other elements need to be contained to ensure the safety of our crew as we endure the effects of Hurricane Sandy. We are in the process of relocating our team to our near by LGA1 data center.
At this point it is encouraged that customers who have the ability to, remotely shut down any critical equipment or services to mitigate the risk of any interruptions to service. Our staff is evacuating the site and remote hands services will be suspended until the threat passes. Customer access is also restricted until further notice.
We will report back as new information becomes available about the condition of our Network Operations Center and Internet Data Center at 121 Varick St."
I guess it will be a late night bringing up downed websites to other servers.. |
| Posted by centauricw, 10-30-2012, 01:59 AM |
|
Spoke with Peer 1. They are running their generators off a gravity-fed tank, so they will be good for quite a while. No outages reported. |
| Posted by voip_expert, 10-30-2012, 02:05 AM |
|
So far I can say from all our NYC Metro facilities we have been running solid from 7 Teleport Dr, sure we lost multiple carriers and connectivity to few peering in NYC, but other then that we are fully operational. |
| Posted by mctDarren, 10-30-2012, 02:12 AM |
|
ezzi.net looks to be back. Scary stuff coming out of some facilities up there. Thoughts are with you folks... |
| Posted by mctDarren, 10-30-2012, 02:45 AM |
|
Hm, did nLayer just go down again at 882 3rd Ave? All my boxes suddenly dropped off again. They must still be battling... |
| Posted by mr fixit, 10-30-2012, 03:05 AM |
|
We have racks of servers at 111 Eighth as well as Telehouse NYC. So far TH running beautifully and receiving updates constantly. I can assure you that most of these outages have nothing to do with flooding or the hurricane, and are a direct result of the city's onerous policy to terminate power everywhere below 38th street where every data center is located. Even NYU Hospital is down and relocating their patients because they ran out of generator power. I have been in Manhattan all day long and there is barely any flooding except for all the way downtown, some underground areas and the coastline in very limited areas. And the flooding that does exist in most places downtown is ankle-high. This hurricane has *not* been a big deal for most of NYC. Shutting down power to half of the city was unnecessary and completely overboard. But that is not relevant, data centers should stay online that is why they have generators and proper planning for disasters. I can assure you all most of the power failures reported at these data centers (minus 75 Broad) are related to improper planning, testing, faulty infrastructure, and human error with hosting providers left to throw up their arms in frustration. This is not the first time 111 or XO has screwed up on power. The last time some idiots reversed polarity on the roof. |
| Posted by voip_expert, 10-30-2012, 03:06 AM |
Quote:
|
Originally Posted by serversphere
Hm, did nLayer just go down again at 882 3rd Ave? All my boxes suddenly dropped off again. They must still be battling...
|
Evrething is down there since 02:08 AM, I think generators did not kick in or so... |
| Posted by HD-Sam, 10-30-2012, 03:42 AM |
Zopim, the live chat software is also down. They seem to be hosted by the affected Internap DC. We use Zopim and have temporarily removed their chat scripts from our website as it was slowing down page requests across our entire site.
The following well known sites are down: Gawker is down, Gizmodo is down, HuffPost was down (now back up). All hosted by Datagram. Hard to believe these companies don't have any type of geographic redundancy in place. Not even a splash page. |
| Posted by jsw6, 10-30-2012, 03:43 AM |
Quote:
|
Originally Posted by mr fixit
I can assure you that most of these outages have nothing to do with flooding or the hurricane, and are a direct result of the city's onerous policy to terminate power everywhere below 38th street where every data center is located. Even NYU Hospital
|
Almost all of the utility power failures in Manhattan resulted from the substation explosion near FDR Drive.
The intentional outages were very limited and did not include one single co-location facility mentioned in this thread with the possible exception of 25 Broadway -- I am not totally clear on that one as it is adjacent to the planned outage zone.
conEdison has not given out any firm repair estimates, of course; but they do not seem very optimistic about how quickly the substation can be repaired, saying only, "up to a week but probably less" for the 1/4-million customers affected by that throughout Manhattan. The Washington Post and CNN have interviewed their spokespeople within the last couple of hours. |
| Posted by SEEKER2007, 10-30-2012, 03:57 AM |
|
Internap said in there update email that there is power backup fuel for next 6 hours and users can gracefully shutdown there servers but my website is down before the email updated reached me ??? |
| Posted by R1CH, 10-30-2012, 04:07 AM |
Quote:
|
Originally Posted by SEEKER2007
Internap said in there update email that there is power backup fuel for next 6 hours and users can gracefully shutdown there servers but my website is down before the email updated reached me ???
|
Internap have multiple affected facilities. LGA11 has the small amount of reserve power which by now almost exhausted. LGA6 dropped off suddenly with what I'm assuming is a power failure, LGA8 is also experiencing a "catastrophic power failure". No official word yet on LGA6. |
| Posted by mr fixit, 10-30-2012, 04:16 AM |
Quote:
|
Originally Posted by jsw6
Almost all of the utility power failures in Manhattan resulted from the substation explosion near FDR Drive.
The intentional outages were very limited and did not include one single co-location facility mentioned in this thread with the possible exception of 25 Broadway -- I am not totally clear on that one as it is adjacent to the planned outage zone.
|
While the transformer explosion affected half of the city many the providers mentioned on this board announced planned cutover to generator power specifically due to the intentional ConEd shut down in their trouble tickets. Cutovers and outages occurred outside the time of the explosion. For instance XO went down at Midnight, that has nothing to do with ConEd. Peer1 is at 75 Broad in the zone, as is 33 Whitehall. Internap along with many others are at XO, which has power problems. Some of the data centers acknowledged phase drops , operator errors, UPS failures, and power panel problems which are happening in specific suites in these buildings, while the rest of the suites in many of those buildings are running on building generator without any issue. The majority of these data centers really should not be down right now. |
| Posted by myuserid, 10-30-2012, 04:26 AM |
|
Looks like Interserver are offline....well my servers are anyway! |
| Posted by jsw6, 10-30-2012, 04:26 AM |
Quote:
|
Originally Posted by mr fixit
The majority of these data centers really should not be down right now.
|
I don't think you have accurate information about the conEd planned outage vs the outage caused by the substation explosion. The planned outage affected about 10,000 customers, all east of Broadway and south of the Brooklyn Bridge. The substation explosion, which obviously did not happen on purpose, is what affected another 250,000 customers which encompasses every major co-location facility except for TELEHOUSE at 25 Broadway.
Your gripe was about the planned cut but that did not cause any of the troubles you are mentioning -- XO at 111 8th Ave and such.
I don't mean to drag this thread off-topic. The reason I am posting this information is because it is relevant to how quickly utility power can be restored to these buildings. conEd says it will be quicker for the "intentional cut" area than the rest of Manhattan. "Up to a week but probably less." |
| Posted by SC-Daniel, 10-30-2012, 04:27 AM |
Quote:
|
Originally Posted by myuserid
Looks like Interserver are offline....well my servers are anyway!
|
Yes, all of our services in Interserver are offline as well. Their website is also offline at this time. |
| Posted by Kakashi, 10-30-2012, 04:40 AM |
|
One of our servers at Reliablesite.net is currently down, as is their website for us. |
| Posted by Jakiao, 10-30-2012, 04:40 AM |
|
After several hours of shaky connectivity, our equipment at Honelive in NYC has gone offline. Good luck, my little XenServer! |
| Posted by keywolf, 10-30-2012, 04:49 AM |
|
We are experiencing intermittent outages with our Atlanta server as some of our traffic gets routed through NY, it appears to be rerouting now but ping times are up and routes significantly longer. |
| Posted by R1CH, 10-30-2012, 06:05 AM |
|
My server in LGA6 just came back up. Didn't lose power, looks like a network outage only. |
| Posted by Jono20201, 10-30-2012, 06:07 AM |
|
My services at ColoCrossing just came back online. |
| Posted by mr fixit, 10-30-2012, 06:11 AM |
Arbinet, Ubiquity, Metanet, Webair, Voxel seem OK except for that troublesome XO site.
60 Hudson, 111 Eighth, 85 Tenth, 25 Broadway, zColo, Telx, Telehouse, and Equinix sites, 65 Halsey all seem to be up and running. Meadowlands Parkway seems to be offline.
Can anyone confirm these? Any details would be appreciated.
What other hosts are offline?
This is pretty bad... lets hope Con Ed moves quickly. |
| Posted by jbfrompg, 10-30-2012, 06:11 AM |
|
I see LGA8 and LGA9 coming back online. |
| Posted by mr fixit, 10-30-2012, 06:13 AM |
Quote:
|
Originally Posted by R1CH
My server in LGA6 just came back up. Didn't lose power, looks like a network outage only.
|
Thanks for this update, we are there and still having some cabinet issues.  XO said something about some transport and some power circuits affected, etc. Earlier we had one rack go black while the other was up, then shortly after another rack went black. I hope we can be in your shoes soon! |
| Posted by jNive, 10-30-2012, 06:14 AM |
|
Back online at LGA8 (111 8th) - System uptime not interupted! (thankfully) |
| Posted by white_tiger, 10-30-2012, 06:37 AM |
my server in honelive also down.. sent email 8 hours ago, no response..
galaxyvisions up
reliablesite up |
| Posted by keywolf, 10-30-2012, 06:41 AM |
|
They have probably moved there staff from the site, life is more important than a reply to your email or your server being online! |
| Posted by dotHostel, 10-30-2012, 06:42 AM |
Quote:
|
Originally Posted by mr fixit
60 Hudson, 111 Eighth, 85 Tenth, 25 Broadway, zColo, Telx, Telehouse, and Equinix sites, 65 Halsey all seem to be up and running. Meadowlands Parkway seems to be offline.
Can anyone confirm these? Any details would be appreciated.
What other hosts are offline?
|
Equinix North Bergen is up but it seems Reliablesite / Atlantic Metro is down there.
EDIT: Reliablesite came back online |
| Posted by white_tiger, 10-30-2012, 06:54 AM |
Quote:
|
Originally Posted by keywolf
They have probably moved there staff from the site, life is more important than a reply to your email or your server being online!
|
i agree life is more important and i understand it is down to hurricane  but i appreciate internap who send me email every hour or so explaining the situation.. it's so scary :| |
| Posted by thehosterdude, 10-30-2012, 07:02 AM |
Quote:
|
Originally Posted by myuserid
Looks like Interserver are offline....well my servers are anyway!
|
mine is offline as well |
| Posted by mr fixit, 10-30-2012, 07:07 AM |
It looks like Equinix in 111 8th did get hit and had generator failures. Cogent 33 Whitehall seems hit with generator problems as well.
This is what I meant before when I said these outages should not have happened, the generators at all these downed buildings should be working, there is no excuse really. |
| Posted by (Stephen), 10-30-2012, 07:13 AM |
Quote:
|
Originally Posted by mr fixit
It looks like Equinix in 111 8th did get hit and had generator failures. Cogent 33 Whitehall seems hit with generator problems as well.
This is what I meant before when I said these outages should not have happened, the generators at all these downed buildings should be working, there is no excuse really.
|
It sounds to me like their generators are fine, but it is their fuel supplies that are the weak link. Either by regulation, or by design to be easy to refuel etc, they seem to be flood susceptible, while the actual generators are higher and not flooded.
You really don't want to have a genset under water, or your fuel for that matter, but fuel is much more replaceable. It sounds like some were only running on daytanks once their fuel supplies were flooded out. |
| Posted by white_tiger, 10-30-2012, 07:28 AM |
update, my server in honelive is up, seems network outage, not power outage
Code:
Outage duration: 7 hours 40 mins
|
| Posted by JonnyQuags, 10-30-2012, 07:34 AM |
|
For information on InterServer please follow our twitter feed at http://twitter.com/interserver where I am providing updates as they come in. |
| Posted by thehosterdude, 10-30-2012, 07:35 AM |
Interserver is back up.
I found this on their Twitter page . . . generators were up and running fine, but there was no links working
Our Cogent link has now gone down, which was our last link. Currently the Secaucus NJ network is down. ETA in progress. |
| Posted by FRH Lisa, 10-30-2012, 07:37 AM |
Quote:
|
Originally Posted by (Stephen)
It sounds to me like their generators are fine, but it is their fuel supplies that are the weak link. Either by regulation, or by design to be easy to refuel etc, they seem to be flood susceptible, while the actual generators are higher and not flooded.
|
First and foremost, I hope all the employees at the affected DCs are safe and sound. Fuel, generators, racks, service credits -- they can all be easily replaced. People can not.
With that in mind, how can you build a datacenter with a weak link like this? I understand that the flooding is an extraordinary event for NYC, so I'm not criticizing. Making it through the storm without an outage is a heroic event. But if you're going to offer redundancy to your customers, shouldn't your fuel pumps / fuel tanks / AC panels / etc be well above ground? We aren't talking about Discount Donny's Internet-O-Rama here, we're talking about major, well-known providers.
Again, I'm not criticizing. I'm genuinely curious. |
| Posted by JonnyQuags, 10-30-2012, 07:40 AM |
Quote:
|
Originally Posted by thehosterdude
Interserver is back up.
Our Cogent link has now gone down, which was our last link. Currently the Secaucus NJ network is down. ETA in progress.
|
Cogent is currently still down for us. According to the tech I spoke to the have a power failure at 32 ave of the Americas and are waiting for a generator to be delivered. ETA was 7:00 AM EST but that is missed. |
| Posted by rustelekom, 10-30-2012, 08:15 AM |
|
Interserver online now. Power was on - uptime stay unchanged. |
| Posted by Crashus, 10-30-2012, 08:19 AM |
Quote:
|
Originally Posted by FRH Lisa
First and foremost, I hope all the employees at the affected DCs are safe and sound. Fuel, generators, racks, service credits -- they can all be easily replaced. People can not.
|
true dat
I hope everyone is okay |
| Posted by Winstyn, 10-30-2012, 08:31 AM |
First of all, thoughts and prayers to everyone involved.
Secondly, as I read through this thread I notice that most of the affected customers have not been affected by power outages just network outages.
Our network is based in Los Angeles, and from reading this I guess that 72 Broad, and 33 are like the One Wilshire of LA. The buildings seems to do a great job of staying online however it seems it only takes one building outage to bring the rest offline in a network context. |
| Posted by mdrussell, 10-30-2012, 08:42 AM |
Quote:
|
Originally Posted by Winstyn
First of all, thoughts and prayers to everyone involved.
Secondly, as I read through this thread I notice that most of the affected customers have not been affected by power outages just network outages.
Our network is based in Los Angeles, and from reading this I guess that 72 Broad, and 33 are like the One Wilshire of LA. The buildings seems to do a great job of staying online however it seems it only takes one building outage to bring the rest offline in a network context.
|
That's also a risky decision. When the earthquake hits LA, One Wilshire is going to hurt bad. |
| Posted by F-DNS, 10-30-2012, 08:42 AM |
Quote:
|
Originally Posted by FRH Lisa
But if you're going to offer redundancy to your customers, shouldn't your fuel pumps / fuel tanks / AC panels / etc be well above ground?
|
For obvious safety reasons (and the practicality of refilling them) the main fuel tanks are most likely at ground level or below ground. They then use electrical pumps to transfer fuel up to much smaller "day" tanks which provide the immediate supply to generators when they kick in. In normal circumstances this is fine - The day tanks run the generators and the main tanks are refilled down below and fuel pumped up to replenish the day tanks every few hours.
The problem that's most likely occured is that the main tanks and pumps are below ground level and the floods have killed the transfer pumps. Hence why InterNAP were able to predict several hours ago that they'd be down once the day tanks run dry. |
| Posted by plumsauce, 10-30-2012, 09:06 AM |
Quote:
|
Originally Posted by F-DNS
The problem that's most likely occured is that the main tanks and pumps are below ground level and the floods have killed the transfer pumps. Hence why InterNAP were able to predict several hours ago that they'd be down once the day tanks run dry.
|
Now ... if the tank vent had been run up to a higher level like a snorkel, and the pumps had been up on the mezzanine ... then the tanks would have been in the same situation as a fuel tank on the old diesel submarines ... unaffected by water up to a depth of X feet.
++ |
| Posted by Damned_Kush, 10-30-2012, 09:17 AM |
|
anyone knows anything about ezzi.net ? There is no annucment or explanation. They are still offline |
| Posted by F-DNS, 10-30-2012, 09:17 AM |
Quote:
|
Originally Posted by plumsauce
and the pumps had been up on the mezzanine
|
That's what happens when you let an accountant decide what can go up on the premium floor space above ground and what gets consigned to the dusty basement |
| Posted by F-DNS, 10-30-2012, 09:25 AM |
Quote:
|
Originally Posted by Damned_Kush
anyone knows anything about ezzi.net ? There is no annucment or explanation. They are still offline
|
Both their (unreachable) nameservers are registered to AccessIT - Hosting Services, 11 Broadway, Ste 1131, New York which suggests they don't have a redundant nameserver elsewhere and their whole infrastructure is out for the count |
| Posted by Damned_Kush, 10-30-2012, 09:42 AM |
Quote:
|
Originally Posted by F-DNS
Both their (unreachable) nameservers are registered to AccessIT - Hosting Services, 11 Broadway, Ste 1131, New York which suggests they don't have a redundant nameserver elsewhere and their whole infrastructure is out for the count
|
What do you think how bad is this ? and any news from the storm is it finished or still going on. |
| Posted by NetworkPanda, 10-30-2012, 09:55 AM |
Quote:
|
Originally Posted by Damned_Kush
What do you think how bad is this ? and any news from the storm is it finished or still going on.
|
If only their nameservers are down and not their server running the sites, then it should be fixed soon.
Having ns1 and ns2 nameservers at different areas of the world is the best solution for such issues (except if a global hurricane strikes the planet ) |
| Posted by bqinternet, 10-30-2012, 10:00 AM |
Quote:
|
Originally Posted by Damned_Kush
What do you think how bad is this ? and any news from the storm is it finished or still going on.
|
The worst of the storm may be past NYC now, but DCs are going to be running on generators for a while. I wouldn't be surprised to see more data centers lose power over the next few days. |
| Posted by FRH Lisa, 10-30-2012, 10:02 AM |
Quote:
|
Originally Posted by plumsauce
Now ... if the tank vent had been run up to a higher level like a snorkel, and the pumps had been up on the mezzanine ... then the tanks would have been in the same situation as a fuel tank on the old diesel submarines ... unaffected by water up to a depth of X feet.
|
This is what I would have expected. I've heard stories of pumps / starter motors connected to the utility grid, main AC panels in basements, all kinds of critical choke points in really bad places.
Why spend so much money on backup infrastructure when it can be so easily taken out? I know flooding of this magnitude is extremely rare in NYC, but what about a water main burst? A careless contractor? A freak weather event? It seems silly to have redundancy tied to such a vulnerable point of failure. |
| Posted by mr fixit, 10-30-2012, 10:03 AM |
|
We are seeing XO come up in 111 8th, no power outage indeed, just showing a cut over to generator. Was network. Last we heard was generator is stabilizing and they will light up the rest. Way to go! |
| Posted by Kusai, 10-30-2012, 10:05 AM |
|
Hope everyone is safe and sound. Coastal areas are way too risky for mission critical environment. |
| Posted by FliesLikeABrick, 10-30-2012, 10:15 AM |
Quote:
|
Originally Posted by mr fixit
We have racks of servers at 111 Eighth as well as Telehouse NYC. So far TH running beautifully and receiving updates constantly. I can assure you that most of these outages have nothing to do with flooding or the hurricane, and are a direct result of the city's onerous policy to terminate power everywhere below 38th street where every data center is located. Even NYU Hospital is down and relocating their patients because they ran out of generator power. I have been in Manhattan all day long and there is barely any flooding except for all the way downtown, some underground areas and the coastline in very limited areas. And the flooding that does exist in most places downtown is ankle-high. This hurricane has *not* been a big deal for most of NYC. Shutting down power to half of the city was unnecessary and completely overboard. But that is not relevant, data centers should stay online that is why they have generators and proper planning for disasters. I can assure you all most of the power failures reported at these data centers (minus 75 Broad) are related to improper planning, testing, faulty infrastructure, and human error with hosting providers left to throw up their arms in frustration. This is not the first time 111 or XO has screwed up on power. The last time some idiots reversed polarity on the roof.
|
There is a lot of misinformation in this thread, but I wanted to correct this one in particular.
ConEd only shut down power south of wall street or 14th street intentionally, as a preventative measure, when the equipment was at risk of being flooded during operation. Even then, they only did this when the water at The Battery was over 9.5ft. Not doing this would have resulted in much more equipment damage as salt water hit high voltage equipment
The power failure of 39th street and below later on was an unplanned event. I suspect (but do not know for sure) that it was related to this explosion, at least partially: http://www.youtube.com/watch?v=ZAqYZ433TeQ |
| Posted by KarlZimmer, 10-30-2012, 11:30 AM |
Quote:
|
Originally Posted by F-DNS
For obvious safety reasons (and the practicality of refilling them) the main fuel tanks are most likely at ground level or below ground. They then use electrical pumps to transfer fuel up to much smaller "day" tanks which provide the immediate supply to generators when they kick in. In normal circumstances this is fine - The day tanks run the generators and the main tanks are refilled down below and fuel pumped up to replenish the day tanks every few hours.
The problem that's most likely occured is that the main tanks and pumps are below ground level and the floods have killed the transfer pumps. Hence why InterNAP were able to predict several hours ago that they'd be down once the day tanks run dry.
|
Yep, from what we've seen, this is the most common issue. Either the fuel tanks getting contaminated with water or the pumps going out due to flooding. It seems like the main thing people didn't plan for is the flooding, and I do have to say that it is probably a 1 in 100+ year event on the flooding side. Now, certainly not saying that is excusable, and we're definitely looking at relocating our NY site ourselves at this point, though for us it was almost entirely for DR/geo-redundancy in the first place anyway. |
| Posted by Rens, 10-30-2012, 11:34 AM |
Peer1 already down?
"06:30 PT - We are going to implement a controlled shutdown of NY Data Center at 10:45 ET. Customer communications is being prepped."
Out of Diesel and refuel is coming too late. Probably also a bit of a problem to refuel when the tanks are underwater..
http://forums.peer1.com/viewtopic.ph...2&p=9463#p9463 |
| Posted by VN-Ken, 10-30-2012, 11:48 AM |
Quote:
|
Originally Posted by F-DNS
For obvious safety reasons (and the practicality of refilling them) the main fuel tanks are most likely at ground level or below ground. They then use electrical pumps to transfer fuel up to much smaller "day" tanks which provide the immediate supply to generators when they kick in. In normal circumstances this is fine - The day tanks run the generators and the main tanks are refilled down below and fuel pumped up to replenish the day tanks every few hours.
The problem that's most likely occured is that the main tanks and pumps are below ground level and the floods have killed the transfer pumps. Hence why InterNAP were able to predict several hours ago that they'd be down once the day tanks run dry.
|
Hmm, even at Oak Tower here in Kansas City, everything is 5 stories up including generators and all tanks. Fuel can then be pumped up to the generator dock from ground level.
It amazes me that the same was not done in a major market like NYC. |
| Posted by dthani, 10-30-2012, 11:55 AM |
Quote:
|
Originally Posted by Damned_Kush
anyone knows anything about ezzi.net ? There is no annucment or explanation. They are still offline
|
It would be helpful if someone at Ezzi could take a second to post and let us know how severe the damage is. Other hosting companies are doing great with the real time updates. Even if we cant get something real time, a note to say what the status is every hour or two would be great. |
| Posted by sirius, 10-30-2012, 12:53 PM |
|
Let's continue to use this thread to provide updates on hosts that are experiencing outages related to the storm. Reviews on a host, comments about a hosts preparation and all other off topic comments will be removed. If you'd like to provide a review on a host or their preparedness for this event, feel free to use the main forums. |
| Posted by Jono20201, 10-30-2012, 01:10 PM |
Quote:
Subject: New York City Update
Type: Outage
Severity: Medium
Affected Zone: NYC:111 8th Avenue:XO
Please note that all devices located in this zone may be impacted
Message:
Hurricane Update: 111 8th Ave, NYC
**EMERGENCY NOTIFICATION**
As of 6AM EST today, ColoCrossing's facility at 111 8th Avenue was transferred to generator power as a result of the ConEd power shut off. At this time, the facility has an ample supply of Diesel fuel on site as well as contractors to refuel the tanks as needed.
At this time we do not expect further service interruptions in this zone.
Information related to Hurricane Sandy can be found here: http://www.nhc.noaa.gov
|
Lets hope no more downtime for my customers, however thanks to the massive news coverage everyones understanding. |
| Posted by dotHostel, 10-30-2012, 01:23 PM |
Quote:
|
Originally Posted by FRH Lisa
if you're going to offer redundancy to your customers, shouldn't [...]
Again, I'm not criticizing. I'm genuinely curious.
|
Please take a look at http://twitter.com/interserver |
| Posted by kidmuto, 10-30-2012, 02:17 PM |
hi,
does anyone know the status of atlantic metro? generators seemed to kick in OK last night but comm went down at some point.. they were updating their twitter feed with status but nothing in several hours
thanks |
| Posted by Kett2004, 10-30-2012, 02:38 PM |
|
^ It seems the generators at 121 Varick Street (LGA4) failed at 1AM EST. I have a server there and power has been out since then. Steadfast is updating their twitter feed every few hours and they have no ETA. |
| Posted by mctDarren, 10-30-2012, 02:47 PM |
|
Seeing signs of life with routing being restored to ezzi.net via nLayer, 1 box up atm. |
| Posted by dotHostel, 10-30-2012, 02:55 PM |
Quote:
|
Originally Posted by kidmuto
hi,
does anyone know the status of atlantic metro? generators seemed to kick in OK last night but comm went down at some point.. they were updating their twitter feed with status but nothing in several hours
thanks
|
Quote:
On Tuesday morning Atlantic Metro reported that it was experiencing “wide spread network and facility power outages due to Sandy and its flooding.”
|
Massive Flooding Damages Several NYC Data Centers |
| Posted by Kett2004, 10-30-2012, 02:59 PM |
Latest update from atlantic metro's twitter:
"LGA4 IDC Update: There is significant flooding in the generator room of this IDC. We are working with FDNY to safely investigate the area"
"LGA1 IDC Update: We are working with local building crew to clear the area of the fuel pumps needed for Diesel generator power." |
| Posted by kidmuto, 10-30-2012, 03:00 PM |
Quote:
|
Originally Posted by Kett2004
^ It seems the generators at 121 Varick Street (LGA4) failed at 1AM EST. I have a server there and power has been out since then. Steadfast is updating their twitter feed every few hours and they have no ETA.
|
i'm at 325 hudson. i'm think their generators kicked in ok but the network links into the building were down. |
| Posted by kidmuto, 10-30-2012, 03:03 PM |
Quote:
|
Originally Posted by Kett2004
Latest update from atlantic metro's twitter:
"LGA4 IDC Update: There is significant flooding in the generator room of this IDC. We are working with FDNY to safely investigate the area"
"LGA1 IDC Update: We are working with local building crew to clear the area of the fuel pumps needed for Diesel generator power."
|
does anyone know which datacenter is 325 hudson? LGA_? -thanks |
| Posted by dthani, 10-30-2012, 03:05 PM |
Quote:
|
Originally Posted by serversphere
Seeing signs of life with routing being restored to ezzi.net via nLayer, 1 box up atm.
|
I have 2 up at Ezzi.net so far. |
| Posted by Kett2004, 10-30-2012, 03:06 PM |
|
325 Hudson is LGA1 |
| Posted by mctDarren, 10-30-2012, 03:40 PM |
|
Spoke too soon, looks like ezzi went right down again. Wondering if this is an nLayer, Cogent, datacenter, or ezzi issue. Blargh. |
| Posted by jsw6, 10-30-2012, 04:13 PM |
Quote:
|
Originally Posted by FRH Lisa
if you're going to offer redundancy to your customers, shouldn't your fuel pumps / fuel tanks / AC panels / etc be well above ground?
|
It is a safety issue and one which the datacenters and building owners are not free to decide for themselves. NYC building code prohibits storing large amounts of fuel above-ground because it could endanger occupants or rescue personnel in the event of a fire. Building codes are similar in most cities where there are skyscrapers, so this is not an issue specific to NYC.
So what folks do is store some fuel on a service level with their generators and keep the majority of it in basements. It is not possible to draw fuel from them without pumps in the basement because a pump can only self-prime, or suck a liquid up a non-primed, dry pipe, a couple of floors before it would never be able to produce enough vacuum to get the pumping process started. That's why the pumps are also located in basements.
It's not inconceivable that fuel pumps could be located just above ground level and be able to self-prime and supply day tanks on upper-floors, but if you surveyed most high-rise datacenters you'd find maybe zero that use such a design.
Basically, these facilities are all outside of flood planning areas, they thought there was zero chance of flooding, and unfortunately they were not right. But it's building code restrictions causing them to keep their main fuel supplies in basements, and physics preventing them from being able to use pumps 13 floors up to draw fuel from those basements. |
| Posted by sirius, 10-30-2012, 06:10 PM |
|
Again... this thread is for updates on outages caused by the storms. Please keep your speculation/commentary/opinion about providers to yourself or if you're a customer, feel free to start a review thread in the main forum. |
| Posted by hussainjawadwala, 10-30-2012, 06:31 PM |
|
my servers are also not responding. No one really telling what is the issue. But as it is act of God, so we cannot blame the DC and their facility. They seem to suffer a lot due to this. |
| Posted by Damned_Kush, 10-30-2012, 06:47 PM |
Quote:
|
Originally Posted by dthani
It would be helpful if someone at Ezzi could take a second to post and let us know how severe the damage is. Other hosting companies are doing great with the real time updates. Even if we cant get something real time, a note to say what the status is every hour or two would be great.
|
Finally, An email came from Ezzi.net a few hours ago. It seems there isn't any server damage, but power outage couse problems.
Quote:
Regretfully, New York was hit by a very major storm and we suffered record-breaking damage as we result of this.
There are a large number of outages in the area and due to unprecedented flooding, we had issues with the facility generator systems.
We anticipate that the site will be up and running again within a few hours.
Ali Dhoon
EZZI.net
|
|
| Posted by hussainjawadwala, 10-30-2012, 07:07 PM |
Quote:
|
Originally Posted by Damned_Kush
Finally, An email came from Ezzi.net a few hours ago. It seems there isn't any server damage, but power outage couse problems.
|
Thanks for making such reply. It really helped me to keep hope. we were waiting for reply from Ezzi.
I pray everything gets better ASAP. |
| Posted by Atlas Global, 10-30-2012, 08:03 PM |
Quote:
|
Originally Posted by jzukerman
CallCentric is down right now. I didn't think to check them this morning when I got into the office, until now.
|
I've got various VoIP lines I use... CallCentric being one of them. Their service has been up and down for the past 2 weeks due to various DDOS attacks ... and now this disaster has taken them totally down. I can't help but feel sorry for them, they've had a very bad month. |
| Posted by mctDarren, 10-30-2012, 08:03 PM |
|
I again have 1 server up at ezzi.net while the others remain down. Sure seems like network issues to me. |
| Posted by mr fixit, 10-30-2012, 08:14 PM |
Quote:
|
Originally Posted by FliesLikeABrick
There is a lot of misinformation in this thread, but I wanted to correct this one in particular.
ConEd only shut down power south of wall street or 14th street intentionally, as a preventative measure
|
Most if not all NYC data centers are at 15th street or below... Telehouse Chelsea and 111 8th tenants all stated that they intentionally switched to generator ahead of the planned de-energized power systems. They were not going to wait for ConEd to pull the plug. Zayo expected the cutover due to this action and smoothly migrated to generator. Quite simply , many 111 Eighth tenants experienced downtime from generator problems. Data Centers which were not flooded should not have had generator problems regardless of ConEd. No one in 111 8th should have been down for any length of time. The 111 8th tenants under repair started to come back up around 10AM mostly there were circuits restored but some rack power was actually lost too. |
| Posted by hussainjawadwala, 10-30-2012, 09:59 PM |
Quote:
|
Originally Posted by serversphere
I again have 1 server up at ezzi.net while the others remain down. Sure seems like network issues to me.
|
What is your server IP which is UP?
i am having 66.199.x.x series and 72.9.x.x series |
| Posted by jfabregat, 10-30-2012, 11:58 PM |
|
In ezzi.net i have 1 server up and running on with IP 66.199.232.X but other in 66.199.242.X that i down, anyone have news on when the outage is going to be over, i saw someone saying +/- 3 hours. Please tell me if you know whats going on. |
| Posted by hussainjawadwala, 10-31-2012, 12:02 AM |
|
I am still getting RTOs ping status. Does anyone has confirm news from Ezzi? none of the ezzi team is making reply right now. |
| Posted by mctDarren, 10-31-2012, 12:22 AM |
Quote:
|
Originally Posted by hussainjawadwala
What is your server IP which is UP?
i am having 66.199.x.x series and 72.9.x.x series
|
66.199.x.x are up, 96.45.x.x are down for me |
| Posted by hussainjawadwala, 10-31-2012, 12:32 AM |
mine is still showing RTO.
66.199.236.x down
66.199.227.x down
96.45.70.x down |
| Posted by hussainjawadwala, 10-31-2012, 12:39 AM |
|
is any one able to contact Ezzi? |
| Posted by jfabregat, 10-31-2012, 12:41 AM |
Quote:
|
Originally Posted by hussainjawadwala
is any one able to contact Ezzi?
|
No, and they are not posting anything. If this is not up before morning im losing the clients, sad but ill keep waiting for service to come up. Leasson learned.... |
| Posted by HostXNow, 10-31-2012, 04:19 AM |
Quote:
|
Originally Posted by bqinternet
The worst of the storm may be past NYC now, but DCs are going to be running on generators for a while. I wouldn't be surprised to see more data centers lose power over the next few days.
|
... so it has stopped now.... watching SkyNews. |
| Posted by Damned_Kush, 10-31-2012, 04:46 AM |
Quote:
|
Originally Posted by hussainjawadwala
agreed.
currently, any update from Ezzi?
|
Sorry but no update after power outage mail. We are still waiting and our server is still offline |
| Posted by voip_expert, 10-31-2012, 05:12 AM |
To those who is effected by Ezzi:
The facility where is Ezzi has colo is very close to the water, in basement where generators are installed and Disel tanks are 8ft of water, facility Managment company is not able to power on generators due to this event, they are doing they best to get the water out and dry out the generators, if all goes well there is a big chances that facility will be up towards end of the day tomorrow. |
| Posted by voip_expert, 10-31-2012, 05:14 AM |
|
Any one have any updates on 33 Whitehall? How many days will it take to come back up? Cogent was promising today by 6:30 PM EST but nothing have happen. |
| Posted by R1CH, 10-31-2012, 05:51 AM |
|
Internap / LGA6 just went down for me (looks like a network issue). |
| Posted by Jono20201, 10-31-2012, 05:57 AM |
|
111 8th Avenue just went down. |
| Posted by mr fixit, 10-31-2012, 06:01 AM |
|
111 Eighth is up here with various providers. But we are seeing issues with Internap in a few data centers right now, so perhaps they are working on something. Inap at 25 Broadway just lost connectivity. Lets hope it gets back up soon. |
| Posted by voip_expert, 10-31-2012, 06:04 AM |
|
Level 3 have been up all of the time at 111 8th without outages. |
| Posted by R1CH, 10-31-2012, 06:12 AM |
Just got this
Quote:
**EMERGENCY NOTIFICATION**
At approximately 05:10 EDT power to Internap's LGA9 datacenter was lost. At this time our power to customer equipment is exhausted and the facility is offline. We continue to work with the building personnel to correct the problem with the failed fuel pumps. We will provide more updates as information becomes available.
|
Quote:
|
At approximately 05:45 EDT connectivity to Internap's LGA11 datacenter was lost. This loss was caused by 2 other datacenter power failures that provide fiber to LGA11. We are working with the building personnel at both sites to correct the problems. We will provide more updates as information becomes available.
|
Seems like there are quite a few single points of failure in their infrastructure. |
| Posted by jNive, 10-31-2012, 06:52 AM |
|
confirmed, our systems in LGA8 are currently inaccessible |
| Posted by Jono20201, 10-31-2012, 06:53 AM |
Quote:
**EMERGENCY NOTIFICATION**
As of approximately 5:50AM EST today, ColoCrossing's facility at 111 8th Avenue is experiencing a network related outage as a result of metro fiber issues within lower Manhattan. Personnel are on site in all facilities working diligently to resolve this problem.
At this current time there is no ETA to resolution. Further updates will be provided as they come in.
|
I wonder what caused this. |
| Posted by jNive, 10-31-2012, 06:55 AM |
|
looks like a fiber conduit has been damaged perhaps? I doubt multiple NOCs/Peerings lost power simultaneously :-0 |
| Posted by jbfrompg, 10-31-2012, 06:55 AM |
|
Same for LGA9(Problem facility) and LGA11(Network issues because of LGA9). |
| Posted by jNive, 10-31-2012, 07:06 AM |
|
back up now at LGA8 - network issue only. |
| Posted by Jono20201, 10-31-2012, 07:07 AM |
|
We're back online in 111 8th. |
| Posted by mr fixit, 10-31-2012, 07:08 AM |
|
We just came back up with Pnap at Broadway. How is everyone else doing in the other locations? Perhaps all of those providers having issues are using Pnap fed from 75 Broad perhaps? |
| Posted by bqinternet, 10-31-2012, 07:12 AM |
Quote:
|
Originally Posted by mr fixit
We just came back up with Pnap at Broadway. How is everyone else doing in the other locations? Perhaps all of those providers having issues are using Pnap fed from 75 Broad perhaps?
|
Our Voxel LGA3 transit at 111 8th seems stable now. We also lost Atlantic Metro briefly, but it came back within a few minutes. |
| Posted by F-DNS, 10-31-2012, 07:13 AM |
Quote:
|
Originally Posted by R1CH
Seems like there are quite a few single points of failure in their infrastructure.
|
From what you quoted I make that TWO pathways to the DC... "This loss was caused by 2 other datacenter power failures that provide fiber to LGA11".
And they're back up   |
| Posted by R1CH, 10-31-2012, 07:23 AM |
Quote:
|
Originally Posted by F-DNS
From what you quoted I make that TWO pathways to the DC... "This loss was caused by 2 other datacenter power failures that provide fiber to LGA11".
And they're back up
|
Didn't realize there were two other failures at the time, only had that first email come in when we went down, figured they had just mixed up EST / EDT.
Seems like there are still some residual network issues, some routing loops to the Voxel DNS servers. |
| Posted by jbfrompg, 10-31-2012, 07:48 AM |
|
LGA8 is back up for us, but LGA9 is still down. |
| Posted by layer0, 10-31-2012, 07:59 AM |
Quote:
|
Originally Posted by jbfrompg
LGA8 is back up for us, but LGA9 is still down.
|
This is what I don't understand. Why is LGA9 affected at all? |
| Posted by jbfrompg, 10-31-2012, 08:02 AM |
|
Seems they are having fuel pump problems. |
| Posted by hussainjawadwala, 10-31-2012, 08:34 AM |
|
What's the status of ezzi.net any news ? |
| Posted by F-DNS, 10-31-2012, 09:14 AM |
Quote:
|
Originally Posted by hussainjawadwala
What's the status of ezzi.net any news ?
|
You can find out a bit by looking at their nameservers status...
s1.ezzi.net 66.199.247.130 is up.
s2.ezzi.net 72.9.98.42 is down but network goes most of the way to it till it hits 69.64.202.50.nyc.electricfiber.net (which is also the same route to NS1)
That suggests to me (only guesswork) that they have part of the network/power back but have more to repair. |
| Posted by hussainjawadwala, 10-31-2012, 09:44 AM |
|
Let's assume to be a bit good news. thanks for information. Do keeping updating of status . I was really worried. Hope that they get back online at earliest . |
| Posted by hussainjawadwala, 10-31-2012, 09:52 AM |
|
Also any update when they will be up ? |
| Posted by layer0, 10-31-2012, 10:11 AM |
This is what Internap/Voxel had to say about LGA9:
Quote:
|
We are currently having a power loss issue at our NYM008, otherwise known as LGA9, starting at approximately 5:00 AM EST, and our facility engineer are currently working to restore power asap. At 8:05 AM this morning our field techs have confirmed that the generators will be coming online shortly. Portions of the data center are slowly being brought back online right now.
|
Let's hope for the best. |
| Posted by mctDarren, 10-31-2012, 10:49 AM |
Quote:
|
Originally Posted by F-DNS
You can find out a bit by looking at their nameservers status...
s1.ezzi.net 66.199.247.130 is up.
s2.ezzi.net 72.9.98.42 is down but network goes most of the way to it till it hits 69.64.202.50.nyc.electricfiber.net (which is also the same route to NS1)
That suggests to me (only guesswork) that they have part of the network/power back but have more to repair.
|
I know some machines came back around 6:00PM yesterday. I'm guessing you're right, it must be a partial power/network problem since both up and down boxes there are routing exactly the same way. |
| Posted by dthani, 10-31-2012, 11:12 AM |
Quote:
|
Originally Posted by serversphere
I know some machines came back around 6:00PM yesterday. I'm guessing you're right, it must be a partial power/network problem since both up and down boxes there are routing exactly the same way.
|
Of the boxes that are back up on Ezzi, can we get some idea of if they're cloud or dedicated or a mixture? Trying to figure if they're prioritizing either or the other based on the power they have available. |
| Posted by mctDarren, 10-31-2012, 11:18 AM |
My personal server is up there, and it is a cloud server. Also, their web site just came back up with the following message:
EZZI.net Status Update
Dear Customers,
Tropical Storm SANDY caused an unprecedented amount of damage in and around New York City.
Two of our main locations 882 3rd Ave, Brooklyn and 75 Broad Street, New York were affected as a result of a level of flooding never seen before in New York.
A State of Emergency was declared and at this time, there are close to 8+ million homes and businesses affected.
Our team members have been working around the clock in an effort to restore any level of normalcy and bring customers online.
At this time, most of our Hosting Customer, who are located at the Brooklyn location are still affected by the loss of power to the facility, and we are working with all possible resources to try and have all systems back to normal during the course of the day today (Wednesday).
The generators are fed by fuel in storage tanks in the basements which were flooded, and caused the fuel pumps to fail. The facility owners have arranged for backup fuel tanks which are in the process of being attached to restore power to the facility.
Unfortunately, our offices within the Brooklyn facility are not usable and our offices in New York City as well as Long Island are affected.
We regret that you have been unable to reach us and as we assure you that we are making every effort to restore facilities as soon as humanly possible while ensuring the safety of our employees.
support@ezzi.net has been restored and you will be able to send us messages to this email.
We hope to have phones restored by sometime tomorrow (Thursday). |
| Posted by jfabregat, 10-31-2012, 11:27 AM |
|
Thank you for this information. |
| Posted by hussainjawadwala, 10-31-2012, 01:22 PM |
Quote:
|
Originally Posted by serversphere
My personal server is up there, and it is a cloud server. Also, their web site just came back up with the following message: EZZI.net Status Update
Dear Customers,
Tropical Storm SANDY caused an unprecedented amount of damage in and around New York City.
Two of our main locations 882 3rd Ave, Brooklyn and 75 Broad Street, New York were affected as a result of a level of flooding never seen before in New York.
A State of Emergency was declared and at this time, there are close to 8+ million homes and businesses affected.
Our team members have been working around the clock in an effort to restore any level of normalcy and bring customers online.
At this time, most of our Hosting Customer, who are located at the Brooklyn location are still affected by the loss of power to the facility, and we are working with all possible resources to try and have all systems back to normal during the course of the day today (Wednesday).
The generators are fed by fuel in storage tanks in the basements which were flooded, and caused the fuel pumps to fail. The facility owners have arranged for backup fuel tanks which are in the process of being attached to restore power to the facility.
Unfortunately, our offices within the Brooklyn facility are not usable and our offices in New York City as well as Long Island are affected.
We regret that you have been unable to reach us and as we assure you that we are making every effort to restore facilities as soon as humanly possible while ensuring the safety of our employees.
support@ezzi.net has been restored and you will be able to send us messages to this email.
We hope to have phones restored by sometime tomorrow (Thursday).
|
That's great news . All the best ezzi |
| Posted by jbfrompg, 10-31-2012, 02:25 PM |
|
Just saw LGA9 come back up. I wonder if it will stay up. |
| Posted by Lancia, 10-31-2012, 02:27 PM |
|
My server in LGA9 is still not up yet. |
| Posted by bigtimbers, 10-31-2012, 02:50 PM |
interesting to see the number of networks Sandy brought down. not sure what they are defining as a "network," but they are showing about 1,250 down at peak in NY alone:
http://www.renesys.com/blog/assets_c...ages-751.shtml |
| Posted by cooldude919, 10-31-2012, 03:00 PM |
Quote:
|
Originally Posted by bigtimbers
|
Anyone with a BGP AS maybe? If you notice it says source of BGP data, so they are using that somehow to get the numbers. |
| Posted by Javo94, 10-31-2012, 03:10 PM |
Just wanted to give an update on all Telx facilities presently:
Clifton, NJ is on utility and in normal operation.
111 8th is on generator but all systems are operating normally at this time. There currently is a 48 hour fuel supply and daily fuel deliveries are scheduled.
60 Hudson is on generator but all systems are operating normally. Electricians are onsite to rewire 2 more HVAC units to generator-backed panel for redundancy and temperatures are well within allowable ranges in all suites. We have daily fuel deliveries scheduled there as well.
We are adequately staffed at this time at all locations.
No ETA on utility power restoration at any NYC facility. |
| Posted by jNive, 10-31-2012, 04:11 PM |
|
Many thanks for the update at facility-level and ConEd ETAs. Hopefully those that are currently offline don't suffer too much disruption. |
| Posted by hussainjawadwala, 10-31-2012, 04:38 PM |
Any sign of progress ? After ezzi announcement there site is again down.
really worried sitting overseas |
| Posted by Javo94, 10-31-2012, 04:48 PM |
|
75 Broad st is down pretty hard right now. Do you know which data center they are in the building? Is it Internap or Peer 1? |
| Posted by voip_expert, 10-31-2012, 04:49 PM |
Quote:
|
Originally Posted by hussainjawadwala
Any sign of progress ? After ezzi announcement there site is again down.
really worried sitting overseas
|
For sure it want happen today... so you better sit tight and simply wait. |
| Posted by mctDarren, 10-31-2012, 04:56 PM |
Quote:
|
Originally Posted by hussainjawadwala
Any sign of progress ? After ezzi announcement there site is again down.
really worried sitting overseas
|
Still up here for me. Might be a DNS error. |
| Posted by bigtimbers, 10-31-2012, 05:07 PM |
Quote:
|
Originally Posted by Javo94
Just wanted to give an update on all Telx facilities presently:
Clifton, NJ is on utility and in normal operation.
111 8th is on generator but all systems are operating normally at this time. There currently is a 48 hour fuel supply and daily fuel deliveries are scheduled.
60 Hudson is on generator but all systems are operating normally. Electricians are onsite to rewire 2 more HVAC units to generator-backed panel for redundancy and temperatures are well within allowable ranges in all suites. We have daily fuel deliveries scheduled there as well.
We are adequately staffed at this time at all locations.
No ETA on utility power restoration at any NYC facility.
|
how'd your weehawken facility do, given the extensive damage just to the south in hoboken and jersey city? |
| Posted by Javo94, 10-31-2012, 05:15 PM |
Here is the response at 300 Blvd East.
Tech Support/ Telx NOC,
Here is the latest update from 300 Blvd East.
"We are still on Generator and stable." |
| Posted by Javo94, 10-31-2012, 05:24 PM |
|
One customer is reporting a hole in the roof at Equinix? Can anyone confirm this? |
| Posted by Kett2004, 10-31-2012, 05:57 PM |
LGA4 is a mess...
Looks like the first chance of having generator power is sometime tomorrow.
It looks like it's time one again to find a "reliable provider" in NYC.
The latest updates from atlantic metro's twitter feed of LGA4:
"#LGA4 Update - Cummings Generator team has determined that there is un salvageable damage to the current genset. We've secured a rollup."
"#LGA4 Update - We've secured a temporary rollup generator for LGA4 that is currently set to arrive in Manhattan tomorrow. updates to follow " |
| Posted by inerail-chris, 10-31-2012, 06:03 PM |
Quote:
|
Originally Posted by Javo94
One customer is reporting a hole in the roof at Equinix? Can anyone confirm this?
|
Which location? |
| Posted by bqinternet, 10-31-2012, 06:10 PM |
Quote:
|
Originally Posted by Kett2004
It looks like it's time one again to find a "reliable provider" in NYC.
|
One with generators that can't fail? Plenty of people will be interested to know when you find that provider. |
| Posted by mctDarren, 10-31-2012, 06:14 PM |
Quote:
|
Originally Posted by bqinternet
One with generators that can't fail? Plenty of people will be interested to know when you find that provider.
|
What we need is those big hamsters in those car commercials. Suckers seem big enough to power a building.
So, Varrick is down for the night. Anyone hear about 882 Third Ave and their generators? |
| Posted by Kett2004, 10-31-2012, 06:15 PM |
Quote:
|
Originally Posted by bqinternet
One with generators that can't fail? Plenty of people will be interested to know when you find that provider.
|
I understand generators can fail, but it seems like the LGA4 facility was getting second class service by AMC. It takes two days to determine the generator it broken?
Also a redundant genset can minimize the impact of a single genset biting the dust. |
| Posted by IdleServ, 10-31-2012, 06:15 PM |
|
My server at Ezzi came back to life for the first time since the power loss 8 minutes ago. |
| Posted by mctDarren, 10-31-2012, 06:19 PM |
Quote:
|
Originally Posted by IdleServ
My server at Ezzi came back to life for the first time since the power loss 8 minutes ago.
|
Cloud or Dedi, and what IP range? |
| Posted by jNive, 10-31-2012, 06:21 PM |
|
LGA8 is off the net again. Time for that splash page to go back up :-) |
| Posted by IdleServ, 10-31-2012, 06:26 PM |
Quote:
|
Originally Posted by serversphere
Cloud or Dedi, and what IP range?
|
Dedicated, 66.199.251.26 |
| Posted by mctDarren, 10-31-2012, 06:29 PM |
|
Thanks, you didn't have to post your whole IP! 66.199.x.x would have been good. |
| Posted by IdleServ, 10-31-2012, 06:30 PM |
Quote:
|
Originally Posted by serversphere
Thanks, you didn't have to post your whole IP! 66.199.x.x would have been good.
|
Don't see why I would need to hide it |
| Posted by jNive, 10-31-2012, 06:33 PM |
|
issue was pretty short-lived thankfully :-) network-specific again. |
| Posted by voip_expert, 10-31-2012, 06:34 PM |
|
882 3rd ave is up. |
| Posted by mctDarren, 10-31-2012, 06:38 PM |
Quote:
|
Originally Posted by voip_expert
882 3rd ave is up.
|
Thanks, I guess ezzi must be in 75 Broad then. On their web site they had both addresses. And that is where they are still having geny probs, right? |
| Posted by voip_expert, 10-31-2012, 07:00 PM |
|
Internap (Voxel) experiencing issue with network at LDJ1. |
| Posted by bqinternet, 10-31-2012, 07:02 PM |
Quote:
|
Originally Posted by voip_expert
Internap (Voxel) experiencing issue with network at LDJ1.
|
More than that:
Quote:
|
We are currently investigating a connectivity issue at our LGA6, LGA8, and LGA9 facilities, located at 111 8th Avenue in New York, and our LGA11 facility located at 75 Broad Street. At approximately 18:22 we were alerted to a loss of response from the Internap NYM007 PNAP. Customers downstream of this facility, including Agile hosting customers in New York, may currently be experiencing suboptimal routing while routes reconverge and traffic is moved to our other upstream providers in the New York area. We will provide an update as soon as we have additional information.
|
Our Voxel connectivity at LGA3 in 111 8th has been down for about 45 minutes. |
| Posted by IRCCo Jeff, 10-31-2012, 07:18 PM |
Quote:
|
Originally Posted by voip_expert
Internap (Voxel) experiencing issue with network at LDJ1.
|
Our Ubersmith instance is at LDJ1, the outages have been very brief and quite tolerable. |
| Posted by XeHSean, 10-31-2012, 07:19 PM |
|
WeLinkNYC also appears to be down - anyone have any info? |
| Posted by dthani, 10-31-2012, 07:27 PM |
|
I got 2 up at Ezzi, 96.45.*.* cloud and 66.199.*.* dedicated. |
| Posted by inerail-chris, 10-31-2012, 07:28 PM |
Quote:
|
Originally Posted by XeHSean
WeLinkNYC also appears to be down - anyone have any info?
|
I know that they have some equipment in 121 Varick (Atlantic Metro LGA4). Power will be down until tomorrow morning for this location. |
| Posted by hussainjawadwala, 10-31-2012, 07:33 PM |
I got 2 up at ezzi 66.199.236.x and 66.199.229.x
waiting for 3 dedicated and 1 cloud .
any one's cloud on ?
Ip series
96.45.x.x ? |
| Posted by voip_expert, 10-31-2012, 07:40 PM |
Quote:
|
Originally Posted by IRCCo Jeff
Our Ubersmith instance is at LDJ1, the outages have been very brief and quite tolerable.
|
For us it all end's right here 0.te6-2.tsr1.lga3.us.voxel.net |
| Posted by WII-Aaron, 10-31-2012, 07:43 PM |
Quote:
|
Originally Posted by IRCCo Jeff
Our Ubersmith instance is at LDJ1, the outages have been very brief and quite tolerable.
|
Same here. Intermittent routing issues, etc. All to our Ubersmith instance, all brief. |
| Posted by mctDarren, 10-31-2012, 08:18 PM |
Quote:
|
Originally Posted by dthani
I got 2 up at Ezzi, 96.45.*.* cloud and 66.199.*.* dedicated.
|
Same here, all servers up now. |
| Posted by mctDarren, 10-31-2012, 08:19 PM |
Quote:
|
Originally Posted by hussainjawadwala
I got 2 up at ezzi 66.199.236.x and 66.199.229.x
waiting for 3 dedicated and 1 cloud .
any one's cloud on ?
Ip series
96.45.x.x ?
|
Yep, two just came back a bit ago. |
| Posted by mr fixit, 10-31-2012, 09:49 PM |
Internap at Telehouse Chelsea (85 Tenth Avenue) has seemed to be working fine throughout the whole hurricane. I wonder where that is being fed from. Does anyone know how come this area is not affected?
Telehouse is giving companies people temporary colocation space for as little as a months time, and they have gotten refueling trucks ready to refill their generators.
Telehouse overall has really been solid at all their sites during this whole crisis. I had thought perhaps Staten Island or 25 Broadway would have been hit since they were vulnerable areas but we have not seen any problems yet there.
60 Hudson already refueled the meet-me room this morning I think, and they are doing good no hiccups. 111 Eighth has trucks ready with thousands of gallons of diesel fuel waiting to refill. |
| Posted by jsw6, 10-31-2012, 11:01 PM |
Quote:
|
Originally Posted by mr fixit
Internap at Telehouse Chelsea (85 Tenth Avenue) has seemed to be working fine throughout the whole hurricane. I wonder where that is being fed from. Does anyone know how come this area is not affected?
|
It is affected. They have no utility power at that building and it is unclear when it will be restored.
I think you should look at a map so you can better understand where the datacenters you have mentioned on this thread are located, in reference to what neighborhoods do and don't have any utility electricity. Both 85 Tenth Ave and 111 8th Ave are in Chelsea, about two blocks apart. Both have no utility power. Both lost power due to the unexpected sub-station failure that darkened the rest of Lower Manhattan (with the exception of the Wall Street area which was intentionally shut off slightly earlier.) ConEd has retracted their guess of Saturday for when these areas will be restored and are now being very vague, so it is really tough to know how much longer these facilities will need to be on generator.
Obviously the problem with that is, the longer these facilities are on generator, the higher the risk that their generators will begin failing. The reason places often have N+2 generators instead of N+1 is exactly because of this. They aren't prime movers that are meant to be run for weeks or months without failure or maintenance.
Basically, it is a good time to make sure your backups are up-to-date and your DR plan is ready, even if your provider is currently up and doesn't foresee themselves going down. That's a good plan no matter where you are hosted, but it's especially good and timely if you are hosted in the NY/NJ area. |
| Posted by bqinternet, 10-31-2012, 11:46 PM |
|
We just got our Voxel LGA3 connectivity back at 111 8th. I've heard reports of other carriers in the building coming back online. |
| Posted by mr fixit, 10-31-2012, 11:47 PM |
|
Hi sorry to be unclear. I wasn't referring to affected area in terms of electricity, I was asking if someone knows why PNAP transit is working at 85T and not in other buildings such as 111, and which site it might be fed from. |
| Posted by voip_expert, 11-01-2012, 12:29 AM |
Telehouse at 7 Teleport is true disaster recovery facility no matter what that place always will be up, and that the reason for paying premium price in general over there.
Does any one have any word on 33 Whitehall, Cogent not letting any one in, we need HDD's from it ASAP. |
| Posted by FHDave, 11-01-2012, 01:03 AM |
Quote:
|
Originally Posted by WII-Aaron
Burstnet, one of the most budget of budget providers, got hit by the storm also. They didn't go offline.
|
My house in Philly's suburb also survived the Huricane better than some of these data centers. But guess what? It means absolutely nothing ...
You are comparing apples and peanuts. Bursnet is not on Sandy's direct path nor close to it like some of these affected data centers.
My thoughts are with all those affected, providers and customers. |
| Posted by jameshurrell, 11-01-2012, 06:04 AM |
Read this blog about 75 Broad Street and the efforts made by some providers (Peer1, SquareSpace and Fog Creek) to keep going.
They are carrying drums of diesel up to the 17th floor of the DC......
http://status.squarespace.com/ |
| Posted by bigtimbers, 11-01-2012, 08:02 AM |
|
good to hear. word was that 300 went down, but rumors are flying fast and thick... |
| Posted by bigtimbers, 11-01-2012, 11:32 AM |
Quote:
|
Originally Posted by Javo94
One customer is reporting a hole in the roof at Equinix? Can anyone confirm this?
|
i can't definitively confirm, but just heard word of a partial roof collapse at one of their secaucus buildings. again, rumors are fast and thick. |
| Posted by bigtimbers, 11-01-2012, 12:32 PM |
Not rumor: http://www.zayo.com/emergency-response
Letter to Customers Regarding Service at 111 8th Ave.
Valued Zayo Customer,
This is an urgent communication regarding the status of the 111 8th Avenue facility in New York. At this time, there are technical difficulties with the fuel pumping system to a subset of generators within the building, affecting the AC power supplied to Zayos space on the 7th floor. While the networking equipment remains powered at this time, the reduced power generation has impacted the cooling systems in this area of the building and the rising temperatures present a risk to networking equipment owned by Zayo and collocated customers.
Zayo is taking emergency measures to mitigate this issue which includes deployment of fans within the facility to eliminate hotspots, powering down any equipment that is not currently supporting live customer traffic, and evaluating the next steps for further load reductions which will include powering down redundant networking equipment and cards to operate in a simplex condition consuming less power. These measures, if required, may impact IP routing performance and result in suboptimal routing, or potential packet loss during peak traffic periods but may be necessary to prevent service outages or otherwise have redundant equipment remained undamaged to expedite recovery following a worst case scenario. The current ambient temperature in the 7th floor space is 93 degrees and slowing rising and we will continue to do everything within our power to prevent or otherwise minimize customer impact.
Please note that this situation currently does NOT impact customers collocated in suites 303 or 311A. We will continue provide updates via the website and via email as further information becomes available. |
| Posted by mr fixit, 11-01-2012, 01:09 PM |
|
At 111 8th avenue there are technical difficulties with the fuel pumping system to a subset of generators within the building, affecting the AC power supplied to space on the 7th Floor and temperatures are rising there. The issue is relatively isolated and not affecting the entire building. |
| Posted by virtuallynathan, 11-01-2012, 02:17 PM |
|
75 broad street is being evacuated until further notice by police order. |
| Posted by Jay Suds, 11-01-2012, 02:49 PM |
Quote:
|
Originally Posted by virtuallynathan
75 broad street is being evacuated until further notice by police order.
|
Yikes. Presumably there is a serious life safety concern? |
| Posted by jNive, 11-01-2012, 03:48 PM |
|
Not trying to jump to conclusion but I wonder if there is a risk from the water+electricity or perhaps diesel being ignited. |
| Posted by voip_expert, 11-01-2012, 05:31 PM |
Quote:
|
Originally Posted by jNive
Not trying to jump to conclusion but I wonder if there is a risk from the water+electricity or perhaps diesel being ignited.
|
The main risk is sea-salt water damaging power cable. And this is what been happening for the most part.
Any one have any updates on 33 Whitehall, We been promised 5 PM update, it is 5:30 still nothing.
Cogent not letting any one in to the building to take even HDD's, and it is been out of control since Sunday night. |
| Posted by nyjimbo, 11-01-2012, 05:43 PM |
Does anyone know the status of Transbeam ? I see their website is up but that doesn't necessarily mean anything about their network. I contacted their tech support and they tell me there are sporadic outages. If I ask them if they are down they don't answer that question. Their support voice line seems to be only taking messages and says they are having an outage.
I can't tell where their hardware is but I think its at 60 Hudson, in NYC. Their offices might be elsewhere in town.
Anybody know more ? |
| Posted by WII-Aaron, 11-01-2012, 06:02 PM |
We should all pay close attention. There's a lot to learn here with regards to disaster planning.
If I were closer I'd help man the bucket brigade. They're doing a great job keeping the fires lit so to speak. |
| Posted by jNive, 11-01-2012, 06:43 PM |
|
Gotta pitch in for a batch of 'Bucket Brigade 2012' or 'The Sandy Bucket Brigade of 2012' t-shirts for those guys! |
| Posted by XeHSean, 11-01-2012, 09:10 PM |
Quote:
|
Originally Posted by inerail-chris
I know that they have some equipment in 121 Varick (Atlantic Metro LGA4). Power will be down until tomorrow morning for this location.
|
Do you know any way to contact them? Their website is still down |
| Posted by PWSI Hosting, 11-01-2012, 09:34 PM |
New to this forum but seeing the impact of the storm, had to post. Our company offered a delay in billing if clients were from that area. Anyone see anything like that from other providers?
BTW Hello all.... |
| Posted by inerail-chris, 11-01-2012, 11:07 PM |
Quote:
|
Originally Posted by XeHSean
Do you know any way to contact them? Their website is still down
|
Afraid not. I've just seen their name on a couple of cabinets in that building. However power was restored to that building a few hours back. |
| Posted by Canter, 11-02-2012, 12:59 PM |
Quote:
|
Originally Posted by WII-Aaron
We should all pay close attention. There's a lot to learn here with regards to disaster planning.
If I were closer I'd help man the bucket brigade. They're doing a great job keeping the fires lit so to speak.
|
I know I have learned a lot paying attention to all this. Although we are not in flood area here, my diesel transfer pumps from the main tank to the day tank are on the floor of the lowest part of the basement. I am having them raised in the next couple weeks as the electrical motors cant run submerged. The pumps themselves are sealed. Also raising the starter batteries up high too. |
| Posted by CW Mike, 11-02-2012, 03:38 PM |
|
I wish I could help them out, but I'm from the UK. But I hope everyone that use our forum are ok and nothing bad has happened to them. It's the worst hurricane ever and we wasn't expecting it. |
| Posted by JackJameson, 11-02-2012, 04:22 PM |
Quote:
|
Originally Posted by MichaelDance
I wish I could help them out, but I'm from the UK. But I hope everyone that use our forum are ok and nothing bad has happened to them. It's the worst hurricane ever and we wasn't expecting it.
|
Hey Michael,
We can donate money to them through the red cross or the american red cross
I hope you are all okay guys! |
| Posted by flam316, 11-02-2012, 05:50 PM |
Quote:
|
Originally Posted by DocHostsWeb
Hey Michael,
We can donate money to them through the red cross or the american red cross
I hope you are all okay guys!
|
I'd suggest donating through AmeriCares instead of the Red Cross. More of your money goes to where it's needed in comparison to the Red Cross. I've worked with them before, good people over there. |
| Posted by mr fixit, 11-02-2012, 09:02 PM |
|
Con Ed is working right now to restore utility power to 111 8th Avenue, commercial power has already been restored to parts of the building. They've been working on this for a few hours. 3rd, 7th, 17th and some other floors, are on stable commercial power as of now. |
| Posted by r00ter, 11-02-2012, 09:17 PM |
Quote:
|
Originally Posted by XeHSean
Do you know any way to contact them? Their website is still down
|
If any WeLink customers are experiencing any issues with their service - they can email noc (at) welink.io - we have engineers standing by.
The website is now back online... apologies for the inconvenience. |
| Posted by Kevin Hillstrand, 11-02-2012, 11:11 PM |
Quote:
|
Originally Posted by mr fixit
Con Ed is working right now to restore utility power to 111 8th Avenue, commercial power has already been restored to parts of the building. They've been working on this for a few hours. 3rd, 7th, 17th and some other floors, are on stable commercial power as of now.
|
Confirmed, 111 8th ave has been running on generator power since Monday. Right now, ConEd is working on the power grid to transition everything back to commercial power. |
| Posted by r00ter, 11-02-2012, 11:24 PM |
Quote:
|
Originally Posted by FTN-Kevin
Confirmed, 111 8th ave has been running on generator power since Monday. Right now, ConEd is working on the power grid to transition everything back to commercial power.
|
111 8th Avenue is completely back on commercial power now. Some providers, such as Internap and Equinix, have already transitioned over to commercial power. Others may remain on generator power until they deem it time to switch over. |
| Posted by Kevin Hillstrand, 11-03-2012, 02:14 AM |
Quote:
|
Originally Posted by r00ter
111 8th Avenue is completely back on commercial power now. Some providers, such as Internap and Equinix, have already transitioned over to commercial power. Others may remain on generator power until they deem it time to switch over.
|
Yep sorry about that. Our technicians actually notified me a few minutes after my post that Internap LGA6 in 111 8th Ave was back online as ConEd restored the power grid.
I can confirm all of our gear in Internap is fully online and operational on commercial power at this time, and has been for the past couple hours Hopefully things will stay this way. |
| Posted by mr fixit, 11-03-2012, 02:20 AM |
|
LGA6 @ XO just lost power an hour ago again ~ 1:10AM but came back up. Not sure why this happened but it seems stable again now. Looks like its back on generator again.. |
| Posted by voip_expert, 11-03-2012, 01:48 PM |
|
It has been 6 days since 33 Whitehall have been down, and cogent simply saying "tomorrow" for past 6 days and not letting any one in to the building, |
| Posted by cooldude919, 11-03-2012, 02:27 PM |
Quote:
|
Originally Posted by voip_expert
It has been 6 days since 33 Whitehall have been down, and cogent simply saying "tomorrow" for past 6 days and not letting any one in to the building,
|
Well you look at that, then you look at all the updates other providers have been giving (Zayo for example http://www.zayo.com/emergency-response), i think it becomes clear who you want to be with in situations like this. In the aftermath i think several providers are going to have to step up in terms of their DR planning and update policy or else risk losing a lot of business to those who did well with this. |
| Posted by LoginTech, 11-03-2012, 02:52 PM |
Quote:
|
Originally Posted by voip_expert
It has been 6 days since 33 Whitehall have been down, and cogent simply saying "tomorrow" for past 6 days and not letting any one in to the building,
|
It's not Cogent that is disallowing access, it's the building, customers of Datagram, VZB, Cogent are not getting in because the building has no operational elevators or emergency lighting...though that is changing. |
| Posted by AdmoNet, 11-03-2012, 04:36 PM |
Quote:
|
In the aftermath i think several providers are going to have to step up in terms of their DR planning and update policy or else risk losing a lot of business to those who did well with this.
|
I think the main lesson to learn is to have a good DR plan for your business and locate critical systems in more than one location When there is a major disaster, or even smaller issues, its possible for the City/Fire Chief etc to restrict building access.
Datagram shouldn't be the one providing a DR strategy (unless that service was specifically purchased from them.) Datagram should learn from this, and take extra measures to keep the facility online or recover more quickly... I agree. The clients utilizing the datacenter should be the persons planning for a building outage and should understand that no building/infrastructure or location is totally exempt from acts of God and power outages etc... |
| Posted by layer0, 11-03-2012, 05:08 PM |
I have to take a moment to congratulate NYI.net for being one of the only facilities in downtown Manhattan to stay online for the entire outage. I'm going to have to strongly consider them for our future servers now. Kudos.
Voxel/Internap handled this pretty poorly IMO, and will not be receiving any more business from us. Yes, this was a disaster, but staying online wasn't completely impossible in my opinion. |
| Posted by mr fixit, 11-03-2012, 06:05 PM |
|
Telehouse 25 Broadway also didnt have any issues whatsoever and is still running fine on generator at this time. In regards to companies having a better strategy, having common sense to not build your data center in a hurricane flood zone would be a good place to start. |
| Posted by mr fixit, 11-03-2012, 06:14 PM |
111 Eighth while not perfect is almost completely outside all flood zones
http://farm7.static.flickr.com/6067/...dbc99ac6_o.jpg
There are also multiple fuel pumps, as well as ConEd service and generators on the roof, which can't get flooded.
Can anyone expound upon why all the outages at Equinix and all of these other facilities which did not get flooded related to faulty generators, and why have they been so unreliable? Equinix had two generator outages both at 111 8th as well as 60 hudson. |
| Posted by voip_expert, 11-03-2012, 10:20 PM |
Telehouse and Fibermedia stayed without outages.
33 Whitehall still down and no access to facility! I know multiple retail calling cards who have gone out of business becouse of this matter.
I'm sure there is way would be for Managment company of the building to allow groups of people let say 10 to go upstears and take hard drives. |
| Posted by voip_expert, 11-03-2012, 10:24 PM |
Quote:
|
Originally Posted by mr fixit
111 Eighth while not perfect is almost completely outside all flood zones http://farm7.static.flickr.com/6067/...dbc99ac6_o.jpg
There are also multiple fuel pumps, as well as ConEd service and generators on the roof, which can't get flooded.
Can anyone expound upon why all the outages at Equinix and all of these other facilities which did not get flooded related to faulty generators, and why have they been so unreliable? Equinix had two generator outages both at 111 8th as well as 60 hudson. |
Also TATA have gone down multiple times at 60 Hudson for hrs... |
| Posted by bqinternet, 11-03-2012, 10:26 PM |
Quote:
|
Originally Posted by mr fixit
Can anyone expound upon why all the outages at Equinix and all of these other facilities which did not get flooded related to faulty generators, and why have they been so unreliable? Equinix had two generator outages both at 111 8th as well as 60 hudson.
|
I would like to know that too. Multiple colocation providers at 111 8th had outages, and some of them had multiple outages. They're supposed to have anticipated multi-day utility outages, so it seems like there's a design flaw somewhere. This wasn't just some isolated issue. |
| Posted by voip_expert, 11-03-2012, 11:45 PM |
|
Level3 did not have any outage at 111 8th Ave |
| Posted by bqinternet, 11-04-2012, 03:01 AM |
Quote:
|
Originally Posted by voip_expert
Level3 did not have any outage at 111 8th Ave
|
Either did our Telx space on the 15th floor. |
| Posted by Crsr, 11-04-2012, 11:21 AM |
Quote:
|
Originally Posted by cooldude919
Well you look at that, then you look at all the updates other providers have been giving (Zayo for example http://www.zayo.com/emergency-response), i think it becomes clear who you want to be with in situations like this. In the aftermath i think several providers are going to have to step up in terms of their DR planning and update policy or else risk losing a lot of business to those who did well with this. |
IMHO. Is not a matter of DR from DC Operators. Are the customers who need to step up to plate and develop DR and the most important: BC against major outages.
DC's only provide resiliency and concurrency against power failures, but they cannot do anything against major floods and massive (and lengty) power failures like those. Is the customer the ONE who needs to put their own BC systems.
I think that people will have learned the lesson and they will start thinking on the other important half of the equation (and majorily neglected and ignored): Business Continuity.
DC owners and their customers will need to learn a lesson from this dreadful past days and need to give one or two thinking sessions on what happened and what could be done to avoid it.
What I think is that 90% of the existing datacenters aren't ready to face a blackout of more than 16-20 hours without some titanic effors like diesel bucket squad (my hat off to them: Bravo for their pain and suffering). |
| Posted by voip_expert, 11-04-2012, 01:29 PM |
|
Did any one was able to get up on 33 whitehall to remove they HDD's? If so please share the steps was taking... cogent simply ditching calls off now. |
| Posted by AgServer, 11-04-2012, 03:33 PM |
Datagram just posted this update on their website (http://www.datagram.com)
11/4 13:55: The owner of 33 Whitehall, Stawski Partners http://www.stawskipartners.com is not allowing us to turn on the water pumps, they are too lazy to go up stairs to check water levels without an elevator so instead they would just assume not having them. They have had more than ample time to get their elevator vendor in to connect to our generator. We offered to go upstairs and place a man there 24x7, or pay anyone $1000 per trip from the building or otherwise and they refused. Verizon has done the same and are working with management to get this straitened out. Our attorneys are doing the same. Stawski Partners has done as much as they can to get in our way of recovery as well as all of the other tenants. They have refused our pumps for clean up and water removal, they are not working 24x7 or with any extra crew, they refused offers for external fuel pumps to refuel the roof, they refused to get a generator for the building - we had to get one for them at our cost. Datagram, Verizon and our contractors have been here 24x7 pushing the building to move faster and doing their work for them. |
| Posted by voip_expert, 11-04-2012, 03:49 PM |
Quote:
|
Originally Posted by AgServer
Datagram just posted this update on their website ( http://www.datagram.com)
11/4 13:55: The owner of 33 Whitehall, Stawski Partners http://www.stawskipartners.com is not allowing us to turn on the water pumps, they are too lazy to go up stairs to check water levels without an elevator so instead they would just assume not having them. They have had more than ample time to get their elevator vendor in to connect to our generator. We offered to go upstairs and place a man there 24x7, or pay anyone $1000 per trip from the building or otherwise and they refused. Verizon has done the same and are working with management to get this straitened out. Our attorneys are doing the same. Stawski Partners has done as much as they can to get in our way of recovery as well as all of the other tenants. They have refused our pumps for clean up and water removal, they are not working 24x7 or with any extra crew, they refused offers for external fuel pumps to refuel the roof, they refused to get a generator for the building - we had to get one for them at our cost. Datagram, Verizon and our contractors have been here 24x7 pushing the building to move faster and doing their work for them. |
Guys down stairs don't take money!!! We offer $3000 cash on spot to let us upstairs to take our hdd's they refused. |
| Posted by WII-Aaron, 11-04-2012, 08:49 PM |
What a cluster.
Things happen to everyone. It's not about if but when it'll happen and about how you handle it when it does. It looks to me like building management is bordering on brain damaged. I would move away from the building asap once you can. |
| Posted by Orien, 11-05-2012, 01:55 PM |
Quote:
|
Originally Posted by bigtimbers
|
Thanks for sharing that! It's been added to the thread summary |
| Posted by jba256, 11-06-2012, 12:48 AM |
Quote:
|
Originally Posted by voip_expert
Did any one was able to get up on 33 whitehall to remove they HDD's? If so please share the steps was taking... cogent simply ditching calls off now.
|
No joke about ditching calls. Two days of waiting on hold for 3+ hours with their non-stop auto-voice telling me to leave a message and no response - pretty clear they're just not answering the phone anymore. I'm lucky it's just a gig-e circuit down for me, i'd be outraged if I had data or systems colo'ed with them. Good luck to all of you. |
| Posted by flecom, 11-06-2012, 01:16 AM |
Quote:
|
Originally Posted by mr fixit
generators on the roof, which can't get flooded.
|
just want to point out that as demonstrated by hurricane Katrina putting the generator on the roof opens you up to other issues...
the main emergency communications system in new orleans was backed up by a roof mounted generator and the storm winds were so strong that they were pushing the radiator fan blades, sending the generator into overspeed and the generators controls (the governor) went into alarm and shut it down...
also roof mounted generators don't help you if your fuel and fuel pumps are in the basement underwater... |
| Posted by ISPrime_dimi, 11-06-2012, 10:08 AM |
|
Hi guys. I posted an article about these happenings in another thread (showthread.php?t=1206846), but thought it would be relevant here as well. 'sandy.isprime.com' really helps you grasp what happened to the Internet on the day of and those following the storm. Cheers |
| Posted by mctDarren, 11-06-2012, 10:38 AM |
|
At this point, since it would seem some DCs are back on the grid, can someone 'in the know' post an update as to who is still on generators and who is on utility power? |
| Posted by inerail-chris, 11-06-2012, 12:43 PM |
On Utility Power:
-60 Hudson (telx, equinix, others)
-111 8th / 76 9th (equinix, telx, internap, xo, others)
-25 Broadway (telehouse)
-85 10th (telehouse, level3)
-325 Hudson (AMC)
-100 William (NYI)
-75 Broad (peer1, internap, others)
-121 Varick (AMC, steadfast)
-32 AoA (coresite, thehub)
-395 Hudson (navistie)
The one building that I would like to know about is 33 Whitehall. I know they don't have utility power back yet, but no one seems to know exactly what the roll up generators are powering. Management really didn't handle it well, but thankfully I don't have any equipment there. |
| Posted by voip_expert, 11-06-2012, 01:15 PM |
Quote:
|
Originally Posted by ISPrime_dimi
Hi guys. I posted an article about these happenings in another thread (showthread.php?t=1206846), but thought it would be relevant here as well. 'sandy.isprime.com' really helps you grasp what happened to the Internet on the day of and those following the storm. Cheers
|
You have stated in your blog that Telehouse was down, this is not true! They where on generators always without a single issue. So get your facts straight. |
| Posted by IRCCo Jeff, 11-06-2012, 07:35 PM |
Also, Equinix claims to have not had any outages.
On the subject of electric infrastructure in the basement, I was just listening to a segment on NPR about how the NYC building code was only recently amended to allow generators to be placed elsewhere on the building (eg. roof). |
| Posted by voip_expert, 11-06-2012, 07:44 PM |
|
33 Whitehall Down Again!!! for over an 1 hr, cogent does not give any updates. |
| Posted by dthani, 11-07-2012, 02:40 PM |
|
Ezzi down again? All servers went offline about 20 mins ago. Anyone have any info? |
| Posted by mctDarren, 11-07-2012, 02:42 PM |
|
Same here, was wondering if just them or whole DC. I talked to them this morning and they said they were going to stay on gennys through this next storm to avoid problems. Hope I didn't jinx it. :| |
| Posted by dthani, 11-07-2012, 02:47 PM |
ahh so it was you!
doesnt help when the phones dont work either, I tried calling to get some info but I'm just getting the recording. Let me know if you hear anything. |
| Posted by mctDarren, 11-07-2012, 03:20 PM |
|
It was me! Whoops. Their site is back, hopefully the rest to follow shortly. |
| Posted by IRCCo Jeff, 11-07-2012, 05:17 PM |
Quote:
|
Originally Posted by bqinternet
|
I was going by their Twitter where they were bragging to be uninterrupted. |
| Posted by bqinternet, 11-07-2012, 10:27 PM |
Quote:
|
Originally Posted by jzukerman
Peer1 says it is still on generator power at 111 8th (I think that's correct)
|
I assume that they're talking about 75 Broad. |
| Posted by mr fixit, 11-07-2012, 11:19 PM |
Telehouse had a small issue in 25 Broadway but it did not impact customers. All their sites were 100% up from what I understand. Though I am not sure about Teleport? They were very prepared for this storm and planned cut-overs very nicely. A lot of constant email updates too.
I think that ISPrime post was nice, but seems a bit confusing because NYIIX is not completely in 111 8th, they just have a node at PAIX (734A) and Zayo (734B) is next door, both of which I believe were affected by the faulty fuel pump. NYIIX is headquartered at 25 Broadway but had no issues there. Were there issues with the NYIIX node at Equinix? Can anyone clarify? |
| Posted by mripguru, 11-08-2012, 01:01 AM |
Quote:
|
Originally Posted by voip_expert
33 Whitehall Down Again!!! for over an 1 hr, cogent does not give any updates.
|
When was this? What specifically was affected - power or connectivity? |
| Posted by IRCCo Jeff, 11-10-2012, 03:50 PM |
Looks like 33 Whitehall is still messed up (from Cogent):
Quote:
|
Due to the widespread outages in New York and New Jersey caused by Hurricane Sandy, Cogent is experiencing higher than normal call volumes and wait times. For customers at our New York hosting center at 33 Whitehall: Customer server reboots and limited access is being allowed by building management and will be available starting at 9AM ET, up until 5pm ET. If you need to have your server rebooted, please send a request to support@cogentco.com with a subject of "33 Whitehall Reboot Request". To request access you must send email to support@cogentco.com with a subject of "33 Whitehall Access Request". Building occupancy is limited to five individuals per escort interval, so only one representative per company at a time is allowed. This person will be allowed access for one hour. Customers will be assigned an access slot based on the order in which their email request was received. Customers will be notified via email of their assigned time. There are currently two generators backing each other up at the 33 Whitehall Hosting Center. The roof top generator at 33 Whitehall is once again fully operational and is being used to back-up the roll up generator. If you are an Off Net customer of Cogent in the lower Manhattan area, Verizon has reported wide spread outages due to flooding of their central offices that are impacting many services. There is no estimated time to repair. There are still multiple buildings in New York City without power that have been impacted by flooding. We are working to restore service as soon as possible. The master case for New York is HD5233575.
|
|
| Posted by mripguru, 11-10-2012, 04:23 PM |
Quote:
|
Originally Posted by IRCCo Jeff
Looks like 33 Whitehall is still messed up (from Cogent):
|
OUCH. That's harsh. |
| Posted by IRCCo Jeff, 11-10-2012, 05:26 PM |
|
What's harsh? |
| Posted by mripguru, 11-10-2012, 06:52 PM |
Quote:
|
Originally Posted by IRCCo Jeff
What's harsh?
|
The buildings/Cogent's policy - not to mention has Cogent not been fulfilling reboot requests for the past week? |
| Posted by IRCCo Jeff, 11-10-2012, 08:22 PM |
|
Well, from what I read here building management was not letting anyone in and now they're requiring escorts. I presume there is some type of super serious safety issue ongoing? |
| Posted by mripguru, 11-10-2012, 08:24 PM |
Quote:
|
Originally Posted by IRCCo Jeff
Well, from what I read here building management was not letting anyone in and now they're requiring escorts. I presume there is some type of super serious safety issue ongoing?
|
I'm not so sure - based on some of the stuff I was reading on Datagram's blog when this whole incident went down originally that basically said that the building management was dragging their feet and effectively hindering an effective restoration of services in a timely fashion.
This is likely why 33 Whitehall was one of the last buildings (DC wise) to come back up on generators. |
| Posted by IRCCo Jeff, 11-10-2012, 08:29 PM |
|
I would like to believe there is a very serious reason why. It's hard to believe that they just don't want to help. |
| Posted by mripguru, 11-10-2012, 09:33 PM |
Quote:
|
Originally Posted by IRCCo Jeff
I would like to believe there is a very serious reason why. It's hard to believe that they just don't want to help.
|
Well, I don't know any more than you do at this point - but, I suppose it could go either way. Though - I guess the question is - what would they gain from keeping work from getting done? |
| Posted by IRCCo Jeff, 11-10-2012, 09:49 PM |
Quote:
|
Originally Posted by mripguru
Well, I don't know any more than you do at this point - but, I suppose it could go either way. Though - I guess the question is - what would they gain from keeping work from getting done?
|
My guess was that there were very serious obstacles to commencing any type of work, and that the tenants were being dismissive of the fundamental problems. I can imagine a scenario where the landlord is trying to organize a recovery, where the DC tenants only care about running up the dark, damp stairs to retrieve data or plugin a generator, etc.
Pure speculation on my part, I just don't believe the situation is how the DC tenants are painting it. |
| Posted by mripguru, 11-10-2012, 09:51 PM |
Quote:
|
Originally Posted by IRCCo Jeff
My guess was that there were very serious obstacles to commencing any type of work, and that the tenants were being dismissive of the fundamental problems. I can imagine a scenario where the landlord is trying to organize a recovery, where the DC tenants only care about running up the dark, damp stairs to retrieve data or plugin a generator, etc.
Pure speculation on my part, I just don't believe the situation is how the DC tenants are painting it.
|
I guess the world may never know? |
| Posted by inerail-chris, 11-11-2012, 01:50 AM |
I can certainly see potential issues with just letting people run in (most of which could involve law suits).
However, it would be nice if building management did a better job communicating.
While I don't have any first hand experience with 33 Whitehall, I've heard from several people that crews have only been on-site M-F 9A-5P, which kind of sucks. |
| Posted by voip_expert, 11-14-2012, 01:44 AM |
|
33 Whitehall have been down today all day... they need to let people in to remove they hardware FYI since weeks no one able to access it. |
| Posted by Keliweb, 11-14-2012, 05:14 AM |
|
hum, i think it's all ok |
| Posted by smerz, 11-17-2012, 01:08 PM |
http://www.theverge.com/2012/11/17/3...urricane-sandy
Interessting read I found.
Shows some pictures from a Verizon building in lower manhatten.
Verizon is ditching their copper wires for fiber.
Including very convincing pictures as to why |
Add to Favourites 
Print this Article
Also Read
Shared Hosting
Shared Hosting
