Immedion Network Outage

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Immedion Network Outage

Posted by Snoork Hosting, 12-24-2011, 04:14 PM
Anyone experiencing network issues at Immedion Asheville facility?

The network seems to went down at 3:45 PM EST on our cabinets and remains down. The client portal for Netriplex technical support seems to be down as well.

Hopefully they can have it resolved ASAP.
Posted by PingRapid, 12-24-2011, 04:21 PM
I have equipment in Netriplex in Raleigh NC, and cannot access anything right now.
Posted by pubcrawler, 12-24-2011, 04:34 PM
Yeah, I can confirm they have an outage of some sort. Been going on for nearly 2 hours at this point.

Seeing 5-10% packet loss between Dallas and Asheville.

From Time Warners network, 100% loss.

Appears HE and Cogent connectivity / routing is nulled too.
Posted by Eleven2 Hosting, 12-24-2011, 04:37 PM
They are down here for us.
Posted by brc_csf, 12-24-2011, 04:37 PM
Seems like a routing problem but I am afraid it has already happened 2 days ago
Posted by brc_csf, 12-24-2011, 04:43 PM
Issues like this didn't happen when this IDC was Netriplex's.

How does the 100% SLA apply in situations like this, where most of the Internet can't reach the IDC ?
Posted by turbovps, 12-24-2011, 04:46 PM
Few of subnets are not reachable from Sweden but some are working fine
Posted by pubcrawler, 12-24-2011, 04:46 PM
Yes, things like this DID happen when it was Netriplex.

Actually very similar appearing thing happened around last Christmas when Netriplex routing became very odd and they were shipping packets all the way to LiNX in London for US bound requests.

That routing issue persisted until at least mid January. A full month or more of problems.
Posted by Prisma-Breck, 12-24-2011, 04:49 PM
There's been a few more incidents than normal lately it seems. I think it's a little too soon to tell if it's the new ownership or just coincidence. Time will tell, I guess. Hopefully it's just bad luck and bad timing.
Posted by techjr, 12-24-2011, 04:54 PM

Quote:

Originally Posted by brc_csf

Issues like this didn't happen when this IDC was Netriplex's.

How does the 100% SLA apply in situations like this, where most of the Internet can't reach the IDC ?

Things like this did happen with Netriplex and I would say some of the past issues were far worse. Had a power outage or two that lasted quite awhile and network issues. They were used for backups so not much of a big deal, did stop putting gear in the facility after the take over and I haven't heard anyone mention an increase of quality nor a decrease in quality so I'm not sure much has changed at all.

Can't answer the SLA question. Never do anything about SLA's myself.
Posted by pubcrawler, 12-24-2011, 05:02 PM
I hadn't noticed the issues lately as we have moved most sites away from this datacenter.

I have been noticing alarmingly high ping times for at least the past week to Asheville.

Ping times averaging 100-120ms from 6PM-10PM eastern.

Clearly, it appears to be a capacity issue on a Level 3 connection.

Today's issue appears to be the private connect/backhaul out of Asheville to Atlanta or DC (if they still have that). Saw this sort of thing in 2010 where Asheville went totally black for a while and had a week or more of strange routing when redundant bandwidth failed during an intended failover test.
Posted by Snoork Hosting, 12-24-2011, 05:04 PM
Also I would like to correct my first post. The network issues started occurring at 2:45 PM EST.

I'm currently seeing much less packet loss on the network than 30 minutes ago, but still unable to connect from Time Warner network.
Posted by dhancock, 12-24-2011, 05:16 PM
Immedion is experiencing an emergency event impacting the following components:

Date: 12/24/2011
Ticket Number: 605667
Start Time (EST): 2:36PM
End Time (EST): ----
Datacenter: AVL01
Components: Atlanta Network Paths
Impact: Emergency

Immedion support team will keep you informed of the progress during the event and
will notify you once it is complete. If you require further information please
contact Immedion Support via email at support@netriplex.com or via phone at
866-270-6000.

Sincerely,

The Immedion Support Team
100 Technology Drive, Suite C
Asheville, NC 28803
866-270-6000
Posted by brc_csf, 12-24-2011, 05:17 PM
Taking too long.. Hope they are near fixing it..
Posted by pubcrawler, 12-24-2011, 05:34 PM
I'd expect this to last for additional multiple hours based on prior experience there.

Makes no sense that they can't route around this outage. Clearly one or more links out of multiple went down. That's in part why facilities have different providers, routes, etc.

A degraded/slower network would be better than no routes to a number of major networks.
Posted by Flapadar, 12-24-2011, 05:38 PM

Quote:

Originally Posted by pubcrawler

I'd expect this to last for additional multiple hours based on prior experience there.

Makes no sense that they can't route around this outage. Clearly one or more links out of multiple went down. That's in part why facilities have different providers, routes, etc.

A degraded/slower network would be better than no routes to a number of major networks.

It's always possible they're getting hit by a huge ddos.
Posted by pubcrawler, 12-24-2011, 05:38 PM
Perhaps premature but just noticed their network spring back to life.

Routes seem to be back in place - can reach HE now.

Time Warner can reach Asheville again.

Still seeing packet loss, but lower...
Posted by Snoork Hosting, 12-24-2011, 05:39 PM
We're back online on our side (Time Warner Network), but still seeing some network packet loss generally outside the US.
Posted by brc_csf, 12-24-2011, 05:40 PM
It looks like as fixed.
Posted by brc_csf, 12-24-2011, 05:43 PM
Should we ask for our SLA credit? I never did it but I was wondering what is the point of having 100% SLA if you don't ever ask for credit?
Posted by pubcrawler, 12-24-2011, 05:44 PM

Quote:

Originally Posted by Flapadar

It's always possible they're getting hit by a huge ddos.

Well it's possible.

I'd expect them to have active monitoring and proactive nulling and relationships with upstream providers to aid in such.

Their facility has TONS of people renting server resources out in it and running all sorts of stuff. The network *HAS* to be a regular target. So they MUST be well versed and monitoring or we'd see these big outages all the time.

I think it's a human that likes to tinker with their network during holiday/low use times myself.

The outage wasn't a dropped cable or broken fiber. It was up and down and variable on packet loss - 1-10%. Then it went dark - 100% loss. Then sprang back to life.
Posted by Snoork Hosting, 12-24-2011, 05:54 PM
Everything seems to back to normal and no packet loss at the moment
Posted by WebManagerNY, 12-24-2011, 05:58 PM
Does it seem odd to anyone else that the Netriplex/Immedion support page (as of 4:55pm (EST)), available at, http://www.netriplex.com/support/ makes no mention of this known network emergency event?

It clearly states on their website that this support page, "It is always up-to-date and will give you information about any problem that we may already be aware of. It will also provide you with information on planned maintenance and outage."

What's the value of this support page if it is not being updated for known network emergency events?

WebManagerNY
Posted by pubcrawler, 12-24-2011, 06:10 PM

Quote:

Originally Posted by WebManagerNY

Does it seem odd to anyone else that the Netriplex/Immedion support page (as of 4:55pm (EST)), available at, http://www.netriplex.com/support/ makes no mention of this known network emergency event?
WebManagerNY

I jumped all over Netriplex over their lack of information during outages last year.

Netriplex said they would improve notification after that. They did a tad bit better and actually sent out post-incident summary once or twice, but failed to mention the many smaller outages that go unnoticed by many.

Immedion hasn't read or learned from prior incidents at Asheville.

Immedion's website and Twitter feed fail to mention anything about this.
Posted by dhancock, 12-24-2011, 06:17 PM
Going to keep my backup servers online for a while until I get notification from Immedion Asheville, but it appears that service has been restored.
Posted by pubcrawler, 12-25-2011, 09:17 AM
Unsure how other folks are faring after this outage...

Just noticed our route between Asheville and Dallas is still experiencing packet loss.

Running 3-4% packet loss right now.

Anyone else experiencing the same?
Posted by Snoork Hosting, 12-25-2011, 09:35 AM
Not seeing any packet loss on the network at the moment. Everything seems to be fine on our side.