Netriplex/Uberbandwidth down

Portal Home > Knowledgebase > Articles Database > Netriplex/Uberbandwidth down

Posted by ishan, 10-04-2010, 12:23 AM
Hello, They went down around 6 minutes ago. Can't get to the Netriplex Portal. Does anyone have any further information ? Ishan
Posted by Eleven2 Hosting, 10-04-2010, 12:26 AM
Down here for me as well...
Posted by ishan, 10-04-2010, 12:28 AM
Not that I am a network engineer, but traceroutes look bad -
Posted by Mike12King, 10-04-2010, 12:30 AM
Same issue. I'm trying to call them right now -- just looking for their number. Anyone know what's up?
Posted by Krazy, 10-04-2010, 12:31 AM
down here also
Posted by brc_csf, 10-04-2010, 12:31 AM
Down for 13 minutes.
Posted by Eleven2 Hosting, 10-04-2010, 12:31 AM
I am on hold with them now. Nice pretty music...
Posted by UH-Bobby, 10-04-2010, 12:32 AM
I've called also. They are thinking a fiber cut or something. They have engineers enroute.
Posted by UH-Bobby, 10-04-2010, 12:32 AM
I've got the number if anyone needs it.
Posted by Eleven2 Hosting, 10-04-2010, 12:32 AM
Lets hope its not a power outage and is just a quick network blip.
Posted by ishan, 10-04-2010, 12:32 AM
None of the traceroutes leave the country they are originating from, does anyone know what this means ? Ishan
Posted by Eleven2 Hosting, 10-04-2010, 12:37 AM
I just got off the phone, they said they are working on a way to route around this downed circuit.
Posted by Rogean, 10-04-2010, 12:38 AM
Looks like routing.. BGP Perhaps? It can't be a fiber cut, they have multiple backhauls.
Posted by UH-Bobby, 10-04-2010, 12:39 AM
That's what they told me also. I hope this is quick.
Posted by turbovps, 10-04-2010, 12:40 AM
DAMN!!!!!!!!!!!!!!
Posted by Rogean, 10-04-2010, 12:40 AM
Stops there...
Posted by brc_csf, 10-04-2010, 12:43 AM
Shouldn't that be automatic?
Posted by UH-Bobby, 10-04-2010, 12:44 AM
It sounds like a level 1 support rep, and they don't really know what's going on yet. Not saying that's a bad thing, because I called nearly 5 minutes after getting the down alerts.
Posted by Rogean, 10-04-2010, 12:45 AM
Was just about to say the same thing. Most people can't route outside their own ISP's towards Netriplex IP's. I'm no expert but that looks like a huge routing issue.
Posted by UH-Bobby, 10-04-2010, 12:48 AM
I would agree with that also. This is my first downtime event with Netriplex. I've had a minute blip every now and then, but no lengthy period.
Posted by Eleven2 Hosting, 10-04-2010, 12:53 AM
Ya we are not getting out of our ISP here.
Posted by Eleven2 Hosting, 10-04-2010, 12:53 AM
Would be nice if Netriplex had a twitter or status page to let us know what is going on.
Posted by Diocia, 10-04-2010, 12:54 AM
this is effecting me as well... what is their uptime guarantee?
Posted by brc_csf, 10-04-2010, 12:55 AM
It is good when stuff like this happen so we don't get other racks on the same facility.. Better to have racks spread on different places.
Posted by ubservers, 10-04-2010, 12:56 AM
Must say I agree here...This downtime is getting long...
Posted by HostingBig, 10-04-2010, 12:59 AM
Main trunk line has gone down feeding the data center engineers are rerouting to back up that's all the info I have
Posted by Eleven2 Hosting, 10-04-2010, 01:00 AM
Why was this not set to failover is the question?
Posted by brc_csf, 10-04-2010, 01:00 AM
Could you confirm the network SLA ?
Posted by UH-Bobby, 10-04-2010, 01:01 AM
That's definitely a good question to ask them.
Posted by Mike12King, 10-04-2010, 01:01 AM
Hoping it's back up soon. Last edited by Mike12King; 10-04-2010 at 01:11 AM.
Posted by ishan, 10-04-2010, 01:10 AM
100% network uptime SLA if you have 2 (redundant) network connections . But I guess, that wouldn't matter here.
Posted by Mike12King, 10-04-2010, 01:12 AM
There's still a network SLA for non-redundant network connections. Not sure of the details off the top of my head but I'm sure someone here knows all the details.
Posted by UH-Bobby, 10-04-2010, 01:13 AM
I'm hoping also . For the majority of my customers, they are asleep right now, so it isn't really bothering them so to speak, but I want this restored ASAP. I did just call, and they believe it's something right outside the building.
Posted by brc_csf, 10-04-2010, 01:14 AM
So.. One hour down and they still do not know what the issue is ? This starts to worry me.
Posted by UH-Bobby, 10-04-2010, 01:16 AM
Well, the issue isn't inside the building. Just making some assumptions, but I would guess if it was inside the building it would be much easier to pinpoint. Outside the building seems much harder. Not defending or attacking, just making observations, and opinions from those observations
Posted by Eleven2 Hosting, 10-04-2010, 01:17 AM
Did they say anything else?
Posted by UH-Bobby, 10-04-2010, 01:19 AM
They did say that they should get an update in the next 10 to 20 minutes.
Posted by FHDave, 10-04-2010, 01:19 AM
I am sorry for those expected ... Lucky for us, we are only using them for our secondary/off-site POP. For the rest, let's hope they are back up soon. Regarding SLA ... 100% SLA and 99.7% SLA for dual and single drop. You can 1 SLA credit for every 5 minutes and 10 minutes of downtime period for dual and single drop, respectively. For example, if you have single drop, then for every 10 minutes you are down you receive 1 SLA credit. And if I am mistaken, 1 SLA credit equals to your 1 day of monthly fee. You get a maximum of 30 SLA credits per month. In other words, if I am not mistaken, if they are down for 5 hours (300 minutes), customers with single drop may expect refund for the whole month fee. Last edited by FHDave; 10-04-2010 at 01:23 AM.
Posted by rafaelmagu, 10-04-2010, 01:29 AM
Hi there. First time posting and just wanted to ask: anyone knows of any other way of contacting them apart from phone or their (currently offline) portal?
Posted by ishan, 10-04-2010, 01:31 AM
Having single or dual drops wouldn't have mattered here. So, shouldn't all clients be subject to the same SLA ?
Posted by rafaelmagu, 10-04-2010, 01:32 AM
I'll try that, thanks!
Posted by brc_csf, 10-04-2010, 01:33 AM
99.7% SLA -> 0.3 Downtime = (60*24*30)/100*0.3 = 129.60 minutes
Posted by brc_csf, 10-04-2010, 01:35 AM
I really thought that we had a 100% SLA. Need to check the contract.
Posted by bhavicp, 10-04-2010, 01:35 AM
What other way would there be heh? Only thing i can think of is via web or phone.
Posted by Eleven2 Hosting, 10-04-2010, 01:36 AM
Sucks they don't know what is going on or have not communicated anything yet
Posted by rafaelmagu, 10-04-2010, 01:37 AM
Was hoping someone knew of a Twitter account or similar. They're great for this sort of stuff, 'cause it's external to their network.
Posted by Mike12King, 10-04-2010, 01:38 AM
They have a Twitter account but there's nothing new on there. http://twitter.com/netriplex
Posted by bhavicp, 10-04-2010, 01:38 AM
Oh yeah! - No i don't think they have anything like that..but they really should. They have a blog (which hasn't been updated in quite sometime) but i think that's hosted on there own network.
Posted by Diocia, 10-04-2010, 01:40 AM
my servers are back online. so is netriplex.com
Posted by ishan, 10-04-2010, 01:40 AM
They do have a twitter, without any updates - http://twitter.com/NETRIPLEX and - http://twitter.com/uberbandwidth
Posted by Eleven2 Hosting, 10-04-2010, 01:40 AM
They are saying it is forsure a fiber cut.
Posted by brc_csf, 10-04-2010, 01:41 AM
Hm. Can't reach netriplex.com nor any of our servers from 6 different locations.
Posted by Mike12King, 10-04-2010, 01:43 AM
Same here.
Posted by ishan, 10-04-2010, 01:44 AM
Where are you located ? It is down from all locations at host-tracker.com
Posted by Diocia, 10-04-2010, 01:44 AM
I live about 60 miles away, so that may have something to do with it. traceroute to 66.219.29.67 (66.219.29.67), 64 hops max, 52 byte packets 1 diocia.poweredbyclear.com (10.1.1.1) 1.858 ms 1.054 ms 0.821 ms 2 10.186.64.1 (10.186.64.1) 9.880 ms 12.117 ms 10.493 ms 3 dtr01spbgsc-gbe-8-17.spbg.sc.charter.com (96.34.64.102) 12.292 ms 11.709 ms 8.195 ms 4 96.34.64.143 (96.34.64.143) 19.487 ms 15.554 ms 13.393 ms 5 96-34-67-179.static.unas.sc.charter.com (96.34.67.179) 13.529 ms 13.565 ms 12.474 ms 6 96-34-67-175.static.unas.sc.charter.com (96.34.67.175) 17.024 ms 20.209 ms 13.351 ms 7 68-115-195-86.static.spbg.sc.charter.com (68.115.195.86) 108.020 ms 128.484 ms 118.291 ms 8 te-5-5.rtr2.avl1.netriplex.com (67.23.161.254) 13.877 ms 14.764 ms 17.374 ms 9 66.219.29.67 (66.219.29.67) 17.178 ms 15.227 ms 15.011 ms
Posted by rafaelmagu, 10-04-2010, 01:45 AM
Oh well, I guess we'll just have to wait. The downtime is already acknowledged on my NMS, so there isn't much I can do. Time to cook dinner.
Posted by ishan, 10-04-2010, 01:46 AM
Up now for me in India
Posted by Mike12King, 10-04-2010, 01:46 AM
Looks to be up for us now.
Posted by brc_csf, 10-04-2010, 01:47 AM
Up on all my locations.
Posted by FHDave, 10-04-2010, 01:48 AM
Their SLA is different for single and dual drops client. So I am not sure what you mean. if you have dual drop, you would have a full refund of your monthly fees for half the amount of downtime (2.5 hours, instead of 5 hours) compared to customers having only single drop. So yea, it does matter ...
Posted by rafaelmagu, 10-04-2010, 01:48 AM
Aaaaaaand we're back up in NZ.
Posted by bhavicp, 10-04-2010, 01:49 AM
Just came back online. (NZ as well )
Posted by ishan, 10-04-2010, 01:49 AM
89 minutes down. So according to Dave, I should get 9 days of credit.
Posted by FHDave, 10-04-2010, 01:52 AM
Sorry, I meant 99.97% SLA for single drop. Check your doc when you signed up.
Posted by HostingBig, 10-04-2010, 01:52 AM
came back up can reach it from our other data center but I can reach it from my location
Posted by ishan, 10-04-2010, 01:53 AM
I meant clients with dual drop and clients with single drop faced the same amount of downtime. Having dual drop wouldnt have caused lower downtime during this outage.
Posted by FHDave, 10-04-2010, 02:00 AM
I am not even sure if this affects customers with dual drop. But assuming it does, of course it wouldn't matter. But your SLA compensation is different. 90 minutes of downtime according to my monitoring. So, if I am not mistaken, 9 days and 18 days of SLA credit for customers with single and dual drops, respectively.
Posted by brc_csf, 10-04-2010, 02:00 AM
Should we all open a ticket asking what happened or would they send an e-mail or something detailing the issue and fixes ?
Posted by Rogean, 10-04-2010, 02:01 AM
Still can't reach my server. My gateway is 67.23.190.65, can anyone ping it?
Posted by HostingBig, 10-04-2010, 02:02 AM
I spoke to them on the phone the will be emailing everyone
Posted by ishan, 10-04-2010, 02:03 AM
Opening ticket would be better. I have never received an email about a downtime, only maintenance emails. No, I cannot ping it.
Posted by FHDave, 10-04-2010, 02:04 AM
I would say, email them if you don't get any contact from them within the next 48 hours. And make sure you ask about SLA compensation within 60 days
Posted by ishan, 10-04-2010, 02:05 AM
It affected both, I think. They wouldn't host the portal or their website on a cabinet with a single drop.
Posted by HostingBig, 10-04-2010, 02:07 AM
All of our servers are up but I cant ping our SSH to any of them
Posted by ishan, 10-04-2010, 02:09 AM
We are able to Ping and SSH our servers.
Posted by Diocia, 10-04-2010, 02:09 AM
64 bytes from 67.23.190.65: icmp_seq=3 ttl=247 time=15.430 ms
Posted by ishan, 10-04-2010, 02:13 AM
Where did you get the 99.97% figure ? I see - The NETRIPLEX IDC Network infrastructure will be available 99.95% of the time in any calendar month for singleline customers.
Posted by Rogean, 10-04-2010, 02:13 AM
My traceroute still isn't getting out of my ISP when going to my server's IP, however I can ping and load all the netriplex/uberbandwidth pages. What would cause that =\
Posted by FHDave, 10-04-2010, 02:15 AM
According to the SLA PDF document I signed.
Posted by Eleven2 Hosting, 10-04-2010, 02:17 AM
Back down for me here...
Posted by FHDave, 10-04-2010, 02:18 AM
works fine for me.
Posted by ishan, 10-04-2010, 02:20 AM
I see, so I will get less credits being in the same DC and facing the same outage. Its not stable
Posted by FHDave, 10-04-2010, 02:20 AM
If you can SSH/login to your server, try reverse traceroute (from your server to you PC/ISP). It could be that the forward and reverse traceroute are different.
Posted by FHDave, 10-04-2010, 02:22 AM
Received this from them:
Posted by turbovps, 10-04-2010, 02:32 AM
Is everything working fine? I don't think so as some of our IP's/routes are not still working. For e.g. One vps with 2 different subnet and one subnet working fine other not reachable... lol
Posted by BChosting, 10-04-2010, 02:58 AM
Everything is set to fail over. The issue is lies elsewhere. I cannot disclose more than that at the moment. I'm sure a press release will be issued sometime soon. I can tell you that Netriplex did everything right, except maybe putting too many eggs in one basket. There was no power or network issues, within the Netriplex network. That's all I can say.
Posted by BChosting, 10-04-2010, 03:00 AM
Is that still the case? If so, PM me.
Posted by HostingBig, 10-04-2010, 03:03 AM
some of our IP's are still dead
Posted by Rogean, 10-04-2010, 03:04 AM
Same; 67.23.190.xx
Posted by HostingBig, 10-04-2010, 03:12 AM
all is back up
Posted by turbovps, 10-04-2010, 03:16 AM
Are you from Netriplex? Why should I PM You?
Posted by Rogean, 10-04-2010, 03:20 AM
I'm curious what was causing those subnets to be unreachable. After the 67.23.'s came back up, my ping on both subnets got bumped up about 20 ms. I've had to contact netriplex a few times for static routes to fix the latency. Is it related?
Posted by BChosting, 10-04-2010, 03:21 AM
Everything good?
Posted by BChosting, 10-04-2010, 03:25 AM
It depends which back haul path your traffic normally rides. Things are mostly back to normal, but there is some additional latency right now for traffic going through Atlanta. This will go back to normal when the underlying carrier fixes their issue.
Posted by turbovps, 10-04-2010, 04:46 AM
All subnets are up now. Thanks.
Posted by brc_csf, 10-04-2010, 08:24 AM
http://uberbandwidth.com/sla.asp " Network Availability 100% network availability " netriplex_overview.pdf: "0 Downtime - Netriplex's 100% uptime SLA means you can expect zero downtime, all the time. This is radically different than datacenters who offer a 99.9% SLA which gives them room for 43 minutes of downtime every month."
Posted by pubcrawler, 10-05-2010, 07:16 AM
"At 12:09AM our Atlanta secondary path went down for scheduled and anticipated maintenance that was slated to be non-customer impacting, however at 12:12AM our primary fiber to Atlanta and Ashburn went down as well, both served by a different provider. While service has been restored via Ashburn at this time, we are still investigating the additional outages and will update you as soon as we have further information." --- SNIP --- So what's the story with Uber's Asheville location? Did they ever clear up what happened? It's clear from accounts here that they have two pieces of fiber only to Asheville. The likelihood of such problems when redundancy is limited to two options is far greater than if they had 3, 5, 15, 18 or more providers like their marketing pretends they do. They apparently DO NOT HAVE: 15 (and growing) Transit providers100 (and growing) Peering network providers At that location. They are apparently connected via private fiber run to datacenters in Ashburn (OC-192 to Equinix) and Atlanta (via OC-192 from AT&T to 56 Marietta?) where these *MIGHT* be true. These are taken from their very own press releases. I am clueless about these 240GBPs network claims. I have my doubts. OC-192 is 9.6GBS/sec - 2 of those = 19.2GBPs. Where are the other 220 gigs of throughput coming from? Let me guess, that's throughput in other remote datacenters where they themselves have gear colo'd. " Upon completion, the network will offer a self-healing system that provides the ultimate reliability by allowing instantaneous re-routing of internet traffic in the event of a fiber cut." So, I suppose their network isn't completed yet?
Posted by Rogean, 10-05-2010, 08:32 AM
They're having a public conference on thursday if you want to ask all those questions
Posted by pubcrawler, 10-05-2010, 08:42 AM
Can you post the link included in that message for the Thursday conference please? Or PM me.
Posted by FHDave, 10-05-2010, 09:41 AM
If you don't get the email, most likely you are not a direct customer of Netriplex/Uberbandwidth. In which case, it would be better not to post the link to the public. Better to keep this to direct customers only.
Posted by Rogean, 10-05-2010, 10:01 AM
The email does say it is open to everyone, including non customers, but I wasnt going to give the link out here.
Posted by pubcrawler, 10-05-2010, 10:52 AM
Well, I *do* have equipment in that facility via another customer. Folks remember the thread on here from a few months back about routes out of this facility and contention that the big list of providers on Uber's sales info wasn't true? Well, doesn't this dual fiber failure - an A-B or A or B path situation failure answer that question? The info on the OC-192 connections to Asheville come from: http://www.netriplex.com/company/arc...-08122009.aspx Washington, DC OC-192 circuit http://www.netriplex.com/company/arc...-09172007.aspx Atlanta OC-192 circuit Both releases clearly say CIRCUIT, which is a single reference.
Posted by RyanD, 10-05-2010, 10:43 PM
Yeah it's been largely documented that their claims are nothing but marketing fluff, they do not have the providers they claim and thats been proven over and over, they do not have 240gbps, thats been proven easily, etc, etc.
Posted by Eleven2 Hosting, 10-11-2010, 05:32 PM
Anyone hear anything else about this outage? They had that conference call scheduled then canceled it?
Posted by Rogean, 10-11-2010, 05:47 PM
Nothing other than the conference call being postponed.
Posted by pubcrawler, 10-12-2010, 08:47 AM
Any reason given for this cancellation? Really gives me the creeps that they haven't cleared up what happened and said how it won't again. In other news, unsure about their actual network bandwidth capacity, but I've been impressed with their network so far. Good routes compared to where have other gear and good throughput. Some fastest speeds I've seen - but that might just be our Gbit connection there