Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Interserver down?
Interserver down?
| Posted by gmoney3, 02-13-2010, 11:42 PM |
Is anybody else having a problem with Interserver. My box has been offline for over an hour now, and just not I cant connect to their webpage.
Edit:
Just got an email from them confirming so ignore my post.
NYC Fiber outage.
We have got been in contact with Abovenet and they have told us there is a fiber cut affecting multiple providers in the NYC area. There is no additional information on where this occurred, but it is affecting the same path to NYC from earlier today. The latest update is a crew is in route. |
| Posted by Dougy, 02-13-2010, 11:56 PM |
|
Yep.. same issue that Abovenet had at 4AM.. and then later in the morning.. this is third time today. |
| Posted by Dougy, 02-14-2010, 12:21 AM |
Quote:
Hello,
Today at 9:49pm EST time, Above.net's fiber to NYC was cut in Amtrack location. This effects both primary and secondary path causing us to use backup providers out of the facility. We're working on a short term plan to bring all ip ranges that are currently down back online however all clients might and most likely will experience slower than usual connections and occasional packet loss. There's currently no ETA on the fiber repair or the non-working ip range issue but it's estimated that it might take up to few hours tonight to get the fiber fixed. This cut is also effecting majority of NJ to NY fiber clients for Above.net and is being treated with highest priority. While we will may not be able to answer all phones calls while we're working on these issues, we assure you this is our highest priority and we will not rest until we come to a solution as soon as technically possible. We will attempt to provide status updates once more information is available to us.
Regards,
Rackvibe.com Network Group
|
Just more info for everyone.. |
| Posted by Khell, 02-14-2010, 01:26 AM |
This is ridiculous. I would have expected to be routed onto a backup solution now hours after the cut. Needless to say this has not happened. My service is completely down for my choice to base my primary point of operations with these people.
Making it worse, the fiber restoral will only be attempted 8 hours from now. Abysmal. |
| Posted by Dougy, 02-14-2010, 01:27 AM |
Quote:
|
Originally Posted by Khell
This is ridiculous. I would have expected to be routed onto a backup solution now hours after the cut. Needless to say this has not happened. My service is completely down for my choice to base my primary point of operations with these people.
Making it worse, the fiber restoral will only be attempted 8 hours from now. Abysmal.
|
Where are you getting eight hours from? |
| Posted by KMyers, 02-14-2010, 01:36 AM |
Greetings,I cant seem to hit one of my boxes at InterServer. I hope it can be resolved with a simple reboot.
|
| Posted by Dougy, 02-14-2010, 01:39 AM |
Quote:
|
Originally Posted by KDisk
Greetings, I cant seem to hit one of my boxes at InterServer. I hope it can be resolved with a simple reboot.
|
Don't get your hopes up. |
| Posted by KMyers, 02-14-2010, 01:44 AM |
Greetings,Here is the latest
Quote:
|
We have got been in contact with Abovenet and they have told us there is a fiber cut affecting multiple providers in the NYC area. There is no additional information on where this occurred, but it is affecting the same path to NYC from earlier today. The latest update is a crew is in route.
|
|
| Posted by Khell, 02-14-2010, 01:54 AM |
Quote:
|
Originally Posted by Dougy
Where are you getting eight hours from?
|
Out of date info. There was originally an advisory on Honelive (Interserver client)'s ticketing page stating such, but it's been changed now. |
| Posted by net, 02-14-2010, 02:11 AM |
Quote:
|
Originally Posted by Khell
This is ridiculous. I would have expected to be routed onto a backup solution now hours after the cut.
|
Yes, our servers are down too while their site is up.
Brilliant! |
| Posted by Dougy, 02-14-2010, 02:23 AM |
Quote:
|
Originally Posted by net
Yes, our servers are down too while their site is up.
Brilliant!
|
A large majority of the servers are online for us at this point, only a small portion of our (at least JustEdge/RackVibe's) equipment is still down. |
| Posted by KMyers, 02-14-2010, 02:29 AM |
Greetings,In my case I have 2 offline and 1 online. I did open a shell connection from the 1 server I can access and was able to connect to both of the other servers.
I do want to say that I still like InterServer, this is the first time this sort of event has happened to me. Perfection cannot be expected.
|
| Posted by styx, 02-14-2010, 03:36 AM |
Doesnt seem like New York (Abovenet) wants to work st Valentines day eve. Just passed 5 hour mark.
While Interserver`s site is up; Servers which are up as well on their network can only be reached domestically and some parts of Europe. The rest of the world is out of reach
Lets pray for Abovenet techs, I hope they go straight to the "cut" and do not stop by DD to get some coffee. |
| Posted by JonnyQuags, 02-14-2010, 03:46 AM |
|
We are aware of routing issues from some locations which are still being fixed. The latest update from Abovenet is that they are on site at the problem location. Once everything is solved a RFO will be sent out as well as steps we will take to ensure a problem like this does not occur in the future. |
| Posted by Dustin Cisneros, 02-14-2010, 03:54 AM |
|
It appears to be fine on my end now. |
| Posted by styx, 02-14-2010, 04:02 AM |
Quote:
|
Originally Posted by semoweb
It appears to be fine on my end now.
|
You are on 64.* network and for those ips traffic as i can see is routed throught XO. The rest is down. I also have 64.* with Justedge and it works. But i also have 209.* and its down. |
| Posted by net, 02-14-2010, 05:49 AM |
Down for 8 hours now.... I can't believe this.
This is my first time to experience such a longest downtime.... wow, can't believe it.
I wish I am just dreaming.... |
| Posted by LP560, 02-14-2010, 06:50 AM |
My interserver servers are online although running slow.
IPs: 69.* |
| Posted by LP560, 02-14-2010, 06:51 AM |
Just received this:
Quote:
The fiber cut is being worked on by Abovenet. They are saying that the crew is in place to replace the span of fiber that is damaged.
They are not able to give us an estimated time of repair yet.
|
Thats two fiber issues within 24 hours!
First was a bend, now a cut, that should never happen on such an important network route. |
| Posted by PersianStar, 02-14-2010, 07:35 AM |
my server is completely DOWN !!!!
what's going on there ?
C:\Documents and Settings\Behz4d>ping 66.45.230.*
Pinging 66.45.230.* with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Ping statistics for 66.45.230.*:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss), |
| Posted by KMyers, 02-14-2010, 08:55 AM |
Quote:
|
Originally Posted by net
Down for 8 hours now.... I can't believe this.
This is my first time to experience such a longest downtime.... wow, can't believe it.
I wish I am just dreaming....
|
Same Here, this is the first time I have seen anything like this at all from InterServer. |
| Posted by Dougy, 02-14-2010, 10:02 AM |
|
This is way beyond our control (RackVibe/JustEdge/Interserver). There is nothing we can do except wait. |
| Posted by Dougy, 02-14-2010, 11:13 AM |
|
I believe at this time all fiber and connectivity is restored to all *RackVibe* clients. |
| Posted by Douglas, 02-14-2010, 11:15 AM |
I agree with Dougy on this... this is definitely beyond anything that InterServer can do/provide. This falls directly on the AboveNET guys to get fixed.
Now that Solar's aware of the underlying cause (Part of our services are also at InterServer and with JustEdge), we're working to notify our customers accordingly.
I want to let the InterServer give the reason for the outage; let's just say that I now understand the reason for the fibre cut.  |
| Posted by JoCortland, 02-14-2010, 12:04 PM |
As annoying as it is to have to get up on a Sunday morning and move the big clients to my backup server, I am really not upset with Interserver. I have had updates along the way, it is out of their control, I have a backup plan and it's not the biggest deal in the world.
I AM interested to hear HOW the fibre got cut though Douglas, you have piqued my curiosity. |
| Posted by colorprint, 02-14-2010, 12:37 PM |
|
But this happens 3 days ago?! |
| Posted by Douglas, 02-14-2010, 12:40 PM |
Yes, and there were issues then, and the issues today are related to the issues caused by the fire three days ago.
I'm sorry, but a fire bursting out of a manhole where the flames reach seven stories tall is bound to have lingering issues and will take some time to get resolved. |
| Posted by JoCortland, 02-14-2010, 01:00 PM |
|
Douglas, thanks for the links. I am totally not upset with either above.net or interserver at this point That was quite a spectacular thing and I am happy nobody was killed! |
| Posted by Douglas, 02-14-2010, 01:13 PM |
|
There's a lot of people upset about this whole thing. While I understand the reasoning behind their feelings, they also need to take a step back and put things into perspective. I honestly believe that no one could forecast such an issue occurring, though it does continue to illustrate just how dependent we are on major internet traffic routing points, and how vulnerable they can be subject to various disasters. |
| Posted by gmoney3, 02-14-2010, 01:19 PM |
|
Happy to say my 68.* is back up as of now. |
| Posted by hayley2005, 02-14-2010, 01:31 PM |
|
^ so is mine Well its a bit slow but its there. |
| Posted by styx, 02-14-2010, 01:34 PM |
Eastern europe is completely "blacked out". No connection to any of Interserver/Justedge stuff from there (russia, ukraine, poland and so on).
If techs were on site at 4-5 Am and its 1Pm right now.. They had 6+ hours to at least make some prognosis which we, customers, need to know. Our customers need some legit explanation as well. |
| Posted by Douglas, 02-14-2010, 01:43 PM |
|
styx, legit explanations above, by way of the links I posted. |
| Posted by dzone, 02-14-2010, 01:48 PM |
Quote:
|
Originally Posted by styx
Eastern europe is completely "blacked out". No connection to any of Interserver/Justedge stuff from there (russia, ukraine, poland and so on).
If techs were on site at 4-5 Am and its 1Pm right now.. They had 6+ hours to at least make some prognosis which we, customers, need to know. Our customers need some legit explanation as well.
|
please give me your ip in EU (Russia ? Poland?) so we can check. (PM or please create a ticket with us so we can troubleshoot).
We're running with very reduced capacity which is why you're seeing all the packet loss. Everything should be accessible, but you can expect slow connectivity while we're working on adding extra capacity that does not utilize the fiber path in question to NYC. The fact it's a Sunday does not help either however we do have couple of alternatives that should yield results before the fiber path is restored.
We've had a chat with Above.net and we're unable to get a straight answer from them yet regarding ETA but it does look like a major issue for them since they're not able to get to the manhole where the break is. This tells me that conditions must not be safe and for safety reason they cannot get inside and restore the service. Above.net also mentioned that they are working on alternatives paths right now and will proceed with the the solution that takes least amount of time.
This is the first time in 10 years that we've had such a major issue that's out of our control and I can assure everyone that this is being treated with absolutely highest priority and that we're working around the clock (literally, I've been personally up for 48 hours now) to restore normal quality of service. |
| Posted by JoCortland, 02-14-2010, 01:51 PM |
My 66.45 is accessible from some IP ranges and not from others. I am getting uptime/downtime/uptime/downtime, so probably safer just to leave everybody on the slower backup VPS elsewhere than to move 'em back until things settle down.
I appreciate the communication by the providers involved, and recognise that this is definitely NOT a normal situation. Thanks for your updates. |
| Posted by Dustin Cisneros, 02-14-2010, 02:04 PM |
Quote:
|
Originally Posted by JoCortland
I am getting uptime/downtime/uptime/downtime
|
Same here my phone is being blasted fail, pass, up, down aswell as my email.
Anyhow i understand the situation but still frustrated which any downtime will do to any one.. We will be refunding client's for the downtime. |
| Posted by LP560, 02-14-2010, 02:36 PM |
From interserver:
Quote:
We are aware some clients are still having connectivity issues. To rectify any remaining issues we are working not on adding a connection to Cogent who is On-Net within our datacenter. They are currently not part of our bandwidth providers.
Abovenet has updated us they are trying to reroute the cut fiber using an alternate path because they are unable to gain access to the fiber for repair due to fire.
We do not have any ETA/ETR estimates at this time. As soon as we have more information we will send another update.
|
|
| Posted by dzone, 02-14-2010, 02:42 PM |
Just received a word that above.net is pulling a new cable in place. They started around noon and it will take about 12 to 18 hours to restore the affected path.
In paralel, we're still working on alternative solution to increase backup capacitty to try to restore service quality as soon as possible. |
| Posted by ConnDNS, 02-14-2010, 03:48 PM |
My 209.159.x is back up now....around 2PM eastern.
Let's hope Interserver and the other hosting folks that colo there learn a lesson from this.
You read all this sales hype about all their backbone providers and redundancy, yet a fiber cut knocks them off the net for what...12-15+ hrs???
Since their website stayed up, would of been really nice if Interserver used their network status page to keep their customers informed.
https://interserver.net/cms/support/...m=news&_a=view
Bad things happen, it's how you handle them that matters most.
Just my 2 cents worth. |
| Posted by styx, 02-14-2010, 04:22 PM |
Quote:
|
Originally Posted by ConnDNS
My 209.159.x is back up now....around 2PM eastern.
Let's hope Interserver and the other hosting folks that colo there learn a lesson from this.
You read all this sales hype about all their backbone providers and redundancy, yet a fiber cut knocks them off the net for what...12-15+ hrs???
Since their website stayed up, would of been really nice if Interserver used their network status page to keep their customers informed. https://interserver.net/cms/support/...m=news&_a=view
Bad things happen, it's how you handle them that matters most.
Just my 2 cents worth. |
Too bad they are all routed through wbcconnect. At least mtr showed it before everything went down. So if wbc goes down.. everything goes down.
But i might be mistaken. |
| Posted by KMyers, 02-14-2010, 04:35 PM |
Quote:
|
Originally Posted by styx
Too bad they are all routed through wbcconnect. At least mtr showed it before everything went down. So if wbc goes down.. everything goes down.
But i might be mistaken.
|
Hello,It is strange, I had one box that was almost unimpacted by this, and 3 others that were dead.
|
| Posted by dzone, 02-14-2010, 05:01 PM |
Quote:
|
Originally Posted by styx
Too bad they are all routed through wbcconnect. At least mtr showed it before everything went down. So if wbc goes down.. everything goes down.
But i might be mistaken.
|
That's not correct. There are multiple providers however from your point of view you may see them mostly through one while another client will see it through another most of the time. The upstream link is not an issue and they're all up. It's the fiber that NJ and NY states that is effected.
All ip ranges are now up through backup and we're still working on increasing backup capacity while Above.net is resolving their issues in the area. |
| Posted by LP560, 02-15-2010, 08:39 AM |
Anyone got speed issues today?
My servers are very slow! |
| Posted by Dougy, 02-15-2010, 09:18 AM |
|
Fiber still not fixed.. |
| Posted by Douglas, 02-15-2010, 10:52 AM |
|
Everything is running on alternate providers (XO and Cogent), so things are being hammered. Once AboveNET finishes their NYC link re-route, things should settle down and speeds should greatly improve. |
| Posted by KMyers, 02-15-2010, 11:17 AM |
Grrr...I really hope this gets resolved soon. I am not blaming InterServer, I know this type of issue can effect almost anyone.
|
| Posted by LP560, 02-15-2010, 12:02 PM |
Looks like its now improving, just received this from interserver:
Quote:
|
The speeds will be improving now. It may take 5 - 10 minutes for it to be showing from all routes. A complete RFO will be sent out and we still expect to get cogent in shortly.
|
|
| Posted by JoCortland, 02-15-2010, 12:09 PM |
I am seeing much better routing today than yesterday. Yesterday from Toronto to Chicago thru a few XO hops and eventually to Interserver, with a few timeouts. Today Toronto -> Chicago -> hwng.net -> Interserver, total of 11 hops, and reasonably speedy.
Thanks Interserver folks for making this thing go again. |
| Posted by dzone, 02-15-2010, 12:50 PM |
Quote:
|
Originally Posted by JoCortland
I am seeing much better routing today than yesterday. Yesterday from Toronto to Chicago thru a few XO hops and eventually to Interserver, with a few timeouts. Today Toronto -> Chicago -> hwng.net -> Interserver, total of 11 hops, and reasonably speedy.
Thanks Interserver folks for making this thing go again.
|
Everything is back to normal now, the fiber path was restored and full capacity is now restored. We're still re-evaluating our backup to make sure we have appropriate backup capacity for the future and we will bring additional backup providers via diverse fiber path shortly. |
| Posted by Dustin Cisneros, 02-15-2010, 12:59 PM |
Quote:
|
Originally Posted by LP560
Anyone got speed issues today?
My servers are very slow!
|
Was yesterday but today seem's to be running like a beast :-p
Or maybe the sign up's that i had during the night made me feel like that lol. |
| Posted by Geeks-Hosting, 02-16-2010, 04:47 AM |
Quote:
|
Originally Posted by KDisk
Grrr... I really hope this gets resolved soon. I am not blaming InterServer, I know this type of issue can effect almost anyone.
|
Yeah, It effect me a little be because of my stream clients I had lol but thank god its back and running smooth again |
Add to Favourites 
Print this Article
Also Read
Shared Hosting
Shared Hosting
