nexpoint outage

Portal Home > Knowledgebase > Articles Database > nexpoint outage

Posted by DeltaAnime, 03-04-2006, 01:17 AM
Nexpoint seems to be offline for me, can anyone confirm? ~Francisco
Posted by LoginTech, 03-04-2006, 01:21 AM
My friend has a couple personal boxen there and they are not entirely offline but it looks like nexpoints core is seeing ~98.5% packetloss..... perhaps a large DoS?
Posted by DeltaAnime, 03-04-2006, 01:28 AM
I'm not even getting PL, just flat off ~Francisco
Posted by LoginTech, 03-04-2006, 01:31 AM
well I have mtr running on it with a very long ttl and i was getting a 12 second response from the router and 700 from a server when the packet actually gets through.
Posted by reject, 03-04-2006, 01:37 AM
My boxes are down. But I guess you already know that, fran. Someone call tech support.
Posted by DeltaAnime, 03-04-2006, 01:39 AM
I already called them and they weren't sure quite WHAT it was, tehy thought it was maybe a bad switch, we'll have to see I guess
Posted by reject, 03-04-2006, 01:41 AM
I've had outstanding uptime with them, so I'm sure they'd get this fixed real quick.
Posted by LoginTech, 03-04-2006, 01:42 AM
I tried calling but no answer, I'm hoping it's because they are feverishly working on the issue, on the upside their hold music is Jimmy Eat World so at least I didn't mind waitng lol
Posted by LoginTech, 03-04-2006, 02:12 AM
I'm able to access a machine now.
Posted by reject, 03-04-2006, 02:13 AM
Yup, me too. Nice work nexpoint.
Posted by LoginTech, 03-04-2006, 02:15 AM
Indeed, if it was a piece of networking hardware failing and it took about 1hr to solve i'd say thats pretty damn good.
Posted by DeltaAnime, 03-04-2006, 02:59 AM
Got this from one of their people just now - ----------------------------------------------------------- Greetings, On behalf of Nexpoint I would like to apologize for the recent outage and for any delay in response to phone calls or chat request, however, even though we have a 24 hour support team, when we have a hardware failure such as the one we just had, we put every available person on the task of resolving the issue as soon as possible. We value every one of you and your patience in letting us deal with the issue in a timely fashion. Sincerely Nexpoint Technologies Inc. Tel: 866-NEXPOINT [639-7646] http://www.nexpoint.net
Posted by reject, 03-04-2006, 05:07 AM
My servers are unreachable again. hmmm.
Posted by LoginTech, 03-04-2006, 07:31 PM
Looks like problems are back.
Posted by DeltaAnime, 03-04-2006, 11:06 PM
Just had another burp a few minutes ago, this is driving me nuts :x
Posted by reject, 03-05-2006, 12:58 AM
and 5 of my 6 servers are down again.
Posted by LoginTech, 03-05-2006, 01:07 AM
anyone call yet?
Posted by DaJoob, 03-05-2006, 01:35 AM
Their network has been going on and off all day. I guess it's kind of a good thing that I bought a new dedicated server from a different provider. Now I'm just waiting for it to come back online so I can resume transferring over my data and go to bed while it finishes.
Posted by LoginTech, 03-05-2006, 01:47 AM
They have been great for the past 6 mos so i think they deserve a little bit of time here lol
Posted by DeltaAnime, 03-05-2006, 05:11 AM
Again? ~Francisco
Posted by LoginTech, 03-05-2006, 12:17 PM
I talked to support late last night and they said it might be spotty for the next 8hrs(this was @ 11pm MST), i think they resolved it, the connection has been stable for 7hrs now.
Posted by DaJoob, 03-06-2006, 01:42 AM
It's down again
Posted by DeltaAnime, 03-06-2006, 01:51 AM
Ya, it was very good for the entire day, I did ping tests all night, 23000 sent, 22990 returned. But now they're out again. I talked to one of the techs and he said that tehy have been applying cisco security patches and did take a ddos over the weekend (i'm thinking the bots were exploiting the routers/etc, rather than just actually hammering them down) ~Francisco
Posted by LoginTech, 03-06-2006, 01:54 AM
Yes they are showing down again... was going so well...
Posted by tree-host, 03-06-2006, 04:33 AM
According to Jacob on the support line, theres a problem with one of the routers. Network techs are looking into it, If its an issue that can be repaired, Jacob says the techs said it will be about an hour, If they need to replace it, it will be about 2. I guess all we can do at the moment is wait to see what happens. From Talking to Tom on friday they where doing Cisco IOS software upgrades, which caused downtime on friday morning. They did take some dDos attacks at somepoint over the weekend according to Jacob. Have been using nexpoint for a while, and im sure they will sort it out as quick as they can. [Edit] Seems the second i posted that, they resolved the problem[/edit] Last edited by tree-host; 03-06-2006 at 04:37 AM.
Posted by DeltaAnime, 03-06-2006, 04:53 AM
Poor Jacob, he has to deal with all of us calling, and he's keeping his cool so nicely. Well done, Jacob ~Francisco
Posted by tree-host, 03-06-2006, 08:19 AM
Looks like its gone down again, poor Jacob is still there. 3 years of using nexpoint, and i must say this is the first continuious lot of problems.
Posted by reject, 03-06-2006, 09:03 AM
I just hope they can get it sorted soon.
Posted by reject, 03-06-2006, 10:56 AM
and it's down again!
Posted by tree-host, 03-06-2006, 11:01 AM
yeah, apparently they think the problem is a dDOS
Posted by RossH, 03-06-2006, 11:42 AM
Are they still down?
Posted by tree-host, 03-06-2006, 11:46 AM
yeah, at the moment they seem to be
Posted by tree-host, 03-06-2006, 12:03 PM
Looks like it back up now
Posted by DaJoob, 03-06-2006, 12:14 PM
but for how long?
Posted by tree-host, 03-06-2006, 12:17 PM
Im havent been told 100% definate what the problem was and what they have done to resolve it. But fingers crossed
Posted by nexpoint, 03-06-2006, 01:37 PM
Good Afternoon, I wanted to take a minute and provide an update and explanation to everyone regarding the recent network-wide connectivity issues people have been experiencing with the Nexpoint network throughout earlier this morning the a couple times earlier this past weekend. The network downtime experienced up to this point has been from a combination DDOS (distributed denial of service: http://en.wikipedia.org/wiki/Denial_of_service) attack and the replacement and reconfiguration of network equipment (in order to better handle and circumvent any further DDOS attempts). We are working closely with datacenter network engineers to identify the source and destination of these attacks so that all related traffic can be dropped prior to entry into our network domain. Additionally, configuration changes have been made so that the network devices better handle the heavy loads during the spike in activity. Nexpoint apologizes to all of its clients for the inconvenience caused during these incidents. We also appreciate all of the clients who have posted the information they knew so that other clients may be informed; this is especially important when such a post from a Nexpoint representative was not time appropriate. We will continue to work diligently on the issue to provide the best service possible to everyone in our network. Your patients is valued as we work to permanently isolated and resolve this issue. Best Regards, C. Aaron Smith Support Operations Manager Nexpoint Technologies Inc. 866-NEXPOINT [639-7646] http://www.nexpoint.net/
Posted by DeltaAnime, 03-06-2006, 01:55 PM
Thanks Aaron I just had a nice chat with Shaun on the live help for more information, nice fellow Thanks! ~Francisco
Posted by LoginTech, 03-06-2006, 02:23 PM
You really do have to pat Nexpoint on the back for the way they handled the situation, even thought the drops were annoying and it's taking a while to fix, at no point have they been completely unreachable, now that is customer service at it's finest.
Posted by missilemike, 03-06-2006, 10:15 PM
....and they're down again...
Posted by LoginTech, 03-06-2006, 10:23 PM
*yawn* who wants to make the call this time? edit: and i hate to be like this but, just null route the targeted DDoS'd IP's let that customer sit it out and not affect the rest of us (or at least find a way to filter that customer from the rest of us as this...yea...this isn't fun. Last edited by LoginTech; 03-06-2006 at 10:29 PM.
Posted by DeltaAnime, 03-06-2006, 11:05 PM
i'm on hold right now, I think Jacob ran away
Posted by LoginTech, 03-06-2006, 11:43 PM
Still on hold Delta?
Posted by DeltaAnime, 03-07-2006, 12:27 AM
I hung up since the music got bad It came back for a bit, been down since then though. ~Francisco
Posted by DeltaAnime, 03-07-2006, 12:32 AM
Thye're still under an attack, Jacob is looking into getting nullroutes put into place.
Posted by DeltaAnime, 03-07-2006, 12:41 AM
They're back online, anyone nulled? ~Francisco
Posted by LoginTech, 03-07-2006, 12:48 AM
We're good.
Posted by markhard, 03-07-2006, 04:25 PM
my server still can't be accessed. its been the whole day down.. but the wierd thing is dnsstuff.com can ping and tracert my server IP or my domain name. i think i don't have internet connection problem because i can browse other web site. does any body here still have problem?
Posted by LoginTech, 03-07-2006, 04:50 PM
submit a ticket then, i've been up all day.
Posted by markhard, 03-07-2006, 06:02 PM
i already try to send email, but never receive reply
Posted by markhard, 03-07-2006, 06:16 PM
i don't understand, i can ping my server from my computer, but still i can't access ssh, website or email
Posted by LoginTech, 03-07-2006, 07:57 PM
mark if you can ping then get on their live chat or something and see if your IP is mis-routed or something and sheesh you know you can call them too!
Posted by DaJoob, 03-07-2006, 11:53 PM
Network's down again.
Posted by LoginTech, 03-08-2006, 12:09 AM
Yep i'm out as well.....
Posted by LoginTech, 03-08-2006, 12:15 AM
Back up..... for new
Posted by LoginTech, 03-08-2006, 02:03 PM
Showing ~20% packetloss now (for the last hour) can anyone else confirm?
Posted by RossH, 03-08-2006, 02:32 PM
--- www.nexpoint.net ping statistics --- 100 packets transmitted, 95 received, 5% packet loss, time 99082ms rtt min/avg/max/mdev = 22.515/22.873/24.435/0.279 ms, pipe 2