Netdepot / Gnax - total downtime so far 25 hours and counting - Knowledgebase

Portal Home > Knowledgebase > Articles Database > Netdepot / Gnax - total downtime so far 25 hours and counting

Netdepot / Gnax - total downtime so far 25 hours and counting

Posted by zymic, 07-20-2008, 06:11 AM
Hello, http://netdepot.com network has been entirely down for over 25 hours so far. I can't access their support desk as their website does not load either. Is anyone else having this problem? It doesn't seem to be working for any UK people, but probably more than just UK too. Netdepot.com ping check: New York, U.S.A. Packets lost (40%) 25.4 25.6 25.9 Florida, U.S.A. Packets lost (70%) 12.7 12.8 12.8 San Francisco, U.S.A. Packets lost (20%) 61.7 62.3 62.7 Vancouver, Canada Packets lost (60%) 95.7 96.7 97.5 Austin1, U.S.A. Packets lost (40%) 77.8 78.0 78.4 Austin, U.S.A. Packets lost (70%) 77.9 78.1 78.3 Chicago, U.S.A. Packets lost (50%) 43.4 43.5 43.6 Amsterdam, Netherlands Packets lost (60%) 97.4 99.3 101.4 Amsterdam2, Netherlands Packets lost (60%) 106.3 106.8 107.0 Amsterdam3, Netherlands Packets lost (40%) 106.2 106.6 107.0 Copenhagen, Denmark Packets lost (80%) 109.1 109.2 109.2 Krakow, Poland Packets lost (60%) 123.8 124.4 124.9 Lille, France Packets lost (70%) 99.5 99.6 99.6 Cologne, Germany Packets lost (50%) 111.6 111.9 112.1 Stockholm, Sweden Packets lost (80%) 120.3 124.0 127.7 Munchen, Germany Packets lost (90%) 122.8 122.8 122.8 Cagliari, Italy Packets lost (50%) 133.0 133.7 134.3 Hong Kong, China Packets lost (80%) 246.1 246.3 246.4 Zurich, Switzerland Packets lost (40%) 126.7 128.6 130.9 Sydney, Australia Packets lost (20%) 211.3 212.2 213.8 Porto Alegre, Brazil Packets lost (60%) 158.9 165.5 174.2 Singapore, Singapore Packets lost (50%) 274.1 278.1 282.1 Shanghai, China Packets lost (30%) 264.4 264.6 264.9 Melbourne, Australia Okay 216.1 217.0 218.1 London, United Kingdom Packets lost (100%) Paris, France Packets lost (90%) 107.7 107.7 107.7 Madrid, Spain Packets lost (100%) Here is my server: http://nofx.vndv.com
Posted by Cats-Computing, 07-20-2008, 06:19 AM
I can get to Netdepot.com but not your server.
Posted by zymic, 07-20-2008, 06:26 AM
Please post your country. Its been down entirely for over 24 hours for all(?) UK users. I can't even get it to resolve using a proxy located in the USA; http://vtunnel.com/ Last edited by zymic; 07-20-2008 at 06:29 AM.
Posted by gate2vn, 07-20-2008, 06:29 AM
I can't reach any of them either. However, netdepot's IP is still pingable, not yours
Posted by zymic, 07-20-2008, 06:31 AM
Yeah its strange, i can resolve gnax/netdepot's IP but can't load their website. Its been down for over a 24 hours so I'm surprised no one has mentioned it on here :| I've just got back from a wedding which had no internet access, which is why I have only had a chance to post this today. I can't get hold of anyone from Netdepot at all. Complete disaster.
Posted by Cats-Computing, 07-20-2008, 06:43 AM
Sorry, but it is under my location :"Australia". We've got a server in GNAX and it's fine, so no idea.
Posted by zymic, 07-20-2008, 06:46 AM
Yeah it seems to be euro countries
Posted by Cats-Computing, 07-20-2008, 06:49 AM
Hmm, we've had a few clients inform us they aren't able to connect periodically to another location in the US, maybe it's a Europe to US problem?
Posted by zymic, 07-20-2008, 06:56 AM
I also posted on my forum about this, to see if the sites load for people, posts so far: http://www.zymic.com/forum/index.php?showtopic=10670
Posted by Digisource, 07-20-2008, 08:57 AM
Same problem here in Portugal. I called them friday... cause our servers were on and off, and with a random behavior, and they told me everything was fine. Their sites wouldn't load at all... and they still wont. Today I called them twice, and they told me to email support[at]gnax.net, but got returned! engineering[at]gnax.net seems to work. Finally, after insisting a lot, they told me its a DOS attack and they are working on it! They also keep telling me "servers are powered up", all the time, which really annoys me. Its been almost two days, and they act as if its not their problem and its not my business to know whats going on!
Posted by zymic, 07-20-2008, 09:01 AM
Digisource, I have also sent a ticket about this, they replied asking for traceroutes, but the thing is the traceroutes are fine, its just HTTP won't load. I can even access my SSH client, so I'm totally confused. There has been no official announcement from them so I assumed it was an isolated problem with my server. They haven't admitted its a problem at their end yet.
Posted by Scott.Mc, 07-20-2008, 09:36 AM
GNAX did recover for us right there however your server is still down Jack.
Posted by zymic, 07-20-2008, 09:48 AM
Their whole network is still not working for me, I did manage to get the support website up, but now that doesn't load again. As mentioned earlier, this is what our members are reporting for netdepot.com/gnax.net: http://www.zymic.com/forum/index.php?showtopic=10670 As you can see its a worldwide problem, I originally assumed that it was just the UK but people from the USA, AUS etc also have problems which are still existing as I type this message.
Posted by bsasninja, 07-20-2008, 09:51 AM
Hi, I have 4 servers in GNAX and everything is fine and I can access perfect to gnax or netdepot but no to your server. The problem is in your server (maybe hanged, kernel panic, bad network card), did you tried the avocent remote reboot tool of them??? But according to your problem and packet lost issues, seems more like some ddosing attack to the rack your server is. Try to contact engineering by phone, this happened to me once (for half an hour) and they mitigated the problem inmediately. The GNAX/Netdepot phone is 404-230-9150 (the may request you the ticket number, but tell them that you have a connection problem and you cant access to the kayako support desk) Good luck! Last edited by bsasninja; 07-20-2008 at 09:58 AM.
Posted by zymic, 07-20-2008, 09:57 AM
bsasninja no its not specific to my server, yes my server may have problems to do with httpd but I'm talking about their network in general and other IP's/servers on their network. For example, do you use any uptime test companies? Pingdom.com is what I use, and none of the sites with gnax result in being up in the tests which use multiple probe servers in different countries. Last edited by zymic; 07-20-2008 at 10:04 AM.
Posted by jNive, 07-20-2008, 10:02 AM
it would appear that pings are fine - but traffic on ports like 80 and 21 etc are periodically dropping or getting blocked. Even trying lynx http://www.gnax.net or http://www.netdepot.com from one of my netdepot servers is failing, but pings are fine. Traffic from my gnax servers to external hosts seem to be suffering the same issue, as well as external access to those ports on my own servers. My Inter-site VPNs etc are fine, just seem to be FTP/HTTP related ports - odd to say the least http://www.gnax.net/mrtg <--- not working https://www.gnax.net/mrtg <---- working Last edited by jNive; 07-20-2008 at 10:06 AM.
Posted by zymic, 07-20-2008, 10:05 AM
jNive, have you contacted them about it? What do they have to say? Its just my ticket is still being worked on and its not apparent that they have established its a problem at their end.
Posted by bsasninja, 07-20-2008, 10:06 AM
Weird, no issues here, maybe some router at the Gnax DC went mad. Talk to them about this issues to do the respective checks from outside their network to see if there is some problem at some particular point.
Posted by zymic, 07-20-2008, 10:08 AM
Yes jnive, exactly, I spotted that earlier that HTTPS is the only thing which seems to be working. What an entirely confusing problem, just plain weird.
Posted by Scott.Mc, 07-20-2008, 10:08 AM
Hmm why isn't: https://www.gnax.net/mrtg/ showing any activity drops? They were totally down, all of our monitoring stations (3 in the US, 1 in the UK and 1 in Australia) all picked several parts of GNAX as down, yet their graphs are showing like nothing happened?
Posted by bsasninja, 07-20-2008, 10:10 AM
Lets open some tickets to gnax support so they can investigate this issue. Maybe is a silly problem and can be fixed right now.
Posted by zymic, 07-20-2008, 10:12 AM
Yeah I have a ticket with them, but my impression is they are unaware of any problems, else they would of already announced. Anyway I gave them this thread URL to see if this helps them. I also made a public uptime log here: http://www.pingdom.com/reports/jrw6s...th=7&year=2008 I started this about 20 mins ago. Please note its best to look at the % uptime instead of the time in minutes as this test has only started and does not represent the day as a whole. I noticed this problem 5AM GMT yesterday morning. Last edited by zymic; 07-20-2008 at 10:25 AM.
Posted by keywolf, 07-20-2008, 10:13 AM
Not working for me either, Might be a routing issue in Europe or somethings happened, Im in the UK on Virgin Media, Its fine when I access it from my Windows box in LA.
Posted by bsasninja, 07-20-2008, 10:14 AM
Scott there is not a connection issue, is a router problem which is causing conflict with some ports. Anyways the server of our friend is down for other problem, have you tried to the reboot the server???
Posted by Scott.Mc, 07-20-2008, 10:18 AM
All of our systems have recovered, we had a total of 12 different systems all down which is ALL of the remaining servers we have gnax. Nothing I know of that was at gnax was available. The issue the OP pointed out is something else and I think this is just a coincidence.
Posted by bsasninja, 07-20-2008, 10:25 AM
I did a check at http://internetpulse.net and http://www.internettrafficreport.com and everything seems fine worldwide. So I think this intermitent issue is taking place at Gnax DC.
Posted by bsasninja, 07-20-2008, 10:27 AM
zymic, did you tried to reboot your server via web???
Posted by Scott.Mc, 07-20-2008, 10:29 AM
Indeed, although I can't wait to see what it is because gnax, steadfast and fidelityaccess were all down at similar times.
Posted by activa, 07-20-2008, 10:32 AM
I have some servers with them . all is up just one that sufring this isseu . also gnax + netdepot not accessible for me .
Posted by bsasninja, 07-20-2008, 10:35 AM
zymic did you called at gnax phone? Is the most effective cause the will answer you. You can also try at engineering@gnax.net
Posted by jNive, 07-20-2008, 10:39 AM
Also i want to reiiterrate that ping traffic is not affected in this case - only port 21 and 80 so far from the looks of it. (both inbound and outbound) through a certain point internal to gnax - since i cannot reach http://www.gnax.net from my netdepot servers (2 servers on two different ranges).
Posted by zymic, 07-20-2008, 10:41 AM
Scott managed to fix the DNS on my server and it now loads, HOWEVER this does not mean their network issue is fixed. Although I can load my server their websites are still down, very strange.
Posted by bsasninja, 07-20-2008, 10:46 AM
I can access http://www.zymic.com/ without problems.
Posted by zymic, 07-20-2008, 10:51 AM
I should hope you could, WebNX never has any outages, silly In other words, that is with another provider. Anyway to sum it up, they had network issues, which are now resolved, and I think the problem now resides in the fact their website is just messed up, e.g. http for their sites doesn't load, but https does, etc.
Posted by Henrik, 07-20-2008, 11:14 AM
I can confirm this, trying from TeliaSonera in Sweden.
Posted by hzDylan, 07-20-2008, 12:41 PM
I have been informed by a gnax tech that their network engineer is currently working on this. I was a bit puzzled myself at first as I wasn't able to get to gnax.net but I could get to gold.gnax.net and mrtg.gnax.net - traceroute reports were fine. A little trial and error..and a helpful GNax tech solved the riddle.
Posted by jNive, 07-20-2008, 01:54 PM
probably a result of a Cisco ASA incorrectly detecting an application attack etc, and banning certain hosts/networks - happened somewhere else recently.
Posted by sailor, 07-20-2008, 08:30 PM
we dont have cisco asa's
Posted by jNive, 07-20-2008, 08:54 PM
well it was just a theory lol :-) - as i said, something else occured somewhere else that was using them where they started filtering traffic incorrectly
Posted by hzDylan, 07-21-2008, 12:33 AM
Just FYI. I was told over the phone this evening that their main website was targeted (ddos). As a result - IP ranges were blocked. I was told that the ban/blockings will be lifted shortly. my IP range is 70.241.x.x and still no direct access to "gnax.net".
Posted by gilman01, 07-21-2008, 09:24 AM
Is anyone stillhaving problems with their servers. Mine (East Coast US) is still down. In fact I cannot get to GNAX forum, cannot get to my WHM account, ssh into my server nor get it to resolve by checking the domain name.
Posted by sailor, 07-21-2008, 10:13 AM
can you get in now?
Posted by hzDylan, 07-21-2008, 10:39 AM
It is working for me now.