Is Joe's DC down for anyone else?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Is Joe's DC down for anyone else?

Posted by desynced, 04-01-2010, 04:57 PM
My server there seems to have went down and I cant even get to Joe's DC main web site. Anyone else having problems?

I tried calling them and it rings once, then goes to a busy signal.
Posted by fgkfgk, 04-01-2010, 05:05 PM
the host does not resolve.

This is why....:

$ host NS1.MORGANSITES.NET
NS1.MORGANSITES.NET has address 208.94.247.155
hoNS1.MORGANSITES.NET mail is handled by 10 MORGANSITES.NET.

$ host NS2.MORGANSITES.NET
NS2.MORGANSITES.NET has address 208.94.247.155
NS2.MORGANSITES.NET mail is handled by 10 MORGANSITES.NET.
Posted by mugo, 04-01-2010, 05:22 PM
Yep, down for me.....
Posted by 227b, 04-01-2010, 05:23 PM
JOES Data Center down for more than 40 minutes now ... (counting)....
Posted by 227b, 04-01-2010, 05:24 PM
Have anyone tried to call them ? Since the main website is down, no information about their phone number.
Posted by desynced, 04-01-2010, 05:27 PM

Quote:

Originally Posted by 227b

Have anyone tried to call them ? Since the main website is down, no information about their phone number.

1-877-JOE-DATA but it only rings once and then gives you a busy signal. Ive tried over 30 minutes now. Also checked our remote monitoring service and looks like they were down last night for a little while too.
Posted by mugo, 04-01-2010, 05:31 PM
Joe's should definitely think about having some Name Servers outside of their network...

Knowing Joe, he's all over the outage, though.
Posted by 3-rx, 04-01-2010, 05:33 PM
Yep, down for me..... :-(
Open ticket 30 minutes ago, no response
Posted by tubcan, 04-01-2010, 05:52 PM
Same here

opened ticked almost 50 mins no response as well, ironically I just received the welcome email for my new box only 5 hours ago...
Posted by desynced, 04-01-2010, 05:53 PM
Over an hour now. Still down and still cant get through on the phone.
Posted by MikeTrike, 04-01-2010, 06:07 PM
I would assume they house all their systems in house. i.e. PBX/Mail/Web/etc.
Posted by 3-rx, 04-01-2010, 06:18 PM
Online !
thank you
Posted by WII-Aaron, 04-01-2010, 06:19 PM
Distribution switch took a crap. It's coming back online now. If you're not up already give it 5 minutes. I just came back downstairs and he was pasting VLAN info in.
Posted by desynced, 04-01-2010, 06:30 PM

Quote:

Originally Posted by WII-Aaron

Distribution switch took a crap. It's coming back online now. If you're not up already give it 5 minutes. I just came back downstairs and he was pasting VLAN info in.

Thanx for running up and checking out whats going on Aaron.
Posted by tubcan, 04-01-2010, 06:47 PM

Quote:

Originally Posted by WII-Aaron

Distribution switch took a crap. It's coming back online now. If you're not up already give it 5 minutes.

my box still down, anyone else still down?
Posted by desynced, 04-01-2010, 07:08 PM

Quote:

Originally Posted by tubcan

my box still down, anyone else still down?

It's been constantly going up and down since Aaron posted. I had to turn off the remote monitoring services since our phones kept getting spammed with servername - up and serversname - down. As of this posting, down.
Posted by MikeTrike, 04-01-2010, 07:16 PM
April fools?

erase startup-config
reload
yes

Posted by WII-Aaron, 04-01-2010, 07:22 PM

Quote:

Originally Posted by desynced

It's been constantly going up and down since Aaron posted. I had to turn off the remote monitoring services since our phones kept getting spammed with servername - up and serversname - down. As of this posting, down.

Had to revert to a different backup. Once you download it from the TFTP server the switch has to literally paste it inside itself. It takes a little bit for it to get through it. The config is pages and pages long. it's almost done.
Posted by WII-Aaron, 04-01-2010, 07:23 PM

Quote:

Originally Posted by MikeTrike

April fools?

erase startup-config
reload
yes

Actually this is an Extreme Alpine so it would be:

unconfig switch
reboot
yes

But no, that's not what happened. Management card went bad.
Posted by MikeTrike, 04-01-2010, 07:26 PM
Naturally cards going bad is not good. I'm not familiar with anything other than Cisco (and maybe some Adtran due to its similarity).

Another April fools joke would be to release a few Kinder and Pre-K classes inside the DC with all cabs and cages open. With the exception of where main power feeds are an telling them to go... play... have fun!
Posted by JOEsDC, 04-01-2010, 08:10 PM
Hey just wanted to update everyone. We had a management card go bad not a lot we could do about that. We did however have a spare card on hand and do daily backups of the configs so we were able to fully recover and everyone should be back up now. The datacenter was actually up it was only one switch that suffered this problem. We apologize for the downtime and understand how frustrating being down is.
Posted by paul_b, 04-01-2010, 08:19 PM
Good to see Joe was on top of this.

Need to put some external name servers out there and maybe do the same with your website/support site. Simple enough and really helps in case of problems.
Posted by JOEsDC, 04-01-2010, 09:44 PM

Quote:

Originally Posted by paul_b

Good to see Joe was on top of this.

Need to put some external name servers out there and maybe do the same with your website/support site. Simple enough and really helps in case of problems.

We actually host the support site off site for that reason and have a cable internet connection here in the office to keep in contact just in case of something like this. I had some problems with the external dns server and had moved it back internal until I got it solved. Since I had moved it of course this is when we have an issue. I was also training a new guy and when it failed and unfortunately he has not been trained on the different communication methods in cases like this so there was nobody else around to do it. I'm just glad the spare card and backups all worked as planned and that we were able to get it 100% back to the point when it failed. All the planning paid off this time and some little fixes with dns and training will make something like this go even smoother if it happens again. I want to thank all my customers who have been very supportive and understanding about this. We have had few issues like this but its good to see customers stay behind us when we do.