joe's datacenter is down?

Portal Home > Knowledgebase > Articles Database > joe's datacenter is down?

Posted by tubcan, 11-07-2010, 04:58 AM
I've a box with them and its' been down since at least couple of hours.. Their website is down too Anyone else?
Posted by DanKl, 11-07-2010, 05:09 AM
Same here, filed a ticket 30 mins ago.
Posted by arthurleung, 11-07-2010, 05:22 AM
Oh, it is the network which is down not the box? I couldn't connect to my box for a couple of hours. Assumed the box stopped working and I sent a reboot request instead.
Posted by The Universes, 11-07-2010, 05:40 AM
Looks like their web site is down as well.
Posted by net, 11-07-2010, 05:49 AM
Moved > Providers and Network Outages and Updates.
Posted by arjones85, 11-07-2010, 05:54 AM
They are down for me as well. A traceroute for me shows the peering point between Comcast and Cogent appears to be broken. Hope this gets resolved soon. According to my outage alerts I got on my cell it's been an hour now.
Posted by lifetalk, 11-07-2010, 05:58 AM
Down for me too. Been down for over an hour now.
Posted by plumsauce, 11-07-2010, 05:59 AM
It's a bit more serious than that. http://just-ping.com/index.php?vh=joesdatacenter.com&c=
Posted by tubcan, 11-07-2010, 06:00 AM
no responses for any of the tickets so far, it seems like everyone's still sleeping there which means no action will be taken before 5-6 hours from now... such a shame
Posted by lifetalk, 11-07-2010, 06:03 AM
That's quite likely, although I hope it's quicker than that. I'll be switching over to another provider, probably, once they're back up.
Posted by The Universes, 11-07-2010, 06:04 AM
Definitely: Its unfortunate. Hopefully Joe can get everything back up shortly.
Posted by arjones85, 11-07-2010, 06:11 AM
What leads me to believe that this is an issue outside of the DC is the fact that the first hop directly off of Comcast for me dies. I know for a fact that normally right off of Comcast the route I normally take has several intermediate hops before getting to the DC. I could be wrong though.
Posted by plumsauce, 11-07-2010, 06:14 AM
You will see that behaviour when a bgp route has been withdrawn. The first router knows that there is no point in going any further.
Posted by plumsauce, 11-07-2010, 06:17 AM
As the network appears down, your ticket may not have reached them, or they may not be able to respond due to the network outage. So, it is likely that while they would *like* to respond, they cannot. Of course, it would be nice for affected customers if they popped in here to post. A likely scenario is that they were alerted via SMS and are feverishly working on it.
Posted by The Universes, 11-07-2010, 06:32 AM
They use a hosted version of Ubersmith (which is offsite), so their customer/support portal is still online and accessible.
Posted by WII-Aaron, 11-07-2010, 06:32 AM
They are answering tickets. The issue is with one of the power busses running through the building.
Posted by tubcan, 11-07-2010, 06:45 AM
Yup, I got an answer as well 20 minutes ago.. it's always good to hear from them!
Posted by dredy, 11-07-2010, 06:48 AM
i haven't got any response to my ticket yet. what are they saying?
Posted by DanKl, 11-07-2010, 06:54 AM
That they know about issue and are working on it right now. No estimates on downtime yet.
Posted by dredy, 11-07-2010, 08:48 AM
still down!!! from more than 4 hours!
Posted by WII-Aaron, 11-07-2010, 09:03 AM
They are working as fast as they can. Im sure they will update you shortly.
Posted by lifetalk, 11-07-2010, 09:08 AM
I got a response too, although it was generic. Told me about the power bus issue. 4 hours (and counting) is a lengthy downtime for me, but I guess I can't help but wait!
Posted by arjones85, 11-07-2010, 10:09 AM
If a power line is having an issue in the building that has knocked out one of your core routers, run an extension cable to plug that bad boy in somewhere else! lol
Posted by styx, 11-07-2010, 10:10 AM
6 hours of downtime for co-location customers as well. Im one of them. First month of service, first impression.
Posted by lifetalk, 11-07-2010, 10:13 AM
6 hours is too much for me. So I asked Joe for an ETA if he had one and he says it'll be a couple more hours before services are restored. Not happy!
Posted by DanKl, 11-07-2010, 10:15 AM
Well, this is definitely not common with them. Over 5 monthes I've had short outages only twice, IIRC, never longer than 1-2 hours. Although day of outage is very frustrating, I hope they will offer something to compensate for this.
Posted by jackpx, 11-07-2010, 10:22 AM
I hope that they can soon solved the problem.
Posted by WII-Aaron, 11-07-2010, 10:34 AM
It may be as short as an hour. Our electrician and the building electrician are both working on getting the bus section replaced.
Posted by chasebug, 11-07-2010, 10:44 AM
This is taking too long.
Posted by arjones85, 11-07-2010, 10:44 AM
A couple more hours isn't bad. At least its not going to be an all day thing. Means I don't have to look for a temporary VPS to restore my offsite backups on to at least get email back up. Everyone *does* keep daily offsite backups right? If not, this is a good reminder to do so!
Posted by tubcan, 11-07-2010, 10:47 AM
Ironically, 8 months ago I got my first box with joe's and in the first day, specifically after 4 hours from the welcome email, their network got down in similar situation (8-12 hours downtime back then). You can imagine how my first impression was! But to be fair, since that 8 months, I never had a single issue with them
Posted by styx, 11-07-2010, 10:49 AM
6-7(8-12) hours of downtime every 8 months is too much.
Posted by chasebug, 11-07-2010, 11:04 AM
Still down?
Posted by styx, 11-07-2010, 11:12 AM
Oh yeah. My Zabbix shows its about to hit 7 hour mark in 5 minutes.
Posted by GCM, 11-07-2010, 11:12 AM
Appears so. I'm sure Joe's on it. Last edited by GCM; 11-07-2010 at 11:17 AM.
Posted by HostKoi, 11-07-2010, 11:18 AM
Yeah my box with them is down too. Hope they get back online soon.
Posted by lifetalk, 11-07-2010, 11:33 AM
Just finished paying for a server on HIVELOCITY. I'm moving as soon as my box comes online. This downtime is costing me too much and I can't risk something like this happening ever again, in the future.
Posted by David, 11-07-2010, 11:37 AM
So stop moving from ****** facility to ****** facility and pay for a bit more redundancy.
Posted by lifetalk, 11-07-2010, 11:39 AM
That is my next plan. My priority is to get the site(s) back up. I have the backups and everything, but by the time I restore them to an alternative vps and change DNS and all, JoesDC will be back up and running. So once am up with the new facility, I'll look into redundancy options.
Posted by chasebug, 11-07-2010, 11:45 AM
Why does it take so long to replace that faulty hardware? Surely they must have planned for such faulty hardware and an expected replacement time?
Posted by chasebug, 11-07-2010, 11:58 AM
Their site is now up but my server is still down.
Posted by desynced, 11-07-2010, 12:07 PM
I agree its time to start looking into other datacenters. Seems alot of the other ones recommended on here at least have a guaranteed uptime on their site, which I cant find on joesdatacenter.com. I'm sure the small built in UPS in my server is dead by now. lol Last edited by desynced; 11-07-2010 at 12:18 PM.
Posted by desynced, 11-07-2010, 12:12 PM
Isnt your datacenter in the same building? Why isnt your site/service down also?
Posted by CretaForce, 11-07-2010, 12:14 PM
Maybe your server needs a fsck ?
Posted by lifetalk, 11-07-2010, 12:15 PM
Doubt that. My box is still down, too. So apparently, the only thing that is up is their site.
Posted by arjones85, 11-07-2010, 12:17 PM
It's under their network overview page. 99.9% Kinda sucky this is my first month with them, but as long as they get it fixed and it doesn't happen again I'm easy going
Posted by desynced, 11-07-2010, 12:20 PM
Well since we're all here, I have a question for fellow JDC customers. When you do a constant ping to your server, does every 5th ping jump up and then back down on the next ping? (i.e. 50ms, 55ms, 50ms, 52ms, 180ms, 56ms, 52ms, etc).
Posted by tubcan, 11-07-2010, 12:21 PM
9 hours of downtime exactly...
Posted by chasebug, 11-07-2010, 12:23 PM
Your server is back online?
Posted by lifetalk, 11-07-2010, 12:25 PM
I don't quite remember, but I'll test this out once my box is back online.
Posted by desynced, 11-07-2010, 12:28 PM
OK Just curious. I have failed to check my nagios logs for a while now, but now since this is happening I checked and it seems for the last month, every 5th - 7th ping, it jumps up (sometimes in the 300ms+ range) and right back down again. I guess customers are waking up and getting on the internet. The phone calls to our support line is starting to get active. Real frustrating when I cant give them a ETA on when their web site will be back online. Last edited by desynced; 11-07-2010 at 12:32 PM.
Posted by styx, 11-07-2010, 12:36 PM
This is the question that bothers me the most too. We are building reputation here and its going downhill. But its so easy to ruin it.
Posted by arjones85, 11-07-2010, 12:47 PM
I'm trying to get an ETA on my box being powered back on given their main site is back online.
Posted by WII-Aaron, 11-07-2010, 12:48 PM
We are in the same building but we don't pull power from the shared building bus (except for some lights). We have our own, seperate power feeds so we wern't affected, except that we use the same elevators that are also on the fritz.
Posted by lifetalk, 11-07-2010, 12:48 PM
Same here. I haven't had a response to my ticket yet, though. Let's hope it doesn't take too long.
Posted by WII-Aaron, 11-07-2010, 12:49 PM
Some temporary power was run to get core systems back up and going but there isn't enough to power everyone. Current ETA from the electrician is 1-2 hours.
Posted by arjones85, 11-07-2010, 12:53 PM
Thanks for keeping us updated on this! I really appreciate you spending the time to keep us informed even though you aren't affiliated with Joe =)
Posted by WII-Aaron, 11-07-2010, 12:54 PM
I take that back. All our stuff on the roof is out. forgot about that.
Posted by desynced, 11-07-2010, 12:56 PM
It's bright and sunny outside. Run to Harbor Freight and buy lots and lots of those $30 solar panels.
Posted by WII-Aaron, 11-07-2010, 12:58 PM
Joe and I are friends. I came down here at 2:30am to help out. He would have done the same for me. Believe me when I say no one is working harder than he is trying to get this resolved. There are other DCs just showing up, 9 hours after they went down, who don't know anything is going on. I'm not talking about little guys. These are top 3 provider guys. (*cough* sprint *cough*) Be glad you host with Joe who doesn't just leave you hanging until morning.
Posted by desynced, 11-07-2010, 01:04 PM
Well, Joe learned from the last outage to not rely on Cogent 100% for their network, so he invested some money and got a backup pipe. Hopefully Joe now invests a little more to get a redundant power source so this doesnt happen again.
Posted by lifetalk, 11-07-2010, 02:04 PM
Just got an email saying service should be restored within the next few hours. That's reassuring, to say the least. But the 'next few hours' phrase is not exactly what I was looking forward to. This is going to be one lengthy downtime, won't it?
Posted by styx, 11-07-2010, 02:08 PM
Its the same email everybody gets. 2 and 1/2 hours ago ETA was 2 hours. Nothing we can do. Ah those electricians, can somebody get them Dunkin Donuts coffee so process can speed up. And... downtime hits 10 hours. Last edited by styx; 11-07-2010 at 02:12 PM.
Posted by Preetam, 11-07-2010, 02:21 PM
I'm starting to get worried...
Posted by danclough, 11-07-2010, 02:25 PM
Sucks to hear that a clogged sink managed to take multiple DCs off line. For how little Joe charges for his services, the fact that he jumps on the issue immediately and doesn't stop working until it's fixed makes me feel a lot more comfortable that my non-critical server is hosted at Joes. For everyone affected who's losing more money than they can handle... redundancy ain't free, maybe it's time to stop pinching pennies! EDIT: Pingdom reports my server is online!
Posted by WII-Aaron, 11-07-2010, 02:26 PM
Don't be. Power has been restored to the east bus. The equipment will be brought back online starting in a few minutes.
Posted by JSCL, 11-07-2010, 02:26 PM
Why are you all moaning? This is not a bash at Joe - you pay cheap, yet expect the world. Joe works his hardest to maintain a stable network and facility - you're all happy to pay his cheap prices but happy to bitch when there's a downtime (who isn't?). If you were paying $200/month elsewhere for a $100/mo box at Joes I'd understand a little bit more because you expect the higher level of service - but the way it seems is that those who pay less in the industry always moan the most than those that pay more - this is a prime example for sure.
Posted by arjones85, 11-07-2010, 02:52 PM
Any further updates? Keep hoping to see my box return some pings. Are they physically going around to make sure all the boxes power back on?
Posted by Preetam, 11-07-2010, 02:55 PM
Our server is back up.
Posted by WII-Aaron, 11-07-2010, 03:00 PM
I haven't been up there sine my last post. I'm sure he's working on it.
Posted by 3-rx, 11-07-2010, 03:03 PM
My box is UP. thx
Posted by chasebug, 11-07-2010, 03:21 PM
Mine is still down!
Posted by lifetalk, 11-07-2010, 03:22 PM
Same. They'll be getting online gradually. Give it some time, I guess.
Posted by jscieza, 11-07-2010, 03:27 PM
Yep, his site is down for me too.
Posted by tubcan, 11-07-2010, 03:28 PM
box is up here, although I still have timeouts.. not that stable yet, maybe its matter of time.
Posted by chasebug, 11-07-2010, 03:34 PM
I can't believe there are no redudant power at Joes...
Posted by chasebug, 11-07-2010, 03:36 PM
Now I am getting "From xx.xx.xx.xx icmp_seq=3 Time to live exceeded" which isn't even my IP address when I ping my server.
Posted by Zack-Hancock, 11-07-2010, 03:41 PM
Dear Customers, We are currently offline due to a power issue in the building. We have staff here ready to get things back online as soon as power is restored. We have our own electricians working with the building to get the problem fixed. We should have service restored within the next few hours. A drainage sink on one of the lower floors got clogged and water backed up and leaked down the electrical closet onto one of the main power buss going through the building. The bus shorted out and the breaker in the basement flipped. Our staff was on site when it happened and contacted the building maintenance right away. We have been working with them since. A piece of the power bus was damaged and needs to be replaced and this is part of the reason why it is taking so long to get it restored. I will update with more information as soon as its available. Joe Morgan - Owner Nov. 7th @ 11:00 AM Central Just found this on there website.
Posted by lifetalk, 11-07-2010, 03:42 PM
My box is up. Pings seem to be fine. What about the rest of you guys?
Posted by chasebug, 11-07-2010, 03:44 PM
Still down! "Time to live exceeded" Routing issue or problem with router. Already over 10 hours, please bring my server back up!
Posted by Zack-Hancock, 11-07-2010, 03:46 PM
Contact there support. The website is backup for myself now and not with a come back later messege. seems back to normal.
Posted by chasebug, 11-07-2010, 03:50 PM
I opened an emergency ticket already, still no response to it.
Posted by benj114, 11-07-2010, 03:53 PM
Both my box's still down, even attempted to access IP-KVM's to see if I just needed to turn the server's on, no luck.
Posted by chasebug, 11-07-2010, 04:06 PM
Finally my box is up. Over 11 hours of downtime. Given the circumstances, would a more expensive provider not run into this issue? It seems this clogging or whatever is really unusual?
Posted by sam0, 11-07-2010, 04:13 PM
Our box is still down.
Posted by benj114, 11-07-2010, 04:17 PM
This is not the providers fault and could have happened to any company no mater what you pay. The other provider I have server's at (they have a few cages at an equinix facility) was down for all most over a day last year when there was a fire on the floor below it at another providers cage and the fire department cut the power to the building, including all backup power.
Posted by chasebug, 11-07-2010, 04:26 PM
I understand these are extraordinary issues. Would having backup power prevent this latest outage at Joe's? If another provider such as SoftLayer, LiquidWeb, HiVelocity, DedicatedNow, etc. that are not budget providers, has this same problem, would they also have been down or would their backup power take over?
Posted by jackpx, 11-07-2010, 04:43 PM
No battery backup system could keep the servers online for 10 hours
Posted by RyanD, 11-07-2010, 04:43 PM
I don't know the other provider's specific infrastructure. But in general, yes you have a primary feed and a secondary feed from your generator or secondary power source. In this case it appears their only power feed was via the shared building bus, if it goes down, it all goes down. We have a similar setup at our location at 55 Marietta, we have our primary power off of the main building (shared) bus but we have our own dedicated generator and separate power systems. Earlier this year the building had to do maint on the bus, we simply ran on generator for 7 hours while they did the work. But yes, these are the types of sacrifices you make for a budget provider. Look at FDC they don't have a generator or a UPS.
Posted by freestater, 11-07-2010, 04:44 PM
FWIW, I've got a full 42U rack at Joe's. We've also hosted that rack at another provider in this same building (not WI). At the previous host, we paid about twice what we pay Joe, to folks that weren't as friendly as Joe. However, in that year, we had 0 downtime. We moved upstairs to Joe's and I've been pretty happy -- been here for about 14 months. There have been outages, but not many, and Joe's always on top of them. (Unfortunately, today's will be the first that costs me $.) A couple things for those who aren't familiar with the building. #1, in terms of redundant power, this building offers two different power grids. Joe is only tied into 1. (Although I really hope he ties into both now...our rack is already wired for it; just waiting.) #2, in terms of "redundant" power, some of the DC's in the building have their own generators on site; Joe does not. From what I've been told (I had to go on site to fire a few things back up and chatted up another tech from a different DC), today's power outage only affected floors 17+...Joe is on 26. Joe, thanks for staying on top of things...and please...it's time to invest in redundant power!
Posted by UNIXy, 11-07-2010, 04:55 PM
Higher end DCs have the option of an A/B setup where the A and B circuits are from different power grids. Not only that but the server has to be a high end one too. It needs to have two redundant power supplies for the A/B setup. But you have to have the means and business requirement for such configuration. Regards
Posted by benj114, 11-07-2010, 04:56 PM
I think the main thing we should be thankful is that no one got hurt from what I've been told. Water + Power = bad combo that could have hurt someone real bad or even killed them. But the bigger question is, how did water get near the electrical closet when to my understanding that area "should" be water tight.
Posted by RyanD, 11-07-2010, 05:24 PM
The oak tower is a very old building, I doubt anything there is 'water tight'
Posted by JSCL, 11-07-2010, 05:54 PM
We could have seen a Take 2 of the The Planet's power room explosion but in the Oak Tower - now that would be VERY dangerous.
Posted by benj114, 11-07-2010, 06:17 PM
Well my one server appears to either be busted or they appear to have bad network issues.
Posted by sam0, 11-07-2010, 06:38 PM
Still down, I asked for an ETA over an hour ago and my ticket was put on low with no response. Does anybody else have server(s) still offline at Joes? We only signed up with Joes a few weeks ago, and we weren't impressed with the setup either so first impressions are pretty poor to be honest.
Posted by DanKl, 11-07-2010, 06:54 PM
My server got online about 2 hours ago, but much later than first reports in this thread. Joe silently closed the ticket when he booted my server, so I think in your case it means they are still working through the queue. In general, they don't leave tickets hanging, so if status isn't 'closed', they will deal with you sooner or later.
Posted by sam0, 11-07-2010, 07:27 PM
Our server came back online about 30 mins ago.
Posted by arjones85, 11-07-2010, 08:31 PM
I came back online about 2 PM, so 4 and half hours ago
Posted by HoratioE, 11-07-2010, 11:06 PM
Good to hear that they got things sorted out. Was it Quiznos' fault?
Posted by DataShack, 11-08-2010, 08:28 AM
The only other DC's I'm familiar with in Oak Tower are DTC and WII. They both have seperate feeds from the main building power and both have their own generators. DTC uses smaller, rack mounted UPS systems in their cabinets and WII uses a combination of larger UPS systems and smaller rack and shelf mounted. DTC is a managed provider but WIIs power is better so the budget vs. nonbudget thing isnt always true. Now that I think about it. I even think there's another DC in the building that buys power from WII and not from the building but I forget who that is.
Posted by styx, 11-09-2010, 01:35 AM
Am i missing something? Its 30 hours+ after everything has come up but was there any statement letter explaining what exactly happened?
Posted by RobM, 11-09-2010, 02:14 AM
There's a post on page 6 about what happen there.
Posted by styx, 11-09-2010, 02:19 AM
So you consider it an official statement from a company, some place on some forum? So clients have to find out themselves by googling about what happened that night? Im as a client will be really disappointed if they do not send an official company letter explaining the situation.
Posted by CD Burnt, 11-09-2010, 02:33 AM
thread cleaned. keep it on topic.
Posted by lifetalk, 11-09-2010, 03:21 AM
They sent out an email explaining what had happened, so yeah, that is pretty much an official statement. They also posted the statement on their website while they were fixing it.
Posted by styx, 11-09-2010, 12:48 PM
As a colocation customer having 3 staff members added to their billing portal i can assure you that none of those 3 staff members got any email with an official information.
Posted by BudWay, 11-20-2010, 12:42 PM
That's why people (some) place drain on the bathroom floor. It's good to see Joe worked hard in getting everybody online