Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > FDC Servers down?
FDC Servers down?
| Posted by SparoHawk, 07-23-2009, 06:29 PM |
Is anybody else having trouble with FDC?
I cannot access neither my sites nor FDC's site.
Genral down? |
| Posted by Victor Lugo, 07-23-2009, 06:30 PM |
Is anyone here able to confirm that the FDC Chicago datacenter has lost internet connectivity?
This seems to include the front-end website and CRM system, along with any and all servers in the datacenter. |
| Posted by zenex5ive, 07-23-2009, 06:30 PM |
|
Yes, FDC is down at the moment. |
| Posted by zymic, 07-23-2009, 06:31 PM |
|
All 4 of my servers down here. |
| Posted by flashwebhost, 07-23-2009, 06:32 PM |
|
My servers at FDC are down. Any news ? |
| Posted by donbiz, 07-23-2009, 06:32 PM |
|
Yeah my colo'd server there is currently down at the Chicago datacenter. |
| Posted by The-Pixel, 07-23-2009, 06:33 PM |
|
Yep, it's completely down. I haven't heard any news yet in regards. |
| Posted by kamal_s, 07-23-2009, 06:33 PM |
|
yes all of our servers down |
| Posted by DME-Geoff, 07-23-2009, 06:34 PM |
Also with servers down at FDC Chicago.
Anyone have updates? |
| Posted by suhailc, 07-23-2009, 06:35 PM |
|
Yes all servers at FDC Chicago down. |
| Posted by brc_csf, 07-23-2009, 06:35 PM |
|
Chicago server down too |
| Posted by infiniteecho, 07-23-2009, 06:36 PM |
|
Ours are all down too. |
| Posted by InnovusHost, 07-23-2009, 06:37 PM |
|
My Chicago colo is down, but my Denver server is fine. The crm is also down. |
| Posted by alexdale, 07-23-2009, 06:41 PM |
all my 5 servers are down.
omg fdc is back with vengeance for no major downtimes in months. lol.
-_- |
| Posted by brc_csf, 07-23-2009, 06:42 PM |
|
We are FDC clients for just some months. Does this happen lot of times? |
| Posted by BELLonline, 07-23-2009, 06:43 PM |
Same here, Chicago down only. I've tried going on to crm.fdcservers but that doesn't seem to work properly - Firefox is giving an 'untrusted connection' error:
Quote:
This Connection is Untrusted
You have asked Firefox to connect
securely to crm.fdcservers.net, but we can't confirm that your connection is secure.
Normally, when you try to connect securely,
sites will present trusted identification to prove that you are
going to the right place. However, this site's identity can't be verified.
What Should I Do?
If you usually connect to
this site without problems, this error could mean that someone is
trying to impersonate the site, and you shouldn't continue.
Technical Details
crm.fdcservers.net uses an invalid security certificate.
The certificate is only valid for *.opendns.com.
(Error code: ssl_error_bad_cert_domain)
I Understand the Risks
|
Seems strange. |
| Posted by whfsdude, 07-23-2009, 06:44 PM |
Quote:
|
Originally Posted by brc_csf
We are FDC clients for just some months. Does this happen lot of times?
|
Been a customer for two years and I can only remember one major outage which was a fire in the power vault that fed CBOT. |
| Posted by BELLonline, 07-23-2009, 06:44 PM |
Quote:
|
Originally Posted by brc_csf
We are FDC clients for just some months. Does this happen lot of times?
|
No, they've really improved since a few years ago, heres the uptime for one of my servers http://bellonline.co.uk/servers/?id=48876 |
| Posted by flashwebhost, 07-23-2009, 06:45 PM |
Quote:
|
Originally Posted by brc_csf
We are FDC clients for just some months. Does this happen lot of times?
|
I have been with them for 2 years, i don't faced much downtimes. |
| Posted by winger, 07-23-2009, 06:45 PM |
|
same here, servers and crm down |
| Posted by brc_csf, 07-23-2009, 06:45 PM |
Quote:
|
Originally Posted by whfsdude
|
Before it got down there was lot of lag. So I believe this is not power related but network. |
| Posted by The-Pixel, 07-23-2009, 06:46 PM |
Quote:
|
Originally Posted by brc_csf
We are FDC clients for just some months. Does this happen lot of times?
|
I've been with FDC for a very long time and I can assure you this does not happen. They are a wonderful datacenter. |
| Posted by CretaForce, 07-23-2009, 06:46 PM |
|
Do we know if it's network or power related? |
| Posted by SC-Daniel, 07-23-2009, 06:46 PM |
Yep, down from here as well  |
| Posted by brc_csf, 07-23-2009, 06:47 PM |
When downtimes like this happen, is there a phone we could contact them? Just to hear "Don't worry, no fire here  " |
| Posted by Quiky, 07-23-2009, 06:47 PM |
|
Second downtime in 24 hours.. last night 10:20PM EST -> 10:52PM EST, and today 6:25PM EST -> xx. |
| Posted by infiniteecho, 07-23-2009, 06:49 PM |
|
I already tried calling the number I have for the DC phone, it is busy. No use right now. |
| Posted by whfsdude, 07-23-2009, 06:50 PM |
Quote:
|
Originally Posted by Quiky
Second downtime in 24 hours.. last night 10:20PM EST -> 10:52PM EST, and today 6:25PM EST -> xx.
|
The one last night was only fore an isolated number of servers. I only had one down out of 4 last night and it was brief. I assumed that was a switch issue. |
| Posted by BELLonline, 07-23-2009, 06:50 PM |
Quote:
|
Originally Posted by brc_csf
When downtimes like this happen, is there a phone we could contact them? Just to hear "Don't worry, no fire here " |
I'd rather know that they were working on the problem rather than answering phones 
EDIT: sorry, quoted wrong person! |
| Posted by Victor Lugo, 07-23-2009, 06:51 PM |
I will assume the power outage was only a coincidence, and that the current downtime in the Chicago datacenter is network-related.
I too was noticing lag/timeouts previous to the downtime, which could mean several things (perhaps a combination).
As of this moment, the Chicago datacenter has been offline for 33 minutes. |
| Posted by infiniteecho, 07-23-2009, 06:52 PM |
Quote:
|
Originally Posted by BELLonline
I'd rather know that they were working on the problem rather than answering phones
|
There is a fine line to be walked here. Yes they should have all hands on the issue, but there should also be *some* communication to the public. |
| Posted by Quiky, 07-23-2009, 06:54 PM |
|
A simple twitter status, or a offsite blog a-la dreamhost would be lovely.. |
| Posted by The-Pixel, 07-23-2009, 06:54 PM |
Quote:
|
Originally Posted by infiniteecho
I already tried calling the number I have for the DC phone, it is busy. No use right now.
|
Perhaps the person managing the phones is busy in the datacenter. We can only hope  |
| Posted by CretaForce, 07-23-2009, 06:54 PM |
|
I hope it's network related. |
| Posted by whfsdude, 07-23-2009, 06:54 PM |
Quote:
|
Originally Posted by infiniteecho
There is a fine line to be walked here. Yes they should have all hands on the issue, but there should also be *some* communication to the public.
|
They should have a status page that has two servers in one DNS pool. Maybe this is something we should suggest to Petr. ... or maybe a twitter account? |
| Posted by admin-caperhosting, 07-23-2009, 06:55 PM |
|
in the meantime, why do they host there own website in there dc, so in the case of the most emergency possible, now there's no way to contact them etc... lol i dunno... |
| Posted by zurca, 07-23-2009, 06:56 PM |
Ohh dear,
I just hope is not one of those loooooong outages like the one we had about a year ago, their network was out for like 12 hrs I think.
cross my fingers.
other than that fdc been an excellent datacenter, hope they keep us updated here, cuz their phone line is bussy. |
| Posted by InnovusHost, 07-23-2009, 06:56 PM |
|
I am surprised they don't have their ticket system spread out over their two datacenters to prevent something like this. |
| Posted by brc_csf, 07-23-2009, 06:57 PM |
Quote:
|
Originally Posted by The-Pixel
Perhaps the person managing the phones is busy in the datacenter. We can only hope
|
How big is FDC ? (or how Small?)
I thought they should have someone just to answer phones |
| Posted by bummer6666, 07-23-2009, 06:57 PM |
|
network related issue that is currently being troubleshooted. Explanation for the outage will be posted on our forums and clients emailed as soon as the problem is fixed and we know what exactly happened |
| Posted by infiniteecho, 07-23-2009, 06:58 PM |
Quote:
|
Originally Posted by atbnet
I am surprised they don't have their ticket system spread out over their two datacenters to prevent something like this.
|
I would have thought they might have had CRM replicated in two locations with a VIP for public traffic.
Oh well.
Thanks for the update Petr!  |
| Posted by whfsdude, 07-23-2009, 06:59 PM |
Quote:
|
Originally Posted by bummer6666
network related issue that is currently being troubleshooted. Explanation for the outage will be posted on our forums and clients emailed as soon as the problem is fixed and we know what exactly happened
|
Thanks! Nice to know I don't have to worry about hardware coming back up from a cold boot. |
| Posted by The-Pixel, 07-23-2009, 06:59 PM |
Quote:
|
Originally Posted by brc_csf
How big is FDC ? (or how Small?)
I thought they should have someone just to answer phones
|
They are fairly big I think(2 or 3 different locations). You would think but they rarely have outages that I've seen and I've been with them for 1 or 2 years. So, I guess I can see why they don't have two different people but perhaps that's something they will look into.
They are doing everything they can to get the servers online and working properly. The best thing to do is update your clients and try to keep claim. Ensure them that technology is not perfect like we wish and there are complication at times. |
| Posted by The-Pixel, 07-23-2009, 07:00 PM |
Quote:
|
Originally Posted by bummer6666
network related issue that is currently being troubleshooted. Explanation for the outage will be posted on our forums and clients emailed as soon as the problem is fixed and we know what exactly happened
|
Thanks for the update! |
| Posted by Victor Lugo, 07-23-2009, 07:01 PM |
FDC has datacenters in both Chicago and Denver, what I don't understand is why a failover configuration was never set up.
The Denver datacenter (as most of you are aware) has not been affected, but the servers belonging to FDC's website and CRM are in the Chicago location.
Update: 43 minutes downtime at the FDC (Chicago) datacenter. |
| Posted by MMH-Moe, 07-23-2009, 07:01 PM |
|
Yea, my support tickets are rolling in now. |
| Posted by brc_csf, 07-23-2009, 07:04 PM |
Downtime happens everywhere. Glad to have a position.
Thanks. |
| Posted by JoCortland, 07-23-2009, 07:07 PM |
|
Strange, I haven't heard from ANY clients yet, and usually they are right on the mark. Thanks for the update, Petr. I have something to tell them anyhow. |
| Posted by admin-caperhosting, 07-23-2009, 07:08 PM |
|
Awsome thanks for an update FDC |
| Posted by Quiky, 07-23-2009, 07:09 PM |
|
And we are regaining connectivity. |
| Posted by Victor Lugo, 07-23-2009, 07:10 PM |
|
It's back up after 51 minutes of downtime. |
| Posted by infiniteecho, 07-23-2009, 07:10 PM |
|
Back up! wo0t |
| Posted by dfh_ie, 07-23-2009, 07:11 PM |
|
seems all back now |
| Posted by KMyers, 07-23-2009, 07:18 PM |
|
It seems to be comming back online |
| Posted by PersonalJ, 07-23-2009, 07:23 PM |
|
I just got back from eating out and it looks like there was a connectivity loss on my colocated machine. It does not appear to have been power related. |
| Posted by JoCortland, 07-23-2009, 07:27 PM |
|
All back to normal here. |
| Posted by admin-caperhosting, 07-23-2009, 07:28 PM |
|
ya were back up as well, but if anyone from fdc or anyone that knows specific details other then just network issues, please post or post the link to fdc current issue info Thanks! |
| Posted by The-Pixel, 07-23-2009, 07:30 PM |
|
Thank you FDC that outage wasn't to bad and you did a great job! Keep it up. |
| Posted by alexdale, 07-23-2009, 07:32 PM |
|
now back up |
| Posted by PersonalJ, 07-23-2009, 07:49 PM |
|
My server just became unreachable. |
| Posted by Unacom-Olisa, 07-23-2009, 07:50 PM |
|
down for me |
| Posted by whfsdude, 07-23-2009, 07:50 PM |
Quote:
|
Originally Posted by Unacom-Olisa
down for me
|
Yup. It bounced. |
| Posted by The-Pixel, 07-23-2009, 07:50 PM |
|
Geez! Not again. |
| Posted by suhailc, 07-23-2009, 07:53 PM |
|
And down again! |
| Posted by PersonalJ, 07-23-2009, 07:55 PM |
Quote:
|
Originally Posted by whfsdude
|
My server hadn't lost power during the first outage, I assume it's unrelated. |
| Posted by zurca, 07-23-2009, 07:55 PM |
It came back online, then outage again for a short period, then back on.
This may be a last minute tweaking, hope it stays back on now.
correction, is still on and off from my end, hopefully they resolve it soon. |
| Posted by The-Pixel, 07-23-2009, 07:57 PM |
Quote:
|
Originally Posted by zurca
It came back online, then outage again for a short period, then back on.
This may be a last minute tweaking, hope it stays back on now.
correction, is still on and off from my end, hopefully they resolve it soon.
|
Good idea. |
| Posted by alexdale, 07-23-2009, 07:59 PM |
|
sheesh down again -_- |
| Posted by HostcrateCEO, 07-23-2009, 08:01 PM |
|
i think we are going to be playing the bouncing game, you bounce online for a bit then back offline for a while until they get it sorted out. |
| Posted by brc_csf, 07-23-2009, 08:03 PM |
|
Down again |
| Posted by zymic, 07-23-2009, 08:04 PM |
|
I was just SLEEPING IN PEACE knowing our servers were back And now boom down. |
| Posted by l0ck_up, 07-23-2009, 08:05 PM |
Down again from here, I think I will move.
I've got 3 down times this month (2 bad hardware & now this) |
| Posted by admin-caperhosting, 07-23-2009, 08:10 PM |
|
wow that's unfortuanante ill tell you tho fdc is great, unfortuanate for this one issue now , but i have faith they came a long way |
| Posted by PersonalJ, 07-23-2009, 08:12 PM |
|
fdcservers.net is back up now. |
| Posted by brc_csf, 07-23-2009, 08:12 PM |
Quote:
|
Originally Posted by l0ck_up
Down again from here, I think I will move.
I've got 3 down times this month (2 bad hardware & now this)
|
How long did it take to have your hardware replaced? |
| Posted by InnovusHost, 07-23-2009, 08:14 PM |
|
It's back again. Hopefully it stays up. |
| Posted by infiniteecho, 07-23-2009, 08:15 PM |
|
I am still down on some machines ... |
| Posted by InnovusHost, 07-23-2009, 08:16 PM |
|
My colo server is on the Internap network. Might be separate issues? |
| Posted by brc_csf, 07-23-2009, 08:22 PM |
|
We're still down |
| Posted by l0ck_up, 07-23-2009, 08:23 PM |
Quote:
|
Originally Posted by brc_csf
How long did it take to have your hardware replaced?
|
I don't know for sure, I know that I got bad motherboard after they replaced it.
Still down for me, it's almost working hour here & my clients surely will yell at me |
| Posted by brc_csf, 07-23-2009, 08:24 PM |
|
Now we are back |
| Posted by JoCortland, 07-23-2009, 08:27 PM |
|
Still having intermittent problems here, but mostly up. |
| Posted by wormy, 07-23-2009, 08:27 PM |
Internap network never went down today?
Still down on my BGP colos. Cannot even ping. |
| Posted by fog, 07-23-2009, 08:28 PM |
|
Seems like they've been up and down quite a few times in the past 20 minutes. Really annoying over ssh. |
| Posted by Eric Davis, 07-23-2009, 08:28 PM |
|
Some of my IPs respond to ping and some others don't. |
| Posted by brc_csf, 07-23-2009, 08:28 PM |
|
Down again |
| Posted by whfsdude, 07-23-2009, 08:29 PM |
|
A few of my colo's are unreachable from outside the network but reachable from inside. |
| Posted by keeleysam, 07-23-2009, 08:29 PM |
|
It's up and down and up and down for me. |
| Posted by PersonalJ, 07-23-2009, 08:31 PM |
Quote:
|
Originally Posted by whfsdude
A few of my colo's are unreachable from outside the network but reachable from inside.
|
I can confirm this as well, I used a proxy that is hosted on the FDC network to connect to my server. |
| Posted by InnovusHost, 07-23-2009, 08:31 PM |
|
Looks like I spoke too soon, back down. |
| Posted by fog, 07-23-2009, 08:32 PM |
Quote:
|
Originally Posted by wormy
Internap network never went down today?
|
Their Internap segment definitely went down. I'm on the 74.63.66.0 block and have been up and down a lot in the past few hours. Seems to be down at the moment again. I thought they were two separate networks, so it'll be interesting to hear what happened. |
| Posted by l0ck_up, 07-23-2009, 08:34 PM |
Up & down from here, mostly down.
I spent too many sms alerts for this..lol |
| Posted by whfsdude, 07-23-2009, 08:34 PM |
|
On a side note, looks like IPv6 is up while IPv4 is down. At least for my servers and for fdcservers.net |
| Posted by admin-caperhosting, 07-23-2009, 08:35 PM |
|
This is getting somewhat redicules, wtf dude up and down like a nvm i wont go there haha, FDC any update of the issue and or an ETA of this goddang up and down **** |
| Posted by brc_csf, 07-23-2009, 08:37 PM |
|
Does anyone know what is going? Network attack? |
| Posted by adrade, 07-23-2009, 08:40 PM |
|
Pretty solidly down here... 66.90.73.* - keeping my fingers crossed - hoping for an update from FDC |
| Posted by zurca, 07-23-2009, 08:44 PM |
Well, it seems like we are back to the same issue as last year's 12 hour outage, and pretty much arround the same time, I just hope this is not an every year thing and hoping is not the world rotation in relation with the sun, or glogal warming. lol.
Anyways any news from fdc yet?? |
| Posted by whfsdude, 07-23-2009, 08:47 PM |
Petr is creating a thread on their forums right now. (Outages - Speed Problems - Routing Issues)
Things are still fubar for me over v4.
PING fdcservers.net (66.90.66.135): 56 data bytes
36 bytes from te-0-10-0-5-pe01.350ecermak.il.ibone.comcast.net (75.149.228.249): Time to live exceeded
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 00 5400 c1db 0 0000 01 01 7c88 192.168.124.103 66.90.66.135
Request timeout for icmp_seq 0
ping6 -c 2 fdcservers.net
PING6(56=40+8+8 bytes) 2001:4830:1600:1ab::2 --> 2001:49f0:1000::3
16 bytes from 2001:49f0:1000::3, icmp_seq=0 hlim=55 time=91.198 ms
16 bytes from 2001:49f0:1000::3, icmp_seq=1 hlim=55 time=109.439 ms |
| Posted by brc_csf, 07-23-2009, 08:48 PM |
Quote:
|
Originally Posted by zurca
Well, it seems like we are back to the same issue as last year's 12 hour outage, and pretty much arround the same time, I just hope this is not an every year thing and hoping is not the world rotation in relation with the sun, or glogal warming. lol.
Anyways any news from fdc yet??
|
When it happened last year, what was going on? |
| Posted by PersonalJ, 07-23-2009, 08:48 PM |
|
I'm up, thanks petr |
| Posted by sticky, 07-23-2009, 08:50 PM |
yeah, IPv6 is working just fine for me.
And, outbound IPv4 seems to be working to most places (SSH via IPv6 and Proxifier over SSH Tunnel elsewhere) |
| Posted by zurca, 07-23-2009, 08:55 PM |
Quote:
|
Originally Posted by brc_csf
When it happened last year, what was going on?
|
I think it was the main datacenter router, or one of their bandwidth providers. I dont remember exactly, but I think it was that, they had to replace it, but took like 12 hours or more the whole journey outage.
cross your fingers this is not the same issue.
but after they fixed it , pretty much strong after that, very little outages over the year. |
| Posted by alexdale, 07-23-2009, 09:05 PM |
@ whfsdude
i see. thanks for the heads up! |
| Posted by wormy, 07-23-2009, 09:08 PM |
Quote:
|
Originally Posted by fog
Their Internap segment definitely went down. I'm on the 74.63.66.0 block and have been up and down a lot in the past few hours. Seems to be down at the moment again. I thought they were two separate networks, so it'll be interesting to hear what happened.
|
Very interesting. Must have to do with an internal dc problem then perhaps. |
| Posted by admin-caperhosting, 07-23-2009, 09:26 PM |
|
Thanks for the link to the issue from fdc thanks! |
| Posted by Katatonic, 07-24-2009, 12:14 AM |
It was one of their main core routers going down, then when they tried to bring it back up didn't realize it was faulty until attempting to recover from a default configuration.
It's understandable and it happens. Perhaps they'll have a failer over system (or something ready to be replaced live) in the future.
Other than that, FDC are extremely reliable and support is second to none. I have high praise for them. |
| Posted by darkjedi, 07-24-2009, 01:20 AM |
announcement has been posted at http://fdcservers.net/vbulletin/showthread.php?t=4020
Our Chicago facility has experienced network issues this evening due to core network hardware failure.
The issue has been resolved and the network should be back to normal now.
The root cause of the problem was a failed 4x10GE blade on one of our core routers. When the traffic failed over to the other core router it has crashed the router thus causing complete network outage.
We have attempted to recover the core setup under existing configuration but once it became apparent that the blade was defective we had to move some uplinks and take the blade offline.
We apologize about any inconvenience this issue has caused
thank you for your patience and understanding
FDCServers |
| Posted by Katatonic, 07-24-2009, 01:28 AM |
|
Yes, it's great that they're back up online, but what steps are they taking to ensure this doesn't happen again, if they haven't taken them already? |
| Posted by brc_csf, 07-24-2009, 08:52 AM |
Downtimes happen everywhere.
Besides this problem, as long as they keep their fast support I am really happy with the service. |
| Posted by Katatonic, 07-25-2009, 09:12 AM |
They still have a few network issues that should be resolved within 24 hours.
I guess some techs aren't sleeping tonight. |
| Posted by quad3datwork, 07-25-2009, 02:36 PM |
|
I think it was handled well in all respect. Enterprise level equipments do fail too. Good job FDC! |
| Posted by Katatonic, 07-29-2009, 01:21 AM |
My servers at Chicago are back down.
Can any one confirm? |
| Posted by wormy, 07-29-2009, 01:35 AM |
|
Both of mine in chicago are up. |
| Posted by Katatonic, 07-29-2009, 02:00 AM |
It seems to be only our servers within an IP range - so not network wide as my other servers in Chicago on a different range are up.
Thanks for confirming. |
| Posted by UNIXy, 07-29-2009, 02:04 AM |
Quote:
|
Originally Posted by Katatonic
My servers at Chicago are back down.
Can any one confirm?
|
A lot of IPs including GWs in your block are responding to pings. So it might be isolated.
Regards |
Add to Favourites 
Print this Article
Also Read
EZPZ US2 (Views: 1189)
Shared Hosting
Shared Hosting
