steadfastnetworks network down? [Merged]

Portal Home > Knowledgebase > Articles Database > steadfastnetworks network down? [Merged]

Posted by David, 03-09-2006, 03:10 PM
Appears to have been out for about 15 minutes now. Any details?
Posted by whmcsguru, 03-09-2006, 03:11 PM
Yeah, here as well. Not sure what's going on, but not a good thing.
Posted by David, 03-09-2006, 03:14 PM
I just can't even begin to understand how unacceptable this is on so many levels.
Posted by renegadeavenger, 03-09-2006, 03:18 PM
their network has been stable without downtime for a long time, I think this short downtime is acceptable, we just have to be patient while they look into it.
Posted by David, 03-09-2006, 03:19 PM
Stable? We have entirely different definitions of uptime. Networks should NEVER go out. Once a month is unacceptable. Once a year I can deal with. But entire networks going out on a monthly basis?!
Posted by blueface, 03-09-2006, 03:20 PM
Could it be related with their recent migration from Equinix 5th floor to the 2nd floor? It's been 20 minutes now and they still haven't sent out any update.
Posted by renegadeavenger, 03-09-2006, 03:20 PM
The network hasn't went down for me on a monthly basis, I have no idea what you are talking about. This is the first outage in a very long time from my experience.
Posted by David, 03-09-2006, 03:21 PM
It's been about 25ish minutes now, just so you're aware (although they'll argue time with me when it comes to SLA but they obviously need new monitors). It's monthly, I can assure you of it. Last year was 'all right' but there were still way too many network issues. This year (who cares about the migration?!) has been entirely disgusting.
Posted by whmcsguru, 03-09-2006, 03:22 PM
Unacceptable how? I've been with 'em for 6 months now, haven't seen anything for downtime at all. I'm sure they're busting their tail ends over there trying to fix the issues at hand. If this was an hour+ downtime, then yeah, it might be time to call out the assassins and go after people , but 15-30 minutes, gotta be accetpable every once in a while, especially with the level of uptime and quality of service they provide.
Posted by renegadeavenger, 03-09-2006, 03:22 PM
Actually, they are aware of this issue, it has nothing to do with their monitoring systems.
Posted by David, 03-09-2006, 03:23 PM
I'm aware that they are aware of the issue. Obviously we're on "alternate network segments" if you haven't been incurring monthly downtime since November or earlier. The SLA shouldn't be there to give me free months every so often -- but I guess it seems it is.
Posted by renegadeavenger, 03-09-2006, 03:27 PM
you know... honestly if you look at their uptime compare to most host you can't complain about them. Their team is by far more dedicated than most of the other datacenters that I've been using and their network uptime has been excellent for me. You can't alway assume issues to be a simple fix with a push of a button.
Posted by surfmanjoe, 03-09-2006, 03:29 PM
I can not access my server at www.steadfastnetworks.com, as well as http://www.steadfastnetworks.com itself. any body may help me to check it? Thanks.
Posted by DeltaAnime, 03-09-2006, 03:32 PM
I'm unable to connect to their site, but my server with them is still alive and well ~Francisco
Posted by David, 03-09-2006, 03:32 PM
Check out the thread in 'provider outtages'. It's been out since 1:55. Note: This appears to only apply to the premium side of the network.
Posted by David, 03-09-2006, 03:35 PM
I'm not out to start a flamewar about how good or not good they are. They're down right now. Networks should have near 100% uptime and should rarely be an issue. I can see a switch going out at random and cutting off a few clients -- that's 'acceptable'. 100% outtage, ha. What a joke. Even the company I work for in Toronto (also known as a "scam") by most on WHT has had 100% network uptime for YEARS.
Posted by TonyB, 03-09-2006, 03:35 PM
Hopefully it's back up soon
Posted by DeltaAnime, 03-09-2006, 03:36 PM
Are you sure it's not just their site? I'm able to ping and ssh my box which is on the premium network ~Francisco
Posted by surfmanjoe, 03-09-2006, 03:40 PM
it has been down for more than 30 minutes for me. I am hoping it will be back soon. any one has idea for how long time the outtage will take? thanks.
Posted by KarlZimmer, 03-09-2006, 03:43 PM
This is the first outage we have had since the Equinix power outage in June outside of the 20 minute network maintenance window during the data center move. Also, not the entire network is down. I would say it is roughly 50% and we have been working on it since it started.
Posted by surfmanjoe, 03-09-2006, 03:44 PM
it's back for now. cheers!!!
Posted by blueface, 03-09-2006, 03:48 PM
It's still down for me.
Posted by Scott.Mc, 03-09-2006, 03:51 PM
David I also have not experienced what you are stating.... It's also a bit odd, you are always first to praise SteadFast and now.... -Scott
Posted by whmcsguru, 03-09-2006, 03:52 PM
No, you're just out to flame them, period. Stop Steadfast Networks hasn't had an outage like this in just about a year, and I'd say that's incredibly good for a budget server provider. Given the quality of service, knowledge of the techs, and support provided by these individuals (Karl included), I'd say it's a safe bet that whatever happened is being fixed and worked on as quickly as possible. While there are still a few sections down, they will be up when it's possible for them to be. Oh, you mean like, a year uptime, without any sort of major problem? Yeah, I believe that's what Karl just said. No, the joke is listening to the constant complaining of someone who obviously has no experience with this provider, or realistic experience in the business. A small DC like this is going to go down every once in a while (small being relevant here). They don't own their own DC, they lease space. They will be hit by issues left and right, and I'm sure ARE. The difference? THEY actually support their customers in a timely fashion, and THEY provide a reasonable uptime, unlike most others out there, even most of the big named companies. Stop complaining about Steadfast. Yes, it's down (still is, for some of us), but this is the first time in quite a while, and yes, it's GOING to happen to ANY network (no matter how much you lie and say it hasn't). This forum isn't for negative comments, or speculation as to why it's down, it's for what is down and what isn't. Take the other stuff to a more appropriate forum
Posted by KarlZimmer, 03-09-2006, 03:53 PM
I would like to note that this is in fact a hardware issue, a switch failure. All the customers who were on one of the network segments are experiencing an outage because of this. We are currently in the process of moving all affected customers onto other switches.
Posted by David, 03-09-2006, 03:54 PM
I'd just like to mention that pushing near a year and a half with steadfast gives me a bit of 'experience' with them. Any network downtime I've counted (each instance during February, January and December) was downtime that they acknowledged. It may not have been network-wide but there have been countless minor incidents. You may not consider 5 minutes here and there even as 'downtime' -- but I do. Last edited by David; 03-09-2006 at 03:57 PM.
Posted by LoginTech, 03-09-2006, 03:56 PM
Guys guys how about if you have a issue with them take it up with them i'm sure they will be able to work it out, lets not argue about uptimes here... They are experiencing issues, other users confirmed it, Karl says it's being resolved no reason to bark at each other.
Posted by WebAfrica, 03-09-2006, 04:03 PM
Just spoke to Karl, the network is down. EDIT: Actually its one of the blade servers providing connectivity to a large chunk of servers. They busy moving clients off the dead one.
Posted by WebAfrica, 03-09-2006, 04:08 PM
No complaints with steadfast. They been really stable the last year.
Posted by WebAfrica, 03-09-2006, 04:24 PM
Back up for me
Posted by KarlZimmer, 03-09-2006, 04:26 PM
I would like to note that we have not had a complete network outage, meaning anything on our network being completely unreachable from the outside world, outside of the single maintenance window, since June. I'm going through our entire list of network related tickets now to check on things and all I am seeing is, Jan. 31, short issue with AT&T. Other than that, the last issue we had, that I am noting, was with congestion/issues with AT&T/Level(3) back in November, for which we did issue SLA refunds even though there was no outage. Just making sure we're on the same page here.
Posted by David, 03-09-2006, 04:29 PM
Karl, That I will agree with (minus February's router issue -- but I can understand that during migrations things can go wrong). I'm not stating that everyone has been out in each of these instances but I most certainly have.
Posted by blueface, 03-09-2006, 04:31 PM
Yeah, it's finally back up now.
Posted by David, 03-09-2006, 04:43 PM
Now I can go back to being happy David since both boxes have returned. It appears there are quite a few colocation customers still out though. When I'm receiving the best of service: I'm the happiest little client in the world. Steadfast has -- the best possible customer service I've ever experienced. What they don't have is the network to back it up -- but their SLA is acceptable. When I'm not receiving the best of service.. well: I get pretty pissed off after 48 hours of no sleep and being woken up for network downtime.. but that's another thread alltogether!
Posted by tical, 03-09-2006, 04:53 PM
A PC router? Oh my!
Posted by Apolo, 03-09-2006, 05:00 PM
I've just moved a thread from Dedicated Server forum and merged it with this one.
Posted by KarlZimmer, 03-09-2006, 06:27 PM
It was a switch blade, not a server blade, he mis-spoke. We use Juniper routers, which had no issues. The issue was with customers on the faulty switch blade. We are currently working on procedures to reduce the chance of such an event occuring in the future and to reduce the effects of such an event. This is the first complete hardware failure we have faced. We have definitely learned a lot from it and will be changing our policies based on what we have learned.
Posted by Energizer Bunny, 03-09-2006, 06:35 PM
Yes david is right stead had almost everymonth a network outage its been happening since december i think.
Posted by Energizer Bunny, 03-09-2006, 07:36 PM
Sorry cannot edit but i just got alerts from monitoring services and definately stead was down for over 25 Mins which is kinda high for any network so Karl is it safe for us to remain with you guys or going to have more network issues in future and almost every month? Alert Time: [2006-03-09] 14:14:06 - Down Alert Alert Time: [2006-03-09] 14:39:25 - Up Alert
Posted by KarlZimmer, 03-09-2006, 07:49 PM
As I said before, we have not had a network outage since June when Equinix lost power. If you are experiencing more issues, please contact us at noc@steadfastnetworks.com if you notice any network related issues. This was caused by a hardware failure, so I do not see how it would indicate there would be any future network issues. With such a failure it takes time to get a replacement fully operational and to reprovision ports, etc. We are definitely working on procedures though to assure that customers will not be so greatly affected by another similar failure.
Posted by tical, 03-09-2006, 08:10 PM
That's refreshing; many hosts around these parts never learn anything from events like these. Some never learn anything at all. It's nice to see a company with an open commitment to excellence, not a commitment to backpeddaling and making excuses as so many others do. Thank you for your active efforts to raise the bar in this profession. My hat's off to Steadfast.
Posted by KarlZimmer, 03-09-2006, 08:26 PM
As an update, tomorrow we will be announcing on our site the procedure changes, hardware/network changes, etc. prompted by this incident for any of those interested. It will be in the support section, under announcements. Anyone who is on our announcements mailing list will also receive the information.
Posted by Energizer Bunny, 03-09-2006, 08:45 PM
Sorry to say karl but that is not entirely true, and i dont want to go into any debate cause you know it too yourself, but please be true to others it doesnt harm too much. In Feb. also there was 15 min network outage at around 2 or 3 am as was shown by hyperspin,uppanel and siteuptime.And i know when the migration took place and it had nothing to do with that migration.
Posted by KarlZimmer, 03-09-2006, 09:04 PM
The only outages that I know of that existed in February were on the 8th and 15th, both of which were announced scheduled maintenance windows which each lasted 15 minutes or less. They were both part of the migration to the new data center and the one on the 8th should have only causd outages with certain providers as we physically moved them to different interfaces. I don't see anything I said which is not entirely true. I know I'm not going to get anywhere by hiding the truth, which is why I don't, and never have.