WebNX down?

Portal Home > Knowledgebase > Articles Database > WebNX down?

Posted by kafloofen, 09-14-2008, 10:25 PM
My sites are down and so is WebNX.com
Posted by Steven, 09-14-2008, 10:26 PM
I was informed they are aware of it, and are working on resolving it.
Posted by swijaya0101, 09-14-2008, 10:28 PM
yes. all servers are down. cogent, mzima and premium network.
Posted by CLKeenan, 09-14-2008, 10:29 PM
Damn it.... such horrible timing. I just launched a new version of my website (new CMS, backend, etc etc) and we had been having some stability issues with the new version the past few days. Now we think we got everything up and running smoothly and then WebNX goes down for like the first time in 2 years? What horrible timing!
Posted by Aussie Bob, 09-14-2008, 10:30 PM
They'll resolve this pretty soon folks. Hang tight.
Posted by Peter-SexyWing, 09-14-2008, 10:33 PM
down here too
Posted by CLKeenan, 09-14-2008, 10:35 PM
Do we know what caused the problem?
Posted by 123finder.com, 09-14-2008, 10:35 PM
2 servers down for me too, any more details on this?
Posted by Asheron, 09-14-2008, 10:36 PM
Its been over 15min...
Posted by Steven, 09-14-2008, 10:37 PM
Dan is working feverishly to get it back online. I just spoke with him a few moments ago. Don't panic
Posted by SoFiMaN, 09-14-2008, 10:37 PM
Yea, all servers are down now and webnx.com also but I'am sure they're working on now.
Posted by kafloofen, 09-14-2008, 10:39 PM
I know they're working on it but do you mind telling us the reason for this outage? I'm sure all of us WebNX customers would like to know.
Posted by skyman, 09-14-2008, 10:46 PM
Network issue. My server has been down approx 25 mins. so far. webnx.com is down also.
Posted by Mikie4648, 09-14-2008, 10:48 PM
YE! WTF is going on? All my boxes, all 3 of them are completely down. Even http://webnx.com is down not to mention the premium bandwidth network.
Posted by cyberhouse, 09-14-2008, 10:48 PM
im sure they know about it and are working on a solution.
Posted by HardLayers, 09-14-2008, 10:49 PM
Me too, hope to hear from them soon are there any weather issues down there in LA , i'm not from the U.S ,thats why i'm asking
Posted by skyman, 09-14-2008, 10:50 PM
they told me it was a network issue and they are working on it
Posted by nonparity, 09-14-2008, 10:50 PM
once they are back up I am sure we can find the reason for the outage they may be changing the network some normally i go through mzima but right now it is going through Global Crossing and dieing..last I knew Global Crossing is not in the premium mix so not sure if it is dieing at linkline or what.
Posted by nonparity, 09-14-2008, 10:52 PM
no weather issues..i suspect they may be changing the network
Posted by Mikie4648, 09-14-2008, 10:55 PM
Changing the network to what? Dont they inform customers before taking down the network for a changeover?
Posted by kafloofen, 09-14-2008, 10:56 PM
there's already a topic on this in the outage section Last edited by kafloofen; 09-14-2008 at 11:01 PM.
Posted by layer0, 09-14-2008, 10:56 PM
Are you even affected by this outage? If not, I suggest you stay out of this thread. I doubt they are making any network changes - this appears to be an issue with Linkline.
Posted by layer0, 09-14-2008, 10:58 PM
As I said in another thread - I doubt they're making network changes. One of their upstreams - Linkline, appears to be down.
Posted by Mikie4648, 09-14-2008, 10:59 PM
If i wasnt affected do you think i would have posted the second message it this thread?
Posted by layer0, 09-14-2008, 10:59 PM
I'm sorry, but that hostility really isn't necessary. Edit: Funny, I see you edited it. Nice.
Posted by nonparity, 09-14-2008, 11:00 PM
if it was just linkline down you would think mzima and cogent customers should still be fine or premium as i am on that network which uses both linkline and mzima something else seems to be going on
Posted by Aussie Bob, 09-14-2008, 11:01 PM
Easy chaps. I'm sure they're working as hard as possible to get services restored. These things happen from time to time, and noone likes downtime, but we ride through it.
Posted by Mikie4648, 09-14-2008, 11:01 PM
No its not, you should learn to read before telling me to stay out of the thread.
Posted by layer0, 09-14-2008, 11:01 PM
That doesn't change the fact that Linkline IS in fact offline - this must be part of the issue.
Posted by Asheron, 09-14-2008, 11:02 PM
omg... how much time should we be waiting?
Posted by skyman, 09-14-2008, 11:02 PM
where is the outage section?
Posted by nonparity, 09-14-2008, 11:02 PM
i would think they would but normally i travel over mzima and now it is going to global crossing then dies thats why i was wondering if they might be adding something although linkline does use global so it could just be going to linkline eventually now
Posted by adamnp, 09-14-2008, 11:03 PM
Hopefully momentarily
Posted by Peter-SexyWing, 09-14-2008, 11:03 PM
em...gettting worried
Posted by Aussie Bob, 09-14-2008, 11:03 PM
http://www.webhostingtalk.com/forumdisplay.php?f=59
Posted by joelin, 09-14-2008, 11:04 PM
yes,it is down now,i hope it be back asap.
Posted by Jay H, 09-14-2008, 11:04 PM
No need to worry yet. Network issues are a part of web hosting, in my experience it is unavoidable and a part of relying on technology.
Posted by nonparity, 09-14-2008, 11:05 PM
it shouldn't be but you may be right and it is
Posted by 2008web, 09-14-2008, 11:05 PM
all my 5 servers, premium and non-premium bandwidth are down, I could not contact them so I do not know what happen
Posted by layer0, 09-14-2008, 11:05 PM
Looks like WebNX is back. Linkline is also back.
Posted by unable2stay, 09-14-2008, 11:05 PM
Obviously, exactly the length of time until it comes back online. This isn't a scheduled outage. Patience in a virtue.
Posted by Aussie Bob, 09-14-2008, 11:06 PM
Patience people. Let's not have everyone breaking into panic here. These things happen in this industry. I have every confidence Dan and his team are sweating bricks on this issue and this will be resolved asap.
Posted by Asheron, 09-14-2008, 11:06 PM
Not when we are loosing money... in fact, did any one read their Service Agreement regarding outages?
Posted by adamnp, 09-14-2008, 11:07 PM
Up :} definately! cheers dan!
Posted by Steven, 09-14-2008, 11:07 PM
Webnx.com and my own dedicated are back up.
Posted by kafloofen, 09-14-2008, 11:07 PM
hooray! it's back up.
Posted by nonparity, 09-14-2008, 11:07 PM
yeah you are right but Peter offers Adult Hosting. I am sure he has down threads, tons of support tickets(if his support is not down) phone calls etc. Adult Hosting customers tend to be very needy. Anytime they are down for 30 seconds they claim to loose thousands.
Posted by Mikie4648, 09-14-2008, 11:09 PM
Only webnx.com is up. We are not.
Posted by aucsynk, 09-14-2008, 11:10 PM
Hmm back up for a bit... then down just again as quickly
Posted by nonparity, 09-14-2008, 11:10 PM
now lets hope everything else gets back online
Posted by 123finder.com, 09-14-2008, 11:13 PM
206.251.x.x up but 67.220.x.x still down
Posted by HardLayers, 09-14-2008, 11:13 PM
one of mines is back too lemme check the others
Posted by nonparity, 09-14-2008, 11:13 PM
i'm not up either on premium network but since webnx.com is up hopefully we all will be soon now
Posted by Asheron, 09-14-2008, 11:15 PM
My server is not back...
Posted by kafloofen, 09-14-2008, 11:15 PM
I'm on 67.220.xx.xx and It's up on my end and on pingdom's end Last edited by kafloofen; 09-14-2008 at 11:20 PM.
Posted by Peter-SexyWing, 09-14-2008, 11:15 PM
down again
Posted by adamnp, 09-14-2008, 11:15 PM
We are up over here :/ 67.201.x.x
Posted by Russ Foster, 09-14-2008, 11:18 PM
Looks like a power issue as 3/6 servers back all with uptimes of minutes
Posted by Asheron, 09-14-2008, 11:19 PM
1 hour downtime... not funny at all...
Posted by CLKeenan, 09-14-2008, 11:19 PM
Back up: 38.99.X.X
Posted by kafloofen, 09-14-2008, 11:20 PM
Mine's uptime is now 14 minutes. There was a reboot.
Posted by nonparity, 09-14-2008, 11:21 PM
that explains why everything went down
Posted by Mikie4648, 09-14-2008, 11:22 PM
67.220 38.119 and 99 still down. Power issues?
Posted by covebt, 09-14-2008, 11:22 PM
Our one and only server with them went down also. It was rebooted twice (10 minutes apart). I think you will find that a lot of (linux) servers will currently be fsck'ing themselves as it wasn't a clean shutdown on both occasions.
Posted by Mikie4648, 09-14-2008, 11:25 PM
Ye well what happens if 2 of your boxes had an uptime of 400 days + and they never rebooted? Only 1 box is online the other two had 400 + uptime have not come back online.
Posted by Steven, 09-14-2008, 11:27 PM
I had a development server with over 650 days reboot, it came back up. Yours may be fscking. Shoot them a ticket.
Posted by Russ Foster, 09-14-2008, 11:27 PM
Got an update from Dan "It looks like for some odd reason an A/C unit took out our main breaker and our diesel generator box breaker. yet we still had 300 amps of 3phase left. If you server isn't up in 10 minutes put in a NORMAL ticket not a reboot ticket"
Posted by nomx, 09-14-2008, 11:27 PM
Our 67.201.x.x server is still down, and their client portal isn't loading for me or a few others (from different ISPs and regions). Edit: Replies coming in so fast here. We just have a bunch of customers throwing us tickets and giving us headaches
Posted by hanylord, 09-14-2008, 11:29 PM
webnex is a good DC and will back up in shortly time
Posted by covebt, 09-14-2008, 11:30 PM
Good to see Dan is on top of things (as always). This is the first outage I've had with them in over 2 years.
Posted by Mikie4648, 09-14-2008, 11:30 PM
I cant even log into my account at your site. CLIENT PORTAL returns a page not found.
Posted by nonparity, 09-14-2008, 11:33 PM
yeah cant put in a ticket if the ticketing system is down
Posted by Mikie4648, 09-14-2008, 11:35 PM
anyone talking to them of the phone can mention that we cannot log into the ticket system.
Posted by Russ Foster, 09-14-2008, 11:35 PM
Dans just working on Ubersmith to get it back up
Posted by Mikie4648, 09-14-2008, 11:43 PM
We are still down all because nobody can submit a ticket because they can t get it UP.
Posted by ovais, 09-14-2008, 11:46 PM
Our 5-6 servers are up from last 30 minutes but rest 8-10 servers are still non accessible.
Posted by HardLayers, 09-14-2008, 11:47 PM
1=up 2=still down
Posted by joelin, 09-14-2008, 11:49 PM
2=still down
Posted by Mikie4648, 09-14-2008, 11:50 PM
Probably needs a reboot. But you cant login to submit a ticket. Try the OFFLINE chat system. At least you can send a msg.
Posted by Asheron, 09-14-2008, 11:50 PM
This is way too annoying, 1.30 hours... Sunday is the most important day for my company... it wouldn't have been a big problem for me if it was any other day... (except saturday, sunday and monday).. Did any one know what should WebNX do if this happends according to their SLA ??
Posted by skyman, 09-14-2008, 11:57 PM
So do I understand that if we are still down that our servers aren't going to come back up themselves and webnx will need to reboot them, but we need to contact them?
Posted by ovais, 09-14-2008, 11:58 PM
well, their customer portal is up now. Here in India its Monday morning which is much important than other weekdays..
Posted by Mekhu, 09-14-2008, 11:59 PM
Something tells me this goes beyond the simple need for reboots. Only time will tell. And yeah, Sundays suck for these things but WebNX has been one of our leading providers and for good reason.
Posted by ovais, 09-15-2008, 12:02 AM
I asked them to check servers under 67.220.199.x range, within 5 minutes all of them are up now..
Posted by 123finder.com, 09-15-2008, 12:08 AM
Our is in 67.220.198.x and still down for an hour or longer. Sent email, sent offline message via chat.
Posted by ovais, 09-15-2008, 12:09 AM
Try to submit ticket through customer portal.
Posted by nonparity, 09-15-2008, 12:11 AM
I already have I am sure they have a lot of tickets to respond to though
Posted by Mikie4648, 09-15-2008, 12:12 AM
Does webnx have generators? I mean come on. If the AC killed the power and they have generators surely those generators could have kept all the server online. When the power blew at TP a few years ago there were excuses as to why the Generators never kicked in as well. Seems like these switchovers need some tweaking.
Posted by swijaya0101, 09-15-2008, 12:14 AM
1 server at cogent network is still down 38.99.248.x
Posted by Mekhu, 09-15-2008, 12:18 AM
We're still offline as well. I'm sure it's just ticket system overload for them. I know we got atleast 200 open atm
Posted by skyman, 09-15-2008, 12:25 AM
I told them to check the range of my servers over 10 minutes ago. Still down :-(
Posted by HardLayers, 09-15-2008, 12:30 AM
all of the 3 = UP
Posted by Mekhu, 09-15-2008, 12:32 AM
33 minute wait here... get in line
Posted by joelin, 09-15-2008, 12:32 AM
i try to open ticket and online live support, but no one replay me at all, it is down for hours.
Posted by Aussie Bob, 09-15-2008, 12:34 AM
Yeah, I'm sure they're working through the reboot requests as fast as they can. Still waiting for 2 servers to come back here.
Posted by Mikie4648, 09-15-2008, 12:47 AM
More than 2 hrs of downtime. I have one box left to go online which is Cogent and it seems that is being left till last.
Posted by WebNX, 09-15-2008, 12:49 AM
I apologize for the delayed response on wht, We had a main breaker trip, from there we switched to generator power, but that breaker also tripped. We had to wait until the building engineer to get onsite to restore power to our suite. When restoring power the main breaker tripped again, from there we tracked back the issue to a faulty A/C, this A/C was not tripping its own breaker, but the main suites power (480v 3 phase 800 amp breaker). All power is online now, and we are digging into the tickets one by one, if your server(s) are not online please send in a ticket and our techs will check it (it would also be a good idea to include the root password incase a fsck is needed) Thank you for your understanding, and we will have more updates as soon as things calm down a bit.
Posted by Mekhu, 09-15-2008, 12:52 AM
Thank you so much for the response.
Posted by -OY-, 09-15-2008, 12:58 AM
Bah, mine must be FSCK'ing. =/
Posted by nonparity, 09-15-2008, 01:02 AM
thanks for the update hopefully you will get to my ticket soon
Posted by skyman, 09-15-2008, 01:04 AM
Join the club :-(
Posted by nonparity, 09-15-2008, 01:05 AM
yeah mine seems to be stuck at that too i bet still down
Posted by Mekhu, 09-15-2008, 01:16 AM
Bob, you guys back online yet?
Posted by kafloofen, 09-15-2008, 01:23 AM
initial update from WebNX http://www.webhostingtalk.com/showpo...0&postcount=72
Posted by Mekhu, 09-15-2008, 01:24 AM
Weak, just got a canned response. I'll submit my information (that's in the ticket already) and see how long this takes...
Posted by Scott.Mc, 09-15-2008, 01:31 AM
So what. They are probably dealing with the largest number of tickets they've had to date, cut them some slack and be patient.
Posted by Mekhu, 09-15-2008, 01:37 AM
Sounds good. You know all.
Posted by Mikie4648, 09-15-2008, 01:38 AM
Tell my clients that, 3hrs and counting. Support ticket submitted over 1hr ago. Only one box out of two submitted was rebooted the second one left to sit and do nothing.
Posted by Asheron, 09-15-2008, 01:41 AM
3 hours 30min... and still not online.. sent tickets and chat... damn this sucks..
Posted by Aussie Bob, 09-15-2008, 01:46 AM
Most servers are but waiting on 2 to come back.
Posted by skyman, 09-15-2008, 01:48 AM
I can handle this as long as it comes up soon with no data corruption or problems. fingers crossed
Posted by Scott.Mc, 09-15-2008, 01:49 AM
There's nothing you can do other than be patient, there isn't any point stressing out over something that isn't in your control.
Posted by Aussie Bob, 09-15-2008, 01:52 AM
Yep, our main Dotable server is down, and it's not good, but no point stressing over it. These things happen.
Posted by Mekhu, 09-15-2008, 01:52 AM
That's not the issue with some of us. For some, there's an issue when you include information a, b, c & d and then you get a response after waiting 1 hour that requests information a, b, c & d. That's my gripe. Not a big one but whatever. Was just venting earlier.
Posted by Scott.Mc, 09-15-2008, 01:53 AM
Luckily we are down to just 1 to go.
Posted by Scott.Mc, 09-15-2008, 01:55 AM
Yeah but you can take that up with them afterwords. No point going on about it just now, especially people who submit so many tickets, add loads of useless information to their tickets. It does nothing but mean there is more to read and ultimately delay the process. When the dust settles you can take it up with them if you have a gripe on how this was handled. In the mean time, no need to stress yourself out for what is ultimately something that will be back soon enough and even if it isn't, so what? You have your backups (right???) so there's nothing to be worried about other than some further waiting.
Posted by Mekhu, 09-15-2008, 01:58 AM
*thumbs up!*
Posted by Asheron, 09-15-2008, 02:18 AM
4 hours and still counting...
Posted by Russ Foster, 09-15-2008, 02:27 AM
We've been on the recieving end of this sort of things (major power outage). I wouldn't be surprised to wait for 6 hours to get an initial response and upto 24 hours to get everything settled. Yeah it sucks but unless you are a colo customer and close there isn't much you can do (we are and aren't in that order)
Posted by Asheron, 09-15-2008, 02:58 AM
5 hours and counting I don't care if I have to wait 6 hours or 365 days to get that fixed. As a client, I'm offered 100% Power Uptime, according to their website.. I pay for that, I should get that. Or am I wrong?
Posted by nonparity, 09-15-2008, 03:03 AM
your not wrong asheron and i'm still down too
Posted by Russ Foster, 09-15-2008, 03:05 AM
Your wrong. You are paying for a PROMISE ONLY of 100% power uptime and if not you get a credit under the SLA
Posted by nonparity, 09-15-2008, 03:07 AM
a2b2 is correct as well things happen you go down but as long as you get a SLA credit all is good
Posted by nonparity, 09-15-2008, 03:10 AM
if i wasnt so tired and frustrated being down i would have commented that too i am normally more used to and relaxed with outages but this is taken so long..i know they have so much they are having to do getting everyone up though
Posted by Asheron, 09-15-2008, 03:14 AM
Well the best days of the week for my company is Saturday, Sunday and Monday... Specially Sunday night, where everyone here is on Internet... Thanks to this outage, I'm giving the wrong impression to my clients and future clients. I've been trying to see their SLA, but for some reason is in blank.. They deleted the content after the went back online? This is business, and this outage was not a good impression for my clients... specially when I have competence in my business..
Posted by nonparity, 09-15-2008, 03:22 AM
I am sure the SLA page was not removed intentionally I just reviewed the SLA and they do not even have to give a credit for the outage "We promise to keep the network up and get it back up ASAP should it fail. As of this writing we are at 99.99% uptime please feel free to verify that by going to: WebNX Status" Does not give SLA credit mention and is more of a promise to bring it up fast which I am sure they are doing as fast as passable. So it appears to me they are within the SLA
Posted by Aussie Bob, 09-15-2008, 03:28 AM
In my years as a host, I've been through downtime that would make your hair curl. You get through it. It's not the end of the world. Tomorrow's another day. Life goes on . . .
Posted by skyman, 09-15-2008, 03:34 AM
Amen. My first downtimes seemed like a matter of life and death. Now it still gives me a quesy feeling until all is right and back up, but I can keep it in perspective. If any customers give me a really hard time, I tell them I'm sure would be happier elsewhere. :-)
Posted by nonparity, 09-15-2008, 03:35 AM
i wish i could always take it easy..i've worked off and on in the hosting industry since 1998 so I know how had and cruel it can be on the hosting side but it sucks on both sides downtime is never fun in my case weekends are my less visited so it inst too horrible..
Posted by Mekhu, 09-15-2008, 03:49 AM
Downtime is a good time to build your business to client relationship if you're good with people and are very honest and open to questions from clients. In my 8-10 years online I have never filed for any type of downtime SLA refund... I find 99% of my clients are this way as well.
Posted by swijaya0101, 09-15-2008, 03:52 AM
Please look into ticket #4644321 Server is still down.
Posted by skyman, 09-15-2008, 03:53 AM
deleted message Last edited by skyman; 09-15-2008 at 04:07 AM.
Posted by hanylord, 09-15-2008, 03:58 AM
67.220. Dowan
Posted by Asheron, 09-15-2008, 04:05 AM
You guys are talking about hosting industry.. I do not sell host... it's another type of business where "downtime" is not acceptable. Not that big to compare, but lets see: if Facebook goes down for over 6 hours, good impression? Users may start looking for a new hobby. Google down for 6 hours, good impression? they will go to another search engine. In my business, 6 hours or more of downtime, my sponsors and clients, will just go with another competence company. Or the other companies can start saying "their website goes down, not a good one, come with us" bla bla bla. In hosting industry at least you can talk with your clients, but.. if you are new, less than a year.. you may loose some of them, specially potencial ones.. I don't, it's just my point of view. But for this big downtime, we all clients of WebNX should get something back... instead of a "promise". Because with a promise I can't go and tell my visitors, clients, sponsors to sit back and wait, but don't leave..
Posted by nonparity, 09-15-2008, 04:07 AM
i'm back up so i'm happy now webnx is still good folks..i still recommend em..i even did in another thread while my server was still down) i dont think the webnx crew is going to get much if any sleep tonight though
Posted by Scott.Mc, 09-15-2008, 04:12 AM
Do you pay for the same infrastructure that google and facebook do? If downtime is not acceptable (and it IS going to happen at some point) then you need to pay to ensure that you have backup alternatives ready and waiting.
Posted by nonparity, 09-15-2008, 04:12 AM
even if you could get an sla credit, you cant give that to your surfers anyway so it doesnt really matter to them SLA credits is more of a marketing tool than anything..they are just being honest but not mentioning them in the SLA. Most hosts never give them out as the ones who beg for them are typically problem customers and they just want to unload them. I am not saying you are a problem customer but from my own experience working for some hosts that is typically how it is. They all had SLA credits and almost never gave them out.
Posted by Asheron, 09-15-2008, 04:12 AM
4 hours and something, later... I got a reply for my ticket. But still down. (7 hours and counting) I know Dan is a good guy, etc etc... but "Please provide some usefull info, like IP address to start with? -- Dan P. WebNX" ???? I don't know if that was a tricky question or a mistake.. BUT, why do we all have a client "account" with all our information?? They should know that and don't ask for it.. that makes me think they don't care to solve the problem and just want the client to continue loosing their time while they asking questions they know the answer already
Posted by Aussie Bob, 09-15-2008, 04:13 AM
Then you're into the mission critical hosting category. There you're looking at the $xx,xxx,xx/mth price range for true mission critical hosting, with true redundancy built into every level. That's not going to be cheap. If you can get by on the 3 9s (99.5%) then webnx is for you, but if you need the 5 9s (99.999%) then you'd need to look for a mission critical hosting provider, for the above price point.
Posted by nonparity, 09-15-2008, 04:14 AM
also very good points amazon and google both have had outages recently
Posted by Asheron, 09-15-2008, 04:16 AM
I know I don't pay for the same infrastructure as them, that why I said "too big to compare with, but..."
Posted by skyman, 09-15-2008, 04:16 AM
Our server admin just got our server back up. Good luck everyone, I hope yours is back up soon. Good night.
Posted by nonparity, 09-15-2008, 04:17 AM
or multiple providers for truly mission critical applications i would spread it across three providers.
Posted by nonparity, 09-15-2008, 04:19 AM
an outage is never good but I think WebNX has handled this very well
Posted by nonparity, 09-15-2008, 04:21 AM
very good news is monday and everyone or just about everyone is online now very good
Posted by Asheron, 09-15-2008, 04:21 AM
Lol... I know that cost a lot... but I just hate to see this big outage and keep waiting for an "answer", and specially now, that I get and answer with a question they already had the answer. Anyway, they just got my server back online. What I was wondering is, if they had this kind of issue.. and they fixed the "power" issue hours ago... why so long to take my server back online? They should have like an "emergency" team, so all of them can get in there and bring servers back. Since I've been with WebNX the only ones who answers my tickets is: Dario and Dan. I know there is a lot more of people working... but it gives me the impression they are the ones who looks at the post and send it to the tech and the answer the post back... Should be a good idea if there was like 10 o 15 more like Dario and Dan in this kind of scenario.. things will be a lot faster.. IMO
Posted by nonparity, 09-15-2008, 04:25 AM
not all servers didn't come up but a lot didn't..they need root login to get it fixed, some servers may also have bad hard drives now due to this so they have to do some hardware replacement, etc
Posted by Asheron, 09-15-2008, 04:27 AM
That's why I think they should get a bigger "team" for emergency cases. Because if it's only Dan and Dario who replies tickets.. and they pass away the info to techs and then techs the answrs to them, and then they reply the ticket... too much time wasted. Unless they have a bigger team... IMO
Posted by Scott.Mc, 09-15-2008, 04:28 AM
Asheron, they are dealing with an emergency and going to have a sleepless night. Everyone wanting something, so the stress levels are going to be very high. It certainly doesn't help to continually complain. If you have an issue with how it's being handled then take that up with them LATER, it's certainly the wrong time now. I am sure they would have had your up quicker if you had provided what they asked for (again if you don't think you have to provide, then again take this up with them, LATER). They did restore power a long time ago but your system obviously needed something else as it didn't boot cleanly, be it a manual fsck or another issue. So for the time being, cut them some slack otherwise you will do nothing but make your relationship with them go south. They probably don't have alot, if any, replacements to do since it's mostly newer equipment. The delays will just be working on each server and all the different needs of the remaining ones. I do know from speaking to webNX that atleast 90% of things are working, it's the remaining ones needing manual fscks, etc that are being worked on.
Posted by Asheron, 09-15-2008, 04:46 AM
I'm not complaining now.. Just giving an opinion of what happened.. I will talk with them later, maybe in a couple of days, about this situation.. But that doesn't mean I can't give my opinion here about what happened. As I said, I'm not 'bitching' WebNX for all this... They tried their best to get this thing fixed ASAP.. But I just saw what was going on and said what I think about all this and gave out my opinion.
Posted by Aussie Bob, 09-15-2008, 05:55 AM
Yeah, power issues are the pits. Servers not shutdown cleanly, and don't come back without a fsck or worse. Bah, you get that.
Posted by Russ Foster, 09-15-2008, 06:33 AM
All up and running here now so overall 8 hours to restoration. Not bad going
Posted by SolarElement, 09-15-2008, 06:45 AM
8 hours down, still counting... no ticket reply.
Posted by sspt, 09-15-2008, 07:04 AM
I was affected by some connectivity issues caused by this power outage but they've done a great job recovering it and I'm sure they're doing their best to bring everything back online as soon as possible.
Posted by SolarElement, 09-15-2008, 07:56 AM
almost 10 hours now, still down. no ticket reply, no one even answering phone.
Posted by SolarElement, 09-15-2008, 08:32 AM
after 10 hours, finally up now.
Posted by 123finder.com, 09-15-2008, 09:32 AM
Our were up after we submitted a ticket with IP/login info and they checked it. Glad it's online before Monday in the US starts. I guess some servers crashed badly and not coming up on its own.
Posted by WebNX, 09-15-2008, 10:33 AM
As of now all servers should be online / all tickets answered, if you still have a server down please open a new ticket with the ip address and login details. A more detailed RFO and SLA info will be posted later today. Thanks Dan
Posted by davef139, 09-15-2008, 01:31 PM
Looks like I was effected with this, I partied way too hard last night to care since my boxes are mostly personal use lol, biggest disappointment will be loss of uptime
Posted by nonparity, 09-15-2008, 07:09 PM
all and all I think webnx handled this the best they could and did a great job getting everyone back online way to go webnx!
Posted by Dougy, 09-15-2008, 08:14 PM
If the RFO has been sent yet, can anyone paste it?
Posted by joelin, 09-17-2008, 09:20 PM
Hi, anyone know where to get the RFO and SLA info ? thanks
Posted by andren, 09-18-2008, 08:15 PM
Don't promise things you don't keep
Posted by WebNX, 09-19-2008, 07:55 AM
I would like to apologize for the delay in publishing our RFO. We wanted to wait until we had a few different companies examine the A/C unit and wiring. Sunday September 14th WebNX's 5,000 sqft suite in 530 West 6th St on the 7th floor lost full power due to a ground fault in one of the Libert 20ton CRAC's. All UPS's and the building generators worked as designed. Unfortunately the ground fault bypassed the fuses in the CRAC along with the units own breaker tripping the main suites 800amp breaker and then the generator breaker causing the UPS's to slowly lose power and then shutdown. Our core networking equipment stayed up as its fed by a DC plant with extensive battery backup. Power restoration was delayed as the main breaker is located in a locked room that only the building engineers have access to. Once they came onsite power was restored to the facility and the CRAC unit was disabled preventing further outages. Total time of power outage was just over one hour. Over 80% of the server came back online at this time and from this point any server that did not recover automatically required manual intervention. At this point techs were checking servers, running fsck's, reboots, hardware replacements or other repairs. We had two different HVAC companies look over the setup and both agree there must be a ground fault somewhere in the unit. This is only way our main 800amp 3 phase 480v breaker could have been tripped without blowing the 30amp and 90amp fuses along the way. The unit will be given a full overhaul before being brought back online. The SLA page on our website will be updated within 48 hours so clients will be able to update their tickets to get credit at that time.