Liquidweb Down [merged]

Portal Home > Knowledgebase > Articles Database > Liquidweb Down [merged]

Posted by neutro, 12-11-2005, 11:49 AM
I have problem reaching to LQ atm.. No ping or trace
Posted by devioustrap, 12-11-2005, 11:50 AM
Yup, same here. Servers are unreachable and their own site is down,
Posted by BlikWerk, 12-11-2005, 11:51 AM
Same here. Can't access PIMS and we have a server down at LiquidWeb.
Posted by argv1900, 12-11-2005, 11:52 AM
all my liquidweb's server down as well
Posted by andren, 12-11-2005, 11:53 AM
Seems like all of LW is down.
Posted by accyroy, 12-11-2005, 11:53 AM
Same with me, 2 servers both unreachable...
Posted by aldee, 12-11-2005, 11:54 AM
Agreed. Traceroutes die pretty early. So it is very likely that all of Liquidweb is currently dead.
Posted by Curious Too, 12-11-2005, 11:55 AM
Looks like a major network problem: C:\Documents and Settings>tracert liquidweb.com Tracing route to liquidweb.com [209.59.139.21] over a maximum of 30 hops: 1 30 ms 28 ms 30 ms h-69-3-206-129.sfldmidn.covad.net [69.3.206.129] 2 29 ms 28 ms 29 ms 192.168.7.161 3 29 ms 29 ms 31 ms h-66-134-103-170.sfldmidn.covad.net [66.134.103. 170] 4 * * * Request timed out. 5 * * * Request timed out. 6 * * * Request timed out. 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out. 10 * * * Request timed out. 11 * * * Request timed out. 12 * * * Request timed out. 13 * * * Request timed out. 14 * * * Request timed out. 15 * * * Request timed out. 16 * * * Request timed out. 17 * * * Request timed out. 18 * * * Request timed out. 19 * * * Request timed out. 20 * * * Request timed out. 21 * * *
Posted by ellinas, 12-11-2005, 11:56 AM
Yes... 3 servers with LW down.
Posted by Xia, 12-11-2005, 12:00 PM
Anyone else not able to access www.liquidweb.com? (As well as sites hosted there)
Posted by universal2001, 12-11-2005, 12:00 PM
Yep all our LW servers are down!! Their site is also down. Hope it gets resolved soon. I know they have techs/engineers probably running around like mad right now. Last edited by universal2001; 12-11-2005 at 12:04 PM.
Posted by jmansoor, 12-11-2005, 12:01 PM
all our servers went down on 6:56 PM GMT +3 try to access liquidweb site, and it is also down. tried to call them, but phone is busy.. any body else face this problem ?
Posted by accyroy, 12-11-2005, 12:01 PM
Strange, for the first time ever, nobody is picking up the phone there....?!
Posted by ekusteve, 12-11-2005, 12:02 PM
Yes, my server is down...first time in over six months....hopefully it's just a short burp.
Posted by eth00, 12-11-2005, 12:02 PM
The probably took them off the hooks from all the people calling in We have 7 boxes all down at LW as well...... looks like a fun day ahead I hope they get this fixed fast
Posted by masssa, 12-11-2005, 12:02 PM
Does anyone know what is the problem?
Posted by Xia, 12-11-2005, 12:02 PM
Yes I made a topic about it the same time as you did... I hope they work on sundays
Posted by ellinas, 12-11-2005, 12:02 PM
Does anyone talked with them ?
Posted by mycroftx, 12-11-2005, 12:02 PM
It dosen't work for me..Connection timed out.
Posted by layer0, 12-11-2005, 12:03 PM
Yup - down here too. Looks like it's probably something wrong with a core network device--it's not even getting out of my local network =\.
Posted by Xia, 12-11-2005, 12:03 PM
First time in 6 months too but I still don't like it
Posted by aldee, 12-11-2005, 12:04 PM
Yes. The problem is that you're posting in the wrong forum section: http://www.webhostingtalk.com/showthread.php?t=468541 Edit: Thread was merged. Last edited by aldee; 12-11-2005 at 12:09 PM.
Posted by despy, 12-11-2005, 12:04 PM
Yep, main site and my sites have been down since 10:45 ET.
Posted by whmcsguru, 12-11-2005, 12:05 PM
That would cover that right there. Maybe next time try reading the thread? If they don't pick up the phone, then how can people talk to them? 2 monitored servers down here too, so whatever it is, it's a pretty big thing.
Posted by Dacsoft, 12-11-2005, 12:05 PM
Same here - all servers down. Not answering the phone is very unusual. That could be explained by the fact they are working the problem and don't want to waste time telling all of us the same thing. I have always had fantastic support from Liquid Web, and I expect the same now. While any outage hurts, this is the first time in 3 years I have had a problem like this. The trick is how fast it can be restored. Wonder what the weather is like there right now?
Posted by Xia, 12-11-2005, 12:05 PM
Why the hell are the PIMS boxes on the same servers anyway? They should be somewhere else completely..
Posted by gennyblake, 12-11-2005, 12:05 PM
It's not even ringing - just a busy signal when I tried. I was going to see if they at least had a recorded message about an outage.
Posted by aldee, 12-11-2005, 12:05 PM
There's a already a thread in the correct sub section of this forum: http://www.webhostingtalk.com/showthread.php?t=468541 Edit: Thread was merged. Last edited by aldee; 12-11-2005 at 12:09 PM.
Posted by jmansoor, 12-11-2005, 12:07 PM
any contacts with liquidweb staff for this down ? i am continously calling them but no answer. This is unprofessional. if they are facing any problem they should attend the phone and let the customer know what is going on..
Posted by nucleo, 12-11-2005, 12:07 PM
I got the same issue... They usually pretty fast on fixing issue... hopefully it is something they can easily fix.. Imagine if it was becasue of terrorism
Posted by Xia, 12-11-2005, 12:07 PM
well why didn't the search function show this then? I searched first..
Posted by aldee, 12-11-2005, 12:07 PM
Wrong sub forum. => http://www.webhostingtalk.com/showthread.php?t=468541 Edit: Thread was merged.
Posted by daraghmacl, 12-11-2005, 12:08 PM
1 server down My own was playing up in the past few days but when i saw their own site is down i knew its trouble. Can someone let the board know if they get feedback - they should at least change their answering machine to tell us a rough idea of the prob!
Posted by Knuttz, 12-11-2005, 12:09 PM
Samething with me, they have a great service and very good costumer support. What makes me yet more worried as people is not being able to talk to them...
Posted by rwoliver2, 12-11-2005, 12:10 PM
I guess I could say "my stuff's down too with lw" but I don't think it would help at this point 800 is busy as well
Posted by MeddlePAL, 12-11-2005, 12:11 PM
They probably disconnected the phones. To many calls - wasting time having someone reply. We all know its dead in the whole anyways, when it comes back it comes back - Invoke your SLA when it comes back though. Must be something big, I've never had a hiccup with LiquidWeb's network. But I do agree why the F*** is PIMS on the same network.
Posted by litobelo, 12-11-2005, 12:12 PM
Entire LiquidWEB is down. Even no phone calls answer. Anybody knows much about this ?
Posted by JNL Host, 12-11-2005, 12:13 PM
I've had intermittent connectivity over the last few minutes. But mostly down. Jason.
Posted by MeddlePAL, 12-11-2005, 12:14 PM
http://www.webhostingtalk.com/showthread.php?t=468541
Posted by Duport, 12-11-2005, 12:14 PM
It has been around 20 minutes now. You might want to check out this topic http://www.webhostingtalk.com/showthread.php?t=468541
Posted by daraghmacl, 12-11-2005, 12:14 PM
might mean its coming back gently? Intermittent nature may be due to high traffic
Posted by blogmaster2003, 12-11-2005, 12:14 PM
I confirm that, seems all liquidweb dc is offline, but i don´t know why.
Posted by universal2001, 12-11-2005, 12:15 PM
I don't understand how LW can offer 100% network uptime when they have been down about 3-4 times in the past 2 years for longer than 2-3 hours. Do they do this as a marketing ploy because I visited their "reburstment" SLA page and the credit seems close to worthless if you do have a server that is down for a long time.....
Posted by Michael Kjærsgård, 12-11-2005, 12:15 PM
I just don't understand why traceroutes stop this early.. Anyone able to explain this?
Posted by jmansoor, 12-11-2005, 12:16 PM
Is there any body who can see and visit their office in lansing to know what is going on ?
Posted by neutro, 12-11-2005, 12:17 PM
connection comes thru ALTERnet and core router now
Posted by Knuttz, 12-11-2005, 12:18 PM
They are back Just accessed one of my boxes
Posted by despy, 12-11-2005, 12:18 PM
I'd noticed that too. I'm in the UK and my traceroute stops at Level3's network in London. Surely it should be stopping closer to LW's network?
Posted by eth00, 12-11-2005, 12:18 PM
That would be pointless, they are going to be busy and most likely the issue will already be fixed by the time soembody looks.
Posted by BlikWerk, 12-11-2005, 12:18 PM
Our server just came back online.
Posted by arwa, 12-11-2005, 12:18 PM
its up now
Posted by JNL Host, 12-11-2005, 12:18 PM
Okay.. We're back up now Still trying to get answers. Jason.
Posted by Curious Too, 12-11-2005, 12:18 PM
My servers are back online.
Posted by cheCCa, 12-11-2005, 12:19 PM
oh gosh....a couple of our servers are down too. how long has Liquid down already?
Posted by accyroy, 12-11-2005, 12:20 PM
OK, just spoke to an engineer. They have had problems with power but the generators are now up and things are coming back online.. My servers just popped back into life...
Posted by eth00, 12-11-2005, 12:20 PM
Yeah all starting to come online here too
Posted by ekusteve, 12-11-2005, 12:20 PM
Maybe I'm not as demanding as most, but down 3-4 times in the past 2 years seems to be a pretty good record...of course, it's not 100%, but 100% uptime is myth anyway. My server is back up now...30 minutes downtime in six months...I can live with that.
Posted by daraghmacl, 12-11-2005, 12:20 PM
Called but didnt press them - in situations like this you just wanna confirm everything is ok for now and let them back at their work back online!
Posted by Servax, 12-11-2005, 12:21 PM
Well, if I remember correctly LiquidWeb uses Verio, which is one of the carriers that seem to be having major latency issues right now.. http://scoreboard.keynote.com/scoreb...assword=public
Posted by PTNHosting, 12-11-2005, 12:21 PM
My server is back, but it was rebooted, so it my be a power problem?
Posted by JNL Host, 12-11-2005, 12:21 PM
Interesting.. Our servers have been rebooted. Indicates maybe a massive power failure and no generator kick in? PIMS is still down. Jason.
Posted by dqh, 12-11-2005, 12:21 PM
network is working .....
Posted by LuisMiranda, 12-11-2005, 12:21 PM
I have a couple of domains hosted with them... their main page is up now, but the server where my domains are hosted is still down, I hope it come up soon
Posted by 1ONE, 12-11-2005, 12:22 PM
My 2 servers are still down, I can't access any of my server... My phone is ringing clients are calling to check what is wrong... argh !
Posted by oshawa, 12-11-2005, 12:22 PM
All quiet on the liquid front .... my machines at that dc are reachable again (although I still can't reach pims or liquidweb.com)
Posted by neutro, 12-11-2005, 12:23 PM
my server might be fscking now..
Posted by MeddlePAL, 12-11-2005, 12:23 PM
My box is back... for the time being.
Posted by JenniH, 12-11-2005, 12:24 PM
We are still down here... did the engineer give any sort of ETA ot have all the servers up and running again?
Posted by 1ONE, 12-11-2005, 12:24 PM
OK, now my servers are up but very very slow... I think someone attack them ?
Posted by Dacsoft, 12-11-2005, 12:24 PM
Guess I have been lucky. In almost 3 years, I have only had 1 outage that was noticed by any customers. I guess it depends on your expectations - to me 2 outages in 3 years is acceptable. Of course with a 100% SLA promise, any outage is bad. Lets just hope it comes back quick.
Posted by dqh, 12-11-2005, 12:24 PM
LQ has been fix . welcome back
Posted by eth00, 12-11-2005, 12:25 PM
Looks like the entire DC lost power, we have 8 boxes all with only a few minutes uptime
Posted by Michael Kjærsgård, 12-11-2005, 12:25 PM
I just think that a lot of mail etc. from the past hour is slowing down the network
Posted by litobelo, 12-11-2005, 12:25 PM
At least they should explain the situation. We are not talking here about small things here like selling chips.
Posted by oshawa, 12-11-2005, 12:27 PM
This is the 'good type' of being down ... not your machine so it is not your problem. When this happens (and I might say it rarely happens at liquid ... only a couple of times in a couple of years), I just tell anyone who phones that technicians are currently working on the problem and that they can expect connectivity shortly. If it turns into a longer call, I thell them that I need to keep the line open .... the best part is that i just sit back and wait for the dc to solve it. Much better than when it is your machine and you have to resort to backups or wait for hardware fixes.
Posted by JNL Host, 12-11-2005, 12:27 PM
Not really.. The 100% is a gaurantee, not so much an indication that it will never go down. Just means that if it does, then we are compensated as required. Jason
Posted by hozt, 12-11-2005, 12:28 PM
My server is also back. Hope they learn from this downtime. Being this close to christmas, it is a bad time to have any downtime.
Posted by gennyblake, 12-11-2005, 12:30 PM
One of my servers is up - the other is still down (of course, THAT one is the money maker GRRR)
Posted by litobelo, 12-11-2005, 12:31 PM
Thanks god
Posted by eth00, 12-11-2005, 12:31 PM
murphy's law of course! We are still waiting on 1 to come back, guessing anything still offline is going to have to be manually investigate or finish the fsck
Posted by JenniH, 12-11-2005, 12:32 PM
Still no sign of life for us. How did you get hold of the engineer accyroy? We can have out main sites up and running at a backup host in about 30 mins, so could do with an ETA to determine whether it is worth that investment of time/money. We have to set a cut-off at some point in time and assume the worst.
Posted by Dacsoft, 12-11-2005, 12:33 PM
I agree. That is why I said 2 outages in 3 years is acceptable. While any outage is bad for business, outages are to be expected. How fast the DC recovers from the problem is the bench-mark.
Posted by accyroy, 12-11-2005, 12:37 PM
I just got lucky on the phone and got through. Just keep trying on the phone
Posted by N_Sami_N, 12-11-2005, 12:38 PM
Hello , This first Topic .. LiquidWeb is Down !! See here : www . liquidweb . com www . 9m . com What's the Problem ? Thank You .. All the best, Sami
Posted by sirius, 12-11-2005, 12:41 PM
Do a SEARCH before you post.... Sirius
Posted by myhken, 12-11-2005, 12:41 PM
Ohhh man....why post when you don't read the forum???
Posted by nucleo, 12-11-2005, 12:47 PM
Back for me for few minutes now but liquids website is still down... Like other said this is my first downtime.. can't really complain
Posted by gennyblake, 12-11-2005, 12:48 PM
one of my servers has been up for 40 minutes now. the other one is still down. I'm on perma-hold with LW now. Sigh.
Posted by Patrick, 12-11-2005, 12:48 PM
Very slow loading, but it is working...
Posted by LuisMiranda, 12-11-2005, 12:50 PM
Liquidweb website is up, but still my websites are down.. However, i'm staying with liquidweb too.
Posted by Michael Kjærsgård, 12-11-2005, 12:54 PM
DNS problem, it is reachable at 209.59.139.21
Posted by 123finder.com, 12-11-2005, 12:54 PM
Yah, mine was down and it was rebooted too. It's now back up but quite slow (seems like a network problem and not load avg). Don't know what's going on?
Posted by JenniH, 12-11-2005, 12:57 PM
We are still down... this outage is becoming long... and serious. It would be nice if they actually provided some information on a websites/forum/anywhere.
Posted by gennyblake, 12-11-2005, 01:00 PM
http://www.liquidweb.com/netstatus/ LOL - "There are no known problems." One server still down. Sigh.
Posted by litobelo, 12-11-2005, 01:00 PM
What bother me is that one of our servers was reboteed after 320 days of uptime. I hope that someone from LW will explain how a company like them can be totaly down
Posted by neutro, 12-11-2005, 01:03 PM
still not up yet for us
Posted by daraghmacl, 12-11-2005, 01:04 PM
Seems they are having probs still - their site was up for me earlier but appears to be gone under again.. hope it all irons out soon!
Posted by 1ONE, 12-11-2005, 01:04 PM
Will we get any compensation for downtime ?
Posted by Gary Flintstone, 12-11-2005, 01:07 PM
I still can't establish an FTP connection to upload some files -- and I'm still pissed. Hope they have a good explanation so it can be ripped to shreds.
Posted by accyroy, 12-11-2005, 01:08 PM
Both of my servers are now having problems resolving domain names. Can anyone else confirm if this is a problem at liquidweb or my servers, just do a simple ping of google from your box and let me know your results, thanks. **edit typical, as soon as I posted this both servers started resolving..! Last edited by accyroy; 12-11-2005 at 01:11 PM.
Posted by litobelo, 12-11-2005, 01:10 PM
The SMTP of all our servers are not working. Emails are down. Anyone with same issue ?
Posted by Dacsoft, 12-11-2005, 01:10 PM
ping to google.com is fine from my box.
Posted by greengiant, 12-11-2005, 01:11 PM
We still can not get to any of our boxes at LW
Posted by Michael Kjærsgård, 12-11-2005, 01:11 PM
No problems here, 2 servers tested..
Posted by ErikZ, 12-11-2005, 01:13 PM
I haven't been able to access any of my sites or the LW home page the entire time. I'd like to have more info, but I don't expect to actually get any until the problem is fixed. Frankly, I'm glad it's just a hardware/power problem. As opposed to LW going out of business and dissapearing with all my sites. I've really been lax with my off-site backups.
Posted by gennyblake, 12-11-2005, 01:14 PM
Yep, the server is responding (now slowly - it was fine earlier), webpages are up, I can ssh and ftp in fine - but when trying to send email I get this: [Mail not sent. Sending error: 451 Temporary local problem - please try] This server has been up for just over an hour. Our other server is still dead.
Posted by 1ONE, 12-11-2005, 01:14 PM
My servers are ok , I don't have any problems with resolving or email... I think your problem is local on you server...
Posted by JenniH, 12-11-2005, 01:15 PM
We have pished the panic button and are restoring at our backup host as I type this. We can't just keep hanging on HOPING they will get it sorted soon.
Posted by manisha_khtri, 12-11-2005, 01:15 PM
after server down i received following errors on phpmyadmin can anybody tell me about this error? Warning: Illegal offset type in /usr/local/cpanel/base/3rdparty/phpMyAdmin/libraries/grab_globals.lib.php on line 87 Warning: Illegal offset type in /usr/local/cpanel/base/3rdparty/phpMyAdmin/libraries/grab_globals.lib.php on line 87 Warning: Illegal offset type in /usr/local/cpanel/base/3rdparty/phpMyAdmin/libraries/grab_globals.lib.php on line 87 Warning: Cannot modify header information - headers already sent by (output started at /usr/local/cpanel/base/3rdparty/phpMyAdmin/libraries/grab_globals.lib.php:87) in /usr/local/cpanel/base/3rdparty/phpMyAdmin/index.php on line 55
Posted by greengiant, 12-11-2005, 01:18 PM
Some of their upstream providers are dropping traffic, the network is just flooded. Everyone relax a bit and it should come back up. They have power restored.
Posted by accyroy, 12-11-2005, 01:19 PM
If its a cpanel server then login to whm and hit the repair mailbox permissions in the email menu. SOmething could have become corrupt during the power cycle.
Posted by universal2001, 12-11-2005, 01:22 PM
Its not bad if you have a few servers but what if you had 100 servers and each server has 600 or so domains? Thats about 6000 customer sites down. When that happens do you think you'll be okay with a few downtimes per year? Also 100% uptime is not a myth. GNAX.net/transactglobal.com has had virtually perfect uptime for the past 3-4 years I believe. They also have the best server pricing and support I've seen anywhere.
Posted by Gary Flintstone, 12-11-2005, 01:30 PM
No wonder they're down, you probably crashed them by having 600 domains crammed on each server.
Posted by Michael Kjærsgård, 12-11-2005, 01:33 PM
I believe a lot of lowcost hosting-companies have quite a lot more than 600 domains pr. server However, i made 600 x 100 = 60.000
Posted by Alex, 12-11-2005, 01:36 PM
Wrong forum... Their up for me.
Posted by Dacsoft, 12-11-2005, 01:40 PM
I guess it depends on your definition of "virtually perfect uptime" - and the qualification of "I believe". A quick seach of GNAX.net in the network outages section seems to disprove your assertions. (sorry, this is a flaw in my personality. I just can't help verifying some assertions). I have no experience with GNAX so can't comment on their service. I do have experience with Liquid Web and believe they are very reliable and professional. That being said - I wish they would get this fixed.
Posted by 123finder.com, 12-11-2005, 01:41 PM
It's faster now. Guess some DNS problem got resolved too.
Posted by voip, 12-11-2005, 01:43 PM
Does somebody have any server down? I have two working fine and one down.
Posted by argv1900, 12-11-2005, 01:44 PM
http://webhostingtalk.com/showthread.php?t=468541
Posted by cyberturk, 12-11-2005, 01:46 PM
all of the WHT is about liquidweb today http://www.webhostingtalk.com/showthread.php?t=468541
Posted by neutro, 12-11-2005, 01:46 PM
yes ours is still down..
Posted by greengiant, 12-11-2005, 01:47 PM
Yes. We have 3 servers down still. They are saying the netowrk is flooded and they are having problems with some of the upstream providers.. Funny thing is i can tracert to the distribution switch in 27ms, yet can't hit our servers..
Posted by universal2001, 12-11-2005, 01:50 PM
1 of our server is still down
Posted by neutro, 12-11-2005, 01:50 PM
yeah our tracert also stuck at dist switch 209.59.157.14
Posted by JenniH, 12-11-2005, 01:50 PM
Yes, still down here.
Posted by masssa, 12-11-2005, 01:54 PM
I'm curious how all this will be ended...
Posted by universal2001, 12-11-2005, 02:01 PM
Did you even read my reply properly? I also guess you just came from the moon because virtually all the biggest shared hosting firms these days put 600 or more accounts on each of their servers. When I was hosting with webhosting.com, one of the biggest hosting firms years ago (before they sold out to a major large telco firm) had 1200+ accounts on each server. I guess you are probably the type who only stacks 100-200 accounts per server claiming you have top support and customers are happy but got no profit to show for it.
Posted by 123finder.com, 12-11-2005, 02:03 PM
I have only one server w/ LW and it's back to the normal performance now.
Posted by universal2001, 12-11-2005, 02:06 PM
Sorry I should of re-worded my reply. I meant to say that GNAX has had no NOC-wide downtime in a very very long time. I just did a search as you did and I see some 15 threads about GNAX being down but these were all individual user issues, (e.g. Users ISP was blocking GNAX ips, machine specific issues etc). If you can point me to a sitewide thread whereby all GNAX servers down (like in several instances of Liquidweb) then please let me know. Thanks
Posted by Gary Flintstone, 12-11-2005, 02:12 PM
Whatever explanation LiquidWeb comes up with, I have a mad pitbull standing by to rip it to pieces when I print it out.
Posted by BLazeD, 12-11-2005, 02:14 PM
My server there is still completely down. No ping at all.
Posted by vechor, 12-11-2005, 02:16 PM
yes... I have 2 up and 2 off. did anybody get to speak with them? does know the problem that happens?
Posted by ruiner, 12-11-2005, 02:22 PM
My two servers are up and running.
Posted by cyanide, 12-11-2005, 02:25 PM
my servers came back up, but now their down again
Posted by Gary Flintstone, 12-11-2005, 02:30 PM
Hehe, did you even look at my smiley face properly? It meant that my comment was made in jest. Hehe, I guess you just came from the no sense of humor planet, because virtually all earthlings can take a joke. Hehe, I guess you are probably the type who is defensive about how many sites you have crammed on each server.
Posted by liquidweb, 12-11-2005, 02:42 PM
At approximately 10:30am EST Liquid Web's commercial power supplier "The Board Of Water & Light" lost power to our facility as a result of winter weather conditions. DC1 has two generators currently, a 313kva and a 650kva unit, the 650kva unit failed to start when the transfer switch attempted to activate it when commercial power ceased. The 313kva unit responsible for cooling and secondary systems started successfully. Technicians were already on site and with the assistance of engineering staff were able to diagnose the equipment start issue and activate the 650kva unit restoring power to battery and UPS systems and power was restored to all systems after an estimated 15-20 minutes of service interuption to customer equipment. Power to all customer boxes returned at this point in time and the large majority of equipment came back into normal operation at this point. Reserve staff has been assembled, and all available technicians are working to restore service to any customers with remaining issues. Please follow up directly with us with any remaining issues, we are working to restore service to any and all clients who's machines have not returned to service properly. Both generator units are load tested monthly and excercised weekly, and had experienced no failures in starting previously. An investigation with the supplier of the generator will be launched immediately. SLA credits will be issues appropriately in accordance with our SLA. Again please follow up with us directly.
Posted by vechor, 12-11-2005, 02:46 PM
I have 600 customers in each server. 2 are off, I don't know more than to do here. they are 1200 customers calling and I don't get to know that happens. does anybody know how to tell me something? I am being crazy here.
Posted by Xia, 12-11-2005, 02:51 PM
I can't submit a ticket to PIMS since https doesn't seem to be working.. cppop is continuously failing.
Posted by eth00, 12-11-2005, 02:57 PM
PIMS is down here too and would hope they can get it online soon. If their phones were working we coudl get in a ticket to get a server back online.... As far as your error do this: service httpd stop service httpd startssl service chkservd restart service cpanel restart and your problems are most likely fixed
Posted by 123finder.com, 12-11-2005, 03:00 PM
I'm glad I got some detail above about the outage at LiquidWeb. On the other hand, Hivelocity hasn't yet given any reponse for their outage of over 10 hours yesterday.
Posted by Dacsoft, 12-11-2005, 03:25 PM
I don't see any activity on this thread. Does that mean everybody is back up? I still have 1 down - trying to figure out if there is still a problem with DC or if it is my server.
Posted by JenniH, 12-11-2005, 03:29 PM
No, still down here.
Posted by neutro, 12-11-2005, 03:30 PM
i am still down
Posted by ram27, 12-11-2005, 03:32 PM
We still have one down, so things are not all fixed yet. I did get into PIMS to submit a ticket, so we will see what happens.
Posted by BLazeD, 12-11-2005, 03:40 PM
server still down with no ping even
Posted by Dacsoft, 12-11-2005, 03:41 PM
I was able to access PIMS to submit a ticket. Phone goes to a recording, but that understandable. I would rather have the technicians fixing servers instead of answering the same question. At least I am down to only 1 servers worth of unhappy customers.
Posted by neutro, 12-11-2005, 04:02 PM
did you get replies for your ticket? i haven't got any.. probably they have to many to reply
Posted by BLazeD, 12-11-2005, 04:08 PM
I dont get why this was posted, which seems to say they think they have fixed the main problem, when there is people like me with aboslutely no ping or anything to our servers. There is still plenty of ppl posting in this thread their servers are down.
Posted by Dacsoft, 12-11-2005, 04:09 PM
Same here. It isn't easy to sit and wait, but isn't much to be done.
Posted by universal2001, 12-11-2005, 04:10 PM
I take some things too seriously.
Posted by eth00, 12-11-2005, 04:16 PM
At this point it is probably just problems with individual servers, not the datacenter as a whole. I would imagine quiet a few of the boxes are going to need manual intervention and it will take awhile since their entire datacenter lost power heh...I wonder when the last time was they tested the generators or if this was just a fluke.
Posted by Dacsoft, 12-11-2005, 04:27 PM
According to their recent post:
Posted by WebHosting Zac, 12-11-2005, 04:27 PM
We're working on getting everything back online right now, so I apologize if your server is still down. The issue which caused the servers to go down originally has been resolved. Our phones have an initial message, but you can still reach a tech if you wait on the line after the message. Our techs are still busy going through our lists of downed servers/services, but if you give us a call or open a helpdesk ticket (just one please) we can get your server back up right away.
Posted by bm_ab, 12-11-2005, 04:37 PM
My server there is still completely down. No ping at all.
Posted by nmedia, 12-11-2005, 04:40 PM
One of our servers is also not pinging. (the other is up)
Posted by Michael Kjærsgård, 12-11-2005, 04:41 PM
Bah! Go change that signature, will you?
Posted by BLazeD, 12-11-2005, 04:51 PM
Great - ticket submitted over an hour ago for my completely dead server - #301233
Posted by BLazeD, 12-11-2005, 05:25 PM
My server is now back up. This is directed to anyone - how does this SLA credit work / calculated? I have never received one before from other DC's Last edited by BLazeD; 12-11-2005 at 05:33 PM.
Posted by Anky, 12-11-2005, 05:35 PM
http://www.liquidweb.com/dedicated/sla/
Posted by BZProd, 12-11-2005, 06:03 PM
Hello everyone, I have a few servers at Liquid Web and they have been down for many hours. I have tried calling the 800 number and there is a message saying that they had a power outage and they are working on it. This has been hours. Are other people having this same problem? Liquid Web has been great for me for nearly a year...until now.
Posted by neutro, 12-11-2005, 06:07 PM
server is back up. (actually it was hung on reboot). if your server still not up yet.. call them (my neck hurts waited for them 1 hr to pick up my call)
Posted by universal2001, 12-11-2005, 06:14 PM
called and posted ticket... havent had an update yet for the past 3 hours! Is LW under staffed during emergencies??? Can someone look at our tickets or pickup the phone!!
Posted by universal2001, 12-11-2005, 06:17 PM
We are in the same situation... Not looking good atm. Looks like they're 100% network uptime is broken and worst of all, it looks like they cannot deal with crisis issues like this... We have a server thats down for over 4+ hours, noone is picking up the phone or replying to support tickets. THere is a big thread in the downtime annoucement section with many people facing similar issues, if you want updates head over there.
Posted by BZProd, 12-11-2005, 06:25 PM
thanks for the heads up...I'll check over there
Posted by Dacsoft, 12-11-2005, 06:27 PM
All my servers are back. Good luck to the rest of you.
Posted by JenniH, 12-11-2005, 06:34 PM
Some stuff still down here..... and still no sign of recovery.
Posted by WinHosting, 12-11-2005, 07:13 PM
We have several servers there. Several hours ago they all were down. Then all became up. Now one of them is down again. Definitely it is not 100% uptime.
Posted by Anky, 12-11-2005, 07:47 PM
You guys need to call them up, while they monitor everything and are going through each one it'll be faster to give them a call since someone will look into all that right there.
Posted by layer0, 12-11-2005, 08:08 PM
Wow....hold up there. The real issue, which was with the generator, was resolved in less than an hour from what I can see. However, many servers probably need to a) go through manual intervention as eth00 said b) go through an fsck process which can take time for servers full of customers So technically, you cannot fault Liquid Web for taking too long to deal with the issue. They took care of it in less than an hour. The outages experienced are isolated to each server. Also WinHosting, That's not at the fault of their electricity or network - it's your server that went down, probably due a crash . . . why are you blaming them for that? Watch what you do here - it sounds very immature and ludicrious berating your provider because your server isn't starting in public (while representing your company in the signature). Just my 2 cents.
Posted by MonteCarloHosting, 12-11-2005, 08:21 PM
I must say, LiquidWeb's responce was once again rapid - I have two servers with them, and admin a third, all where back up within 20 mins of going down, with only one server experiencing some issues (win 2k3 clamAV didn't start). Fixed that, everything's back to normal I figured it was something to do with the power grid... I heard that they're having pretty bad weather in Michigan right now, so it's no surprise that power went out... As for gen-sets, LW's site lists that they are Cummins diesel engines... cold + diesel engine = no start Anyway, enough rambling Thanks LW... very fast responce, as usual -Christian
Posted by BZProd, 12-11-2005, 08:27 PM
We had 8 servers down, some for many hours. I was able to get someone on the phone and he handled the remaining servers immediately. Although todays outage was ridiculous, LW has been great to us and I appreciate their response. The only thing worse than going down for hours is going down for hours and not hearing anything from the company. A+ reaction in my opinion. Hopefully they will be better prepared if this ever happens again.
Posted by macdonaldp, 12-11-2005, 08:41 PM
They have tons of customers. You really expect them to take phone calls from all of them? I'm not sure any company would have that many staff to take calls from hundreds of people, while trying to fix the servers. They did the correct thing in just fixing the servers so that it could minimize the amount of phone calls with server issues. They resolved this very quickly, and I congratulate them on that. Last edited by macdonaldp; 12-11-2005 at 08:46 PM.
Posted by BLazeD, 12-11-2005, 09:45 PM
Only 20 mins? Dang, mine was more like 5 hours from what I can tell. But yeah, apart from the amount of time my particular server was down, good response from LW.
Posted by jayzee, 12-11-2005, 10:13 PM
Liquidweb.com still appear down for me.
Posted by macdonaldp, 12-11-2005, 10:20 PM
Definitely up. Probably just an error on your side.
Posted by jvreeland, 12-11-2005, 10:55 PM
I am still down completely - been so for a few hours.
Posted by macdonaldp, 12-11-2005, 10:59 PM
That's too bad. I assume you must have tried phoning them, right? (1-800-580-4985) I've talked with liquid web tech support on the phone before and they always were quick to fix any issue.
Posted by MonteCarloHosting, 12-12-2005, 12:22 AM
LW's site is up for me too, so is PIMS. I agree with Patrick, try phoning them, I'm guessing your server is suffering from a hard-error caused by the power-outage. Sending in a ticket doesn't seem to help right now - I sent one in for another issue this afternoon, but haven't had a responce yet; then again, it's not that important, first get everyone back up -Christian
Posted by hozt, 12-12-2005, 01:15 AM
Why didn't the UPS kick in after the power went out? They claim that they can run on batteries for an hour: From their website: "The intermediate load is capable of being handled for over an hour by the UPS..."
Posted by JenniH, 12-12-2005, 03:04 AM
There are still problems from here. We have sites via several hosts up and running, but issues persist with others. For example, the Affordablehost server is still down, and our Site5 control panel can't be opened, even though the sites their load. As some of these are ok via Proxify (but not others) maybe there are some residual IP/network issue around? Don't know, but definitely still plenty of problems.
Posted by JenniH, 12-12-2005, 04:01 AM
For one of our domains, a tracert from my PC gives a timeout at: lw-core2-ge4.rtr.liquidweb.com [209.59.157.26] lw-dist4-ge.rtr.liquidweb.com [209.59.157.54] I'm trying to understand what specifically that indicates. Server? Nameserver? Router? Something else? Can anyone assist?
Posted by Arno|VDH, 12-12-2005, 06:25 AM
The 1st one seems to be a core router the 2nd seems to be the 4th dist. router, i only can assume it's either district or distance or any kind of that. tho i'm not sure about it. maybe some 1 else has an other view of that
Posted by JenniH, 12-12-2005, 06:59 AM
That makes sense. Thanks. The problem does seem to be variable as well... always dead from some locations, but access from others comes and goes. I raised a ticket with LW a couple of hours ago. No response yet.
Posted by layer0, 12-12-2005, 08:39 AM
How is a networking issue like that related to the recent power problems? Also, my.site5.com is on NAC bandwidth. JenniH, I find it quite surprising that you apparently have any account with just about every company that has a major outage (which is recorded at WHT).
Posted by JenniH, 12-12-2005, 08:58 AM
I don't know... that's obviously I'm asking the question. Maybe a router was damaged, or similar. Don't know. Really? We host with quite a number of hosts, and try hard not to have too many sites in the same data center. Hence, we are obviously liable to be hit by most major DC outages: but the impact on us when it happens, is far less than if we had all our eggs in one basket and that basket happened to be hit. It's a strategy we are comfortable with as it mitigates risk for us in terms impact. Perhaps now you are less surprised? Others adopt the same strategy, but to a lesser degree.
Posted by bhaputi, 12-12-2005, 10:27 AM
FYI - dist would most likely be a distribution layer device - see http://www.mcmcse.com/cisco/guides/h...al_model.shtml if you do not know what that would mean.
Posted by mgoodin, 12-12-2005, 11:06 AM
Wow - over the last two weeks we have researched a new data center for our hosting servers. We narrowed the search to rackspace and liquid web. After reading this thread, I have serious concerns using liquid web. We have been down a similar path over the last few years with a different data center - periodic down time, etc. We are trying to get away from downtime as much as possible. If we need to pay a few extra dollars, so be it. I realize there is no 100% uptime Good luck to all in this thread
Posted by Website Rob, 12-12-2005, 11:27 AM
mgoodin, don't fret over this thread/outage as you would be hard pressed to find another like, regarding LW. Stuff happens and is to be expected. We use the same methodology described by JenniH, spread things around, and it works well. Although we only have one Server with LW at this time, for approx. the last 5 - 6 months, it was only down for about an hour. I can only presume that is how long it took to correct the outage situation. I did notice that our Server was rebooted and again, can only presume other Servers had to be rebooted and some people had longer outages due to problems specific to their Server(s). Although an outage of this type is usually far & few between, it can be more scary when having a Server rebooted.
Posted by mgoodin, 12-12-2005, 12:00 PM
Thanks for the advise Rob. We have been with different data centers throughout the last 5-7 yrs one of which was/is CI Host ... ahhhh yes, the famous CI Host CI Host has had nothing but bad things mentioned on different forums, but in the last 12+ months we have only experienced 3-5 periods of downtime ... this would typical occur on Monday mornings about 8:00 am The most frustrating part is not be in control. Nonetheless, there is an uneasy feeling when your clients are calling and you find yourself completely dependent on the data center to keep your business floating. If going down 2-3 times a year is good/acceptable in the industry, maybe CI Host is on the rebound
Posted by MonteCarloHosting, 12-12-2005, 12:47 PM
I believe the UPS did kick in... but it's only there for an hour, after which the gen-sets need to be running, and producing power. Diesel engines need a minimum of 30 minutes run-time before they're at their operating temparature (and, at peek output). Of course, bad weather, cold mornings, and humid air don't help in starting an engine I'm guessing it was either this, or that the starter failed... which is another all-to-common occurence in DEs. Just my 0.02
Posted by Michael Kjærsgård, 12-12-2005, 02:39 PM
Well, then they've had ½ hour to find a pair of jumper cables
Posted by andren, 12-12-2005, 07:33 PM
Liquidweb hasn't been able to fix our issues, we have gotten only canned responses so far. Problems started after the crash yesterday, and LW hasn't fixed any. We depend on our customers downloading large files. We would be interested in download speeds: http://www.ankeservers.com/typo3_src-3.8.1.tar.gz (11.6MB) Any guess what could be the cause? We got more than 1MB/s sustained before the crash to the NAC, now 20kb/s. Please state the obvious too - we may have missed something. Also we have extremly high broadcast traffic (40GB/day). Where could that come from? (Liquidweb: 'We don't understand the problem') Any help highly appreciated. We start to get fed up with Liquidweb.
Posted by macdonaldp, 12-12-2005, 07:37 PM
I'm sorry to hear that. In regard to the download speed issue. I was able to download that file at a speed of ~15 - 22 KB It definitely seems that there is a problem. I know when it comes to specific issues it seems Liquidweb cannot answer them. I had experience with them trying to fix a weird issue on one of their servers and it took forever to get resolved. Overall the service was good though. Last edited by macdonaldp; 12-12-2005 at 07:41 PM.
Posted by andren, 12-12-2005, 08:17 PM
Patrick thanks for the test. Forgot to attach the bandmin screenshot, here it is. Attached Images lw.png (34.1 KB, 23 views)
Posted by Website Rob, 12-12-2005, 08:55 PM
http://www.ankeservers.com/typo3_src-3.8.1.tar.gz A steady 150K/s through a Browser download on DSL. A steady 587K/s using Server to Server transfer.
Posted by andren, 12-12-2005, 08:59 PM
Yup, switch has been rebooted. It took 32 hours, 20hours from our first ticket...
Posted by MonteCarloHosting, 12-12-2005, 09:14 PM
andren, Of course, this is another server in the LW network I'm guessing someone is hogging all of your bandwidth? -Christian [EDIT] Sorry - didn't see your post on Page 14[/EDIT]