is liquidweb.com down? [merged]

Portal Home > Knowledgebase > Articles Database > is liquidweb.com down? [merged]

Posted by BLazeD, 06-21-2006, 06:31 AM
Their website, PIMS, my server... all down.
Posted by ellinas, 06-21-2006, 06:32 AM
Same here... all my servers down and their website also.
Posted by ThomasO, 06-21-2006, 06:32 AM
Yeah... everything is on the floor here...
Posted by accyroy, 06-21-2006, 06:33 AM
yup, 2 servers down here as well.....
Posted by radiofm, 06-21-2006, 06:33 AM
my server also down. www.liquidweb.com also down
Posted by Drac, 06-21-2006, 06:36 AM
Could it be anything to do with new the data center?? Regards, Drac
Posted by accyroy, 06-21-2006, 06:37 AM
Can't seem to be able to get them on the phone, mainly getting busy tones but just got through and its ringing and ringing.....no answer.... If anyone does manage to get through please let us know whats going on!
Posted by cyclone2k, 06-21-2006, 06:42 AM
Yuck! I thought it was just me. All of my sites are down. Hope it resolves soon or my clients will kill me!
Posted by solidhostingph, 06-21-2006, 06:48 AM
can't reach our server or even liquidweb. is there a problem?
Posted by SDH1, 06-21-2006, 06:50 AM
Certainly looks like it. We've been offline for the past 10 minutes and their site is down as well.
Posted by bigfeet, 06-21-2006, 06:51 AM
I was just about to ask the same question, as I can't access my server or the liquidweb.com site either!
Posted by chiehkai, 06-21-2006, 06:52 AM
My servers at LW went down too. DOWN at Jun 21 2006 06:25 PM (GMT +8:00) Err: ping: request timeout
Posted by mindnetcombr, 06-21-2006, 06:53 AM
yes, the liquidweb network is down. I have some servers in LW all unreachable. 8-(
Posted by greengiant, 06-21-2006, 06:53 AM
They are back up, at leat partially. SOme of our servers are responding and there site is up.
Posted by gonz0, 06-21-2006, 06:54 AM
NS.LIQUIDWEB.COM 209.59.139.20 NS1.LIQUIDWEB.COM 64.91.251.155 NS9.LIQUIDWEB.COM 64.191.66.85 only last address is working. i can;t get any of my site it's down for 25 minutes now ...
Posted by tjawatts, 06-21-2006, 06:54 AM
Mine are down too. Tried to call support but I got "The person you are trying to reach is unavailable" for all the support options. Tony
Posted by radiofm, 06-21-2006, 06:54 AM
it's look like power outage. phone or busy or no answer.
Posted by JTY, 06-21-2006, 06:54 AM
They appear to be up from here. I can access their website.
Posted by manisha_khtri, 06-21-2006, 06:54 AM
Guys All my websites are nt working server down from last 15 mintues . i tried to go there site(liquidweb.com) bt it is also down.
Posted by andren, 06-21-2006, 06:55 AM
Their site may be. Our servers still down.
Posted by solidhostingph, 06-21-2006, 06:55 AM
we called LW and they answered in the first ring. they were strucked by a lightning and they are now recovering everything back to normal.
Posted by Drac, 06-21-2006, 06:55 AM
Yeah, site seems to be working again. Not sure about the servers, but I think they should be up soon Regards, Drac
Posted by skywin, 06-21-2006, 06:55 AM
My servers already is UP but they have been reinitiated....
Posted by accyroy, 06-21-2006, 06:55 AM
Yup definately looks like power, I've just logged into one of my servers and it has power cycled.
Posted by gonz0, 06-21-2006, 06:57 AM
good to hear that koolme
Posted by chiehkai, 06-21-2006, 06:57 AM
UP at Jun 21 2006 06:52 PM (GMT +8:00) Was down for: 27min (From HyperSpin)
Posted by radiofm, 06-21-2006, 06:59 AM
my VPS still down but the good is that liquidweb site is up.
Posted by matteo, 06-21-2006, 06:59 AM
my vps is already down this is not good :°°
Posted by tjawatts, 06-21-2006, 06:59 AM
Mine seems to be back up now too. Hopefully there will be an explanation? Tony
Posted by Drac, 06-21-2006, 07:00 AM
If the power did go out, it will take some time for all systems to become online again, as all their networking equipment would have to be loaded. Regards, Drac
Posted by andren, 06-21-2006, 07:04 AM
Server is back. Power issue, server was rebooted.
Posted by nowanda, 06-21-2006, 07:09 AM
our servers are back to normal again. what a relief!
Posted by LiquidWebJer, 06-21-2006, 07:11 AM
It appears that we were hit by lightning. During an overnight electrical storm, a very loud "boom" was heard by everyone in the office. Moments later almost every breaker in our building was blown. We've reset all breakers, UPS and PDU, and power has been restored. We will be paying an SLA credit to anyone that would like it. More information will be posted as it becomes available.
Posted by RossH, 06-21-2006, 07:12 AM
Why would you pay an SLA credit on an "act of god"?
Posted by crystalz, 06-21-2006, 07:17 AM
A thread is already running in the Providers and Network Outages and Updates forum. Mods please merge this thread.
Posted by vechor, 06-21-2006, 07:19 AM
Please, check Show Current CPU Usage and tell me what has. has several of those: mailnull 0 0.0 0.1 / usr/sbin/exim - bd - q60m? or will it be that it is alone with me?
Posted by universal2001, 06-21-2006, 07:22 AM
we still have servers down, and noones picking up the phones, are you looking at the tickets..
Posted by neutro, 06-21-2006, 07:24 AM
yah i have been calling like crazy fr International.. PIMS is out too
Posted by cyclone2k, 06-21-2006, 07:26 AM
My servers are back!! Glad, that the problem sorted out, not so late!
Posted by greengiant, 06-21-2006, 07:29 AM
We still have 3 servers at LW that are down. Noone is picking up the phone, PIMS is down, and we did not recieve a autoresponce back when we emailed support..
Posted by neutro, 06-21-2006, 07:35 AM
my cell running out of batt calling intl non stop.. please bring pims up!
Posted by andren, 06-21-2006, 07:49 AM
May be because a good electrical setup would prevent this? (Not an electrician, may be I am wrong.)
Posted by cyanide, 06-21-2006, 07:49 AM
PIMS is back up, but our servers are still down
Posted by VER-Mo, 06-21-2006, 07:55 AM
My Server is STILL down. I cannot access PIMS on the Liquidweb website either.
Posted by Tyler, 06-21-2006, 08:02 AM
Threads merged once again.
Posted by FredForrest, 06-21-2006, 08:03 AM
Actually if they took a direct strike and got things back up as fast as they did, the electrical system as well as network infrastructure is very well protected. The fact that most servers appear to be back up and not fried after the building took a direct hit indicates some pretty good protection in place.
Posted by tomcupr, 06-21-2006, 08:09 AM
What can be wrong? I can access my site through Putty, I can see from logs that people are accessing the site but I can't access it....
Posted by neutro, 06-21-2006, 08:16 AM
So everybody up? Mine still down not even SSH - nothing
Posted by greengiant, 06-21-2006, 08:20 AM
neutro, No there are still a number of boxes not up. I have 3 down still.. They called in all staff and are working to get everyone back up. They have not posted an ETA yet though..
Posted by neutro, 06-21-2006, 08:25 AM
thanks.. grengiant.. 8.24am there.. thank god world cup is on.. right now.. or else i am swamped with calls
Posted by Rasbelin, 06-21-2006, 08:26 AM
Well, I assume I'll be the first one daring to questionise this, but I can't say pretty much anything else that it's a pretty weak and badly planned datacenter, if there's no lightning rods/protectors on the roof of the building, no matter how low or tall the building is. It's just a basic safety measure, just like fire suppression is. Because if the lightning bolt would have been diverted to ground with the help of the lightning protector, the damages would have been next to none. Or if there was lightning protection, how was the groundwire routed and in what shape was it?
Posted by LiquidWebJer, 06-21-2006, 08:34 AM
Our power company has confirmed that we did take a direct lightning strike. The fuses up on the pole are blown, and they are in the process of testing the cables before replacing the fuses. We're on generator right now and have plenty of fuel. We've got every staff member that we could contact called in and rebooting/fixing servers that didn't come up on their own. We also have a problem with our phone system and are working on restoring normal service.
Posted by masssa, 06-21-2006, 08:49 AM
My server is now online, but I have a problem. APF is blocking smtp. Does anyone know what could be a problem?
Posted by BLazeD, 06-21-2006, 08:55 AM
I had to restart mail servers after my server came up - dont know if that helps
Posted by Burnspot, 06-21-2006, 08:56 AM
SMTP problems are all we're seeing right now too...EXIM seems to be having a fit every once in awhile. Hopefully, they'll have it all cleared up soon. We need to send mail.
Posted by subnet_rx, 06-21-2006, 08:58 AM
Is this the same issue affecting Site5 servers? They are saying a MI facility is down. But my server isn't up yet.
Posted by ASTHost, 06-21-2006, 09:12 AM
Server have been running extremly slow after its back. Anyone experiencing the same thing?
Posted by subnet_rx, 06-21-2006, 09:14 AM
Hmm, so Site5 reseller and LiquidWeb host in the same facility. I was thinking of switching to LiquidWeb because of all the downtime I've been experiencing lately, but that would've been a mistake.
Posted by TorrentBits, 06-21-2006, 09:19 AM
To Ross / STL Act of God? What Act of God... A lightning, storm etc isn't an act of God, you can prepare yourself for that type of situation (eventhough is rather unlikely it does happens) A meteorite falling on your head IS an act of God however... Hope our data is fine though...
Posted by dolay, 06-21-2006, 09:29 AM
Okay, No need to panic like that i believe, let them work on the issues calmly. I am pretty sure liquidweb is taking care of the any problems as human can do now. We will be past this issue with 2-3 hours of outgate only.
Posted by Knuttz, 06-21-2006, 09:44 AM
I solved mine this way: /sbin/service syslog restart /sbin/service exim restart In one of my boxes, it worked right out. On other box, I had to reboot the server. The antirelayd was failing due to /var/log/maillog rotation problems (runing Fedora/CPanel on the boxes) Got the tip on CPanel's forum: http://forums.cpanel.net/showpost.ph...56&postcount=4
Posted by matteo, 06-21-2006, 10:19 AM
Hi Jer, where i can find more information about this refound? i don't wanna speculate, i love LW support, but i have over 100 clients very very bad :°° the server down was append in italy, during the most important part of the day. thanks a lot. matteo.
Posted by VER-Mo, 06-21-2006, 11:02 AM
I too would like to know more about this.
Posted by spdfox, 06-21-2006, 11:21 AM
liquidweb is not down now; I enter well to their place web.
Posted by bhaputi, 06-21-2006, 01:23 PM
To me (granted, I am an outside observer) it seems that this was handled phenomenally well, and I say kudos once again to LiquidWeb. It looks like they were down (for the most part) for only about 20 minutes after taking a direct lightning hit. I would say the people that are saying that a better design would have kept this from happening are wrong. Good job, LiquidWeb.
Posted by Mitra, 06-21-2006, 02:26 PM
It sounds like the systems were brought back online as quickly as possible and that LiquidWeb has been nice about communicating such here on this board. However, This is proof positive that LiquidWeb does not have the infrastructure in place to support hosting of revenue-generating sites or mission-critical sites that need to be up 24x7. The Italy post above is just one example. To argue otherwise is silly: lighting = downtime = not good. If you can deal with the occasional outage LiquidWeb seems fine. If you really need uptime, you need to go with a better host. I'm with Rackspace for several reasons... two of them are that they've had no power outages and no network outages. They have properly grounded facilities as well as battery and generator backups.
Posted by LiquidWebJer, 06-21-2006, 03:40 PM
Hi All, I just wanted to answer some questions and clarify what happened: 1. SLA credit - if you'd like one, please log into you PIMS and submit a ticket requesting this. We'll give credit for 10 hours even though the vast majority of servers were down for around 25-35 minutes. 2. Lightning protection - we do in fact have lightning protection at both facilities in the form of "lightning arrestors". In this case the strike was so severe that the arrestor exploded, but according the the power comany's crew, this is not uncommon. We asked if there was a better lightning protection system, but according to our power company lightning will pretty much destroy anything when the strike is that severe. Our UPS also has built in protection, and this appears to have worked exactly as advertised - it immediately sent a signal to all of our battery cabinets and power distribution units telling them to trip their breakers in order to protect all of the equipment. Absolutely no equipment inside the DC was damaged. 3. What happened? The lightning directly hit the pole that we draw power from. It destroyed the cable on the pole and the lightning arrestor first. It then traveled through underground cables, blew the connectors off of one transformer, and blew a hole through part of the underground cable insulation, which caused a severe ground fault. We actually have feeds coming from two different sides of the building for redundancy, but unfortuately there was damage on both sides of the feeds. We ran successfully on generator power for most of the day while the power company fixed the cables and transformers. At around 2:30PM we were able to switch back to city power and return to normal electrical operation. We of course would like to offer our sincere apologies to anyone that this severely effected. Contrary to some speculation in this thread, we really did put a lot of time, planning and money into building our electrical system. For the most part it did exactly what we would have wanted in a situation as rare and extreme as this. No inside DC equipment was damaged, and the UPS and generators were able to easily keep everthing up as our power company did all of their outside work. I hope that this sheds some light on the situation. Thanks for your time. Jeremy Hill Liquid Web
Posted by matteo, 06-21-2006, 04:48 PM
extremely clear Thanks jeremy, no problem. Now i know what is SLA, and i confirm: we don wanna speculate. only info for our clients. LW rox. i know rackspace is actually best server solution, but we need a VPS, and rackspace offers only dedicated solutions. best regards mattep
Posted by coight, 06-22-2006, 06:29 AM
I just want to say, I'm not your customer but that's bloody outstanding. Many don't even offer credits when they are down an hour or longer. That's superb
Posted by Drac, 06-22-2006, 06:38 AM
Yeah I wish I didn't have to close my old server with LiquidWeb. They offered some of the best services money can buy. I do look forward to renting a server from them again when I have the budget and need for it. Regards, Drac
Posted by amps, 06-22-2006, 09:55 AM
I am not a customer but having deep knowledge of electrical systems I cannot fault LW's response or breif downtime. When a strike hits the main power line, losing power is sometimes unavoidable, regardless of protection in/on the building. I'm actually very suprised the UPS's weren't destroyed, at the very least. Opening a fuse isn't going to stop the energy burst of lightening from jumping across.... sooo if the story is truely as told, wow, consider your data at LW lucky to be alive!!
Posted by subnet_rx, 06-22-2006, 11:50 AM
I wonder how many web hosts use the center and were affected yesterday.
Posted by ASTHost, 06-22-2006, 02:02 PM
Well done Liquidweb.