Hivelocity Down?

Portal Home > Knowledgebase > Articles Database > Hivelocity Down?

Posted by lingates, 12-23-2004, 01:06 PM
Since 9am Mountain time, Hivelocity.net is having issues. Confirm this? http://www.hivelocity.net
Posted by Whocares, 12-23-2004, 01:12 PM
Yes. This is same for me. It seems down
Posted by Kayvon, 12-23-2004, 01:18 PM
This is not good. I have 18 customers that are currently down. WEbsite, email, phone, everything. What a great christmas, If anybody finds out anything about the reason why hivelocity is down, Please, Please let us know. Thier phone has a busy signal too all day. This is not good.
Posted by Whocares, 12-23-2004, 01:21 PM
I wonder how they will recover our loss. I am arguing now with my customers...
Posted by dollabillz, 12-23-2004, 01:44 PM
Hey down for me too... They r not on aim, no way to contact em at all. WTF!!!!!!! Pisses me off cuz I just read there post here talkin bad about a company that Hivelocity is now doing 2 and a half weeks later! webhostingtalk.com/showthread.php?threadid=350679
Posted by odenx, 12-23-2004, 01:44 PM
Me too, just recieved a dedicated server from them this morning. I was in the middle of setting up DNS, and POW the everything is down. I have sent email, called etc.. Hopefully we hear something soon.
Posted by Reseller-Center, 12-23-2004, 01:45 PM
Yea their whole /20 is down so probably a hardware/networking failure.
Posted by iSew, 12-23-2004, 01:47 PM
Ditto. Definitely down :-( I was seriously looking into buying a reseller plan from them... :-/
Posted by Kayvon, 12-23-2004, 01:54 PM
Probably the whole pipeline is gone... I wish someone from Hivelocity would post sometihng here and give us some estimate of what is happening and guidance...
Posted by nitra, 12-23-2004, 01:59 PM
Everything is dead and can't reach them on the phone. Nitra
Posted by KUNOFF, 12-23-2004, 02:00 PM
I am there largest reseller. 300 clients so needless to say that we haven't heard anything is not good. We are a non-profit hosting MANY organizational sites and email. My phone has NOT stopped ringing and my inbox... They need a way to let the resellers know what's going on. It seems like a power failure or something.
Posted by cbths1, 12-23-2004, 02:03 PM
I have been with them for a few years and this is not like them at all. The last time they went down (only for about 15 mins) their phone system went down as well due to a power outage, so I would guess it is a power issue.
Posted by ceridius, 12-23-2004, 02:28 PM
We have had a main transformer go down , and we are in the process of having the transformer replaced.
Posted by nitra, 12-23-2004, 02:30 PM
This is totally not right. Since Nov 24th being with them, I've had nothing but outages. I was told that the box would be up in 24-48 hours it took them 72 hours. Then the rack that my box was on needed work, the switch was being replaced. this cause my box to be rebooted 3 times while they did this. I lost a SQL database when the server was rebooted, because while the system was doing a FSCHK and was fixing itself they rebooted it yet again. Then following that, the UPS on my rack needed to be replaced, 2 more outages. 3 nights ago the PSU in my server blew, more downtime. Then yet again 2 nights ago I get this in my e-mail at 12:44 AM Dear Valued Customer, Hivelocity will be rearranging a portion of our datacenter tonight in order to better disperse power utilization. During this brief period of transition, your server/servers may experience brief disruptions in service. If you are not located in the affected portion of the datacenter, you should continue to experience the fast, reliable service you are accustomed to. If you are affected, then we appreciate your understanding and know that we will do our best to make the disruptions as brief as possible. Thank you again for your continued business and we wish you all the best. Thank you, Kevin Boyle Account Manager Now this would have been fine had they sent the e-mail at a normal time and not 12:44AM. But I did get the email, I called them asking how long the outage would be and also when I should shutdown the box to avoid SQL issues, The tech advised me that the server can be shutdown now, and that it woudn't be more then an hour until the work was complete. So I sat and waited. A little over 3 hours later the server was switched back on. Ever since that point I couldn't upload to the server at faster then 50KB/sec. I spent most of yesterday on the phone with them hearing that 50KB/sec is not that bad, finally I showed them my wget stats from thier own bandwidth test, which was local to thier network, and my box only got 500KB/sec to it. My computer at home on cable 2400 Miles away was getting 700KB/sec. Finally they said that there might be a switch problem, and the engineers would have to look into it tommorow. It took me from noon yesterday until 1AM for them to admit that there might be a problem. and 50KB/sec in my mind is not acceptable. Sorry for the rant Nitra
Posted by odenx, 12-23-2004, 02:31 PM
Thanks for the heads up...
Posted by KCLexGS, 12-23-2004, 02:33 PM
Thanks for the update. Any time estimate so we can set expectations with customers?
Posted by europac, 12-23-2004, 02:34 PM
I thought you guys had redundant power supply - that's what you advertise. David
Posted by nitra, 12-23-2004, 02:36 PM
sorry for the rant guys This is totally not right. Since Nov 24th being with them, I've had nothing but outages. I was told that the box would be up in 24-48 hours it took them 72 hours. Then the rack that my box was on needed work, the switch was being replaced. this cause my box to be rebooted 3 times while they did this. I lost a SQL database when the server was rebooted, because while the system was doing a FSCHK and was fixing itself they rebooted it yet again. Then following that, the UPS on my rack needed to be replaced, 2 more outages. 3 nights ago the PSU in my server blew, more downtime. Then yet again 2 nights ago I get this in my e-mail at 12:44 AM Dear Valued Customer, Hivelocity will be rearranging a portion of our datacenter tonight in order to better disperse power utilization. During this brief period of transition, your server/servers may experience brief disruptions in service. If you are not located in the affected portion of the datacenter, you should continue to experience the fast, reliable service you are accustomed to. If you are affected, then we appreciate your understanding and know that we will do our best to make the disruptions as brief as possible. Thank you again for your continued business and we wish you all the best. Thank you, Kevin Boyle Account Manager Now this would have been fine had they sent the e-mail at a normal time and not 12:44AM. But I did get the email, I called them asking how long the outage would be and also when I should shutdown the box to avoid SQL issues, The tech advised me that the server can be shutdown now, and that it woudn't be more then an hour until the work was complete. So I sat and waited. A little over 3 hours later the server was switched back on. Ever since that point I couldn't upload to the server at faster then 50KB/sec. I spent most of yesterday on the phone with them hearing that 50KB/sec is not that bad, finally I showed them my wget stats from thier own bandwidth test, which was local to thier network, and my box only got 500KB/sec to it. My computer at home on cable 2400 Miles away was getting 700KB/sec. Finally they said that there might be a switch problem, and the engineers would have to look into it tommorow. It took me from noon yesterday until 1AM for them to admit that there might be a problem. and 50KB/sec in my mind is not acceptable. Sorry for the rant Nitra
Posted by KCLexGS, 12-23-2004, 02:38 PM
HiVelocity, Any time estimate so we can set expectations with customers?
Posted by mbgrafix, 12-23-2004, 02:39 PM
I have had some upset clients contact me... in fact I think I may lose one over this as he is very upset. He is retail and it is Christmas and his site is down. Hivelocity needs to be prepared for stuff like this ESPECIALLY just before Christmas!!! I am not a lawyer so I do not know what kind of liability I have as a host, but I hope that noone can sue me over their lost sales! What is the ETA for repair?
Posted by lingates, 12-23-2004, 02:43 PM
Yeah it sucks being down. But despite today, since I've been a customer with them in October 04, their support is really awesome. I couldn't be happier. I just hope that this problem today get's fixed asap.
Posted by ceridius, 12-23-2004, 02:45 PM
Some Services should be restored within the hour, as for the main transformer replacement I am being told 6PM EST. The Main Core Network and DC2 should be back online as I said within the hour.
Posted by lingates, 12-23-2004, 02:48 PM
thanks for the update
Posted by ceridius, 12-23-2004, 02:50 PM
We appologize for the problem , this is not a common thing , the Main transformer has blown.
Posted by mbgrafix, 12-23-2004, 02:53 PM
Why isn't anyone at Hivelocity answering the phones? That is a BAAAD move! It does nothing to calm nerves or instill confidence.!
Posted by Kayvon, 12-23-2004, 03:07 PM
Please post a note when you guys are up and running so we can immediately inform our customers. Thanks
Posted by BadLeroy2000, 12-23-2004, 03:25 PM
It appears that service is being restored.
Posted by BadLeroy2000, 12-23-2004, 03:26 PM
Web Sites are back up - mail still having issues
Posted by odenx, 12-23-2004, 03:30 PM
I wouldnt start calling your customers back yet. My Server, as well as the HiVelocity.net websites still appears to be down.
Posted by lingates, 12-23-2004, 03:34 PM
yep, things are coming back up. My server is backup.
Posted by cddinc, 12-23-2004, 03:38 PM
My dedicated server is still down with them and i estimate atleast $1500 dollars damage in loss sales. Ofcourse I am in Canada so it went out very early in the AM while I had several pay per click engines sending me traffic that ended up nowhere. Just talked with the call center and we have had nothing but complaints because customers can't access the site. This is just plain redicoulous. I lost track of the amount of times I called them this morning and still no answer or return of call to let me know what was going on. It was only through this forum that I got an idea what was going on. Horrible support.
Posted by nitra, 12-23-2004, 03:39 PM
I too am from Canada and the losses for these continued outages I've had in not measurable.
Posted by mavherzog, 12-23-2004, 03:50 PM
I assume this is going to delay the build of my server I just purchased...
Posted by mavherzog, 12-23-2004, 03:56 PM
This is truly disheartening as I JUST purchased a dedicated server from them a couple of days ago and am waiting for it to be built. I can only hope that this is an abberation and that I will not experience outages such as this...
Posted by pl4tinum, 12-23-2004, 04:11 PM
i've read these forums for about 2 weeks and i can already tell you all that you should expect this kind of thing from just about ANY host, at just about ANY time. dont act all surprised even if youve been getting the best service imaginable for 15 years in a row. sh!t happens, yes it can happen to you. 100% uptime DOESNT EXIST
Posted by pl4tinum, 12-23-2004, 04:13 PM
you already wrote this in the other thread... im assuming where you wrote 500kb/sec is a typo also, considering if i was getting 500kb/sec i'd be pretty happy
Posted by mavherzog, 12-23-2004, 04:16 PM
Of course it does. MY concern is with the fact that there was such a disconnect between the customers and the hosting provider. Customers are FAR more tolerant if they know what is going on. Outages can and WILL happen. It's how you DEAL with outages that is the difference between a good and a GREAT company.
Posted by ceridius, 12-23-2004, 04:16 PM
I will be posting the official statement here within the next 15 min , all core network routing gear has been restored and all of DC and most of DC1 are back up, If your still having an issue please create a trouble ticket at http://support.hivelocity.net or use our livechat , as I am sure you also know our phones have been jammed , we have not been ignoring anyone , but this issue did also take out our phone system.
Posted by mavherzog, 12-23-2004, 04:18 PM
Are you making it your mission to jump down the throats of all those who are experiencing troubles? (if that is the case, you better quit your job as you will have your hands FULL here!)
Posted by mavherzog, 12-23-2004, 04:21 PM
Excellent. Thanks for the communications follow through.
Posted by ceridius, 12-23-2004, 04:23 PM
Wish this had not happened , we all feel like real S$%&T.
Posted by nitra, 12-23-2004, 04:28 PM
thats 500KB/sec to a machine internal to thier network. Not to the internet. but within thier own datacenter. That is a major problem. Nitra
Posted by Whocares, 12-23-2004, 04:47 PM
I think servers are again down...
Posted by nitra, 12-23-2004, 04:49 PM
Down again
Posted by lingates, 12-23-2004, 04:51 PM
down again
Posted by iSew, 12-23-2004, 04:53 PM
Ditto again.
Posted by Whocares, 12-23-2004, 04:54 PM
I have lost many customers. And I do not know what I will do.I am very confused and upset
Posted by KUNOFF, 12-23-2004, 05:01 PM
I just alerted all my clients! And then down we go again . . .
Posted by cddinc, 12-23-2004, 05:01 PM
This is just plain unacceptable. I consider myself to be a very understanding person but as I loose more and more sales to competitors with more reliable hosts in the busiest time of the year no less... ARRggg. Im too mad to type right now.
Posted by KCLexGS, 12-23-2004, 05:11 PM
EVERYTHING is down again...:-(
Posted by mavherzog, 12-23-2004, 05:19 PM
Things happen. If you aren't with a hosting provider that has a backup generator, you are partially at the mercy of their electrical company. Sure it sucks when something like that happens, but as someone else in this thread mentioned, "stuff" happens. As someone who just recently prepaid for over a years worth of HiVelocity service (15 months to be exact), it certainly doesn't give me a warm fuzzy to see something like this occur. However, I can certainly empathize with them and it is good to hear that they are working hard to get this resolved. Now, if they are STILL down tonight/tomorrow, then I might be a TAD more concerned.
Posted by cddinc, 12-23-2004, 05:28 PM
Its easy to say that when your not loosing thousands from the downtime.
Posted by ceridius, 12-23-2004, 05:29 PM
This has nothing what so ever to do with our generator or the power company. Just FYI , as I said previously I will be posting a full report once the issue is completely resolved.
Posted by KCLexGS, 12-23-2004, 05:31 PM
HiVelocity, I thought the power was up, what happened that it went down again?
Posted by ceridius, 12-23-2004, 05:32 PM
I am still waiting to find out exactly the issue , from the electricians , but all of DC-2 should be up.
Posted by mavherzog, 12-23-2004, 05:45 PM
Whoops. Sorry about that. Speaking from ignorance can certainly have its disadvantages.
Posted by KCLexGS, 12-23-2004, 05:55 PM
HiVelocity, what do you mean DC2 is up. I cannot access ANY of my sites nor your own web site, hivelocity.net! When will it be up?
Posted by nitra, 12-23-2004, 05:57 PM
They have 2 data centers DC1 and DC2, DC2 is online and never really went offline, my uptime on my box states that it never lost power. However from what i can tell the power is still out in most of DC1 which is where most of the core equipment is. Just my 2 cents. Nitra
Posted by ceridius, 12-23-2004, 05:59 PM
You are correct on that Nitra.
Posted by KCLexGS, 12-23-2004, 06:02 PM
Hi velocity, Ok, I guess all my sites are at DC1 then. 6pm ET is still the plan for DC1 to be up?
Posted by nitra, 12-23-2004, 06:08 PM
After speaking to Steve E. at HV I was advised to remove My credit card info from thier billing servers. I can't get to thier billing server to do that. So i messaged thier AIM account and got this. [17:05] nitra1234: Steve it's Matthew, I cant get to your servers to remove my CC information. please advise when you have more time [17:06] HiVelocityH: thanks for kicking us when we are down [17:07] HiVelocityH: and FYI this is not Steve [17:07] nitra1234: not kicking you [17:07] HiVelocityH: ummk [17:07] nitra1234: i spoke to steve he advised me to do that until we have further time to talk [17:07] nitra1234: i've had other issues then the current 1's [17:08] HiVelocityH: you very well know what the issue is and why you cant take your cc out [17:08] HiVelocityH: and you have also made a very good point to let everyone on WHT know we are having an issue. [17:09] HiVelocityH: Merry Christmas to you Nice..... Nitra
Posted by ceridius, 12-23-2004, 06:09 PM
Yes that is still a good ETA if not sooner.
Posted by ceridius, 12-23-2004, 06:15 PM
I guess we will have to put him at the top of the list to have his issue looked at as soon as all services are restored.
Posted by mavherzog, 12-23-2004, 06:19 PM
Perhaps not the most professional customer interaction...But you should at least be able to understand their frustration.
Posted by cbths1, 12-23-2004, 06:27 PM
I am sure the guys and gals at Hivelocity have taken a real beating today and I realize the frustration nitra and others are having. I have been with hivelocity for quite some time as a reseller and upgrade to dedicated server a few months back, and I can tell you I am sure they are doing everything in their power to get us back. I take my hat of to Steve, Shannon, Ben, Jay and especially my sales person Kevin B. They have proven in the past, at least to me, to do their best to put the customer first and resolve issues as quickly as possible. Just my 2 cents.
Posted by nitra, 12-23-2004, 06:34 PM
After I spoke to Steve E. I figured maybe there was a way for them to remove that information themselves from thier DB, hence the reason I contacted them on AIM. I really didn't deserve to have that attitude given to me. After all if it wasn't for us customers they wouldn't be in business. Nitra
Posted by KUNOFF, 12-23-2004, 06:35 PM
I agree with cbths1. You forgot Jason, Sam, Nick ... Anyway, I just hope they learn from this. Just as they did in getting rid of the spammers.
Posted by ceridius, 12-23-2004, 06:39 PM
I am just as frustrated as you are , I appologize , I had a temporary lapse of sanity.. I should not have said what I said and again I appologize.
Posted by ceridius, 12-23-2004, 06:43 PM
on a side note Nitra I think your issue might have something to do with a QOS setting on your switch to help prevent DDOS attacks , and we will be looking into it as soon as things stablize.
Posted by nitra, 12-23-2004, 06:44 PM
Whoever it was had no reason to act unprofessional towards me. This problem they are having is not my fault and they should not be directing any anger or other towards and clients. It was wrong and uncalled for. Nitra
Posted by Axodious, 12-23-2004, 06:48 PM
Well.... I was considering very much to buy a server at HiVelocity.... Up until this.... I read the entire post and figured they must be going through a bad time because I've heard good reviews in the past but after seeing this chat log I'm deffinently NOT going with HiVelocity.
Posted by KCLexGS, 12-23-2004, 06:49 PM
OK guys, back to the "fun" sufff HiVelocity, is the power going up in 12 minutes (6pm ET/DC1)?
Posted by nitra, 12-23-2004, 06:56 PM
And down again they go.
Posted by cddinc, 12-23-2004, 07:06 PM
any idea if we are going to be compensated for this? 6 months free hosting?
Posted by KCLexGS, 12-23-2004, 07:07 PM
HiVelocity, Customers are killing me......I need time estimate update.
Posted by ceridius, 12-23-2004, 07:13 PM
Last word , it should be any minute.
Posted by KCLexGS, 12-23-2004, 07:50 PM
Nothing yet..., any minute was 45 min ago....update?
Posted by ceridius, 12-23-2004, 07:54 PM
Now the electricians are saying 30 more minutes that puts us at 7:30PM
Posted by KCLexGS, 12-23-2004, 07:56 PM
Thanks for the update.
Posted by ceridius, 12-23-2004, 08:02 PM
I just wish it was better news.
Posted by KCLexGS, 12-23-2004, 08:10 PM
Well, at this point, any news about it being back up in the next 1-2hrs is good news. I know you guys work hard to get i fixed. Thanks again for the update, it helps to deal with the customers.
Posted by nitra, 12-23-2004, 08:18 PM
http://baynews9.com/content/36/2004/12/23/65939.html
Posted by cddinc, 12-23-2004, 08:20 PM
Im glad to see that your webdite is up and running, now what about ours!
Posted by odenx, 12-23-2004, 08:28 PM
Now I do understand that this outage really hurts, however so far I have had nothing but great service from HiVelocity, and will continue use them for my business. Slamming support or HiVelocity will not correct the issue and I am sure that after this is resolved, they will not only take measures to prevent this type of issues in the future, but will try to create another way of getting the message out to all customers so they do not have to find this info from a third party forum. Take care, odenX
Posted by ceridius, 12-23-2004, 08:31 PM
Truth is I was just about to email everyone when the power went out again. In this business you are not give much notice when such an outage will occur.
Posted by Steven, 12-23-2004, 08:39 PM
What no generator? What about ups?
Posted by mavherzog, 12-23-2004, 09:12 PM
Did you not read this entire thread?
Posted by zcwd, 12-23-2004, 09:14 PM
What's the latest HiVelocity?
Posted by KCLexGS, 12-23-2004, 09:15 PM
went out again? what's the issue now? ETA please.
Posted by zcwd, 12-23-2004, 09:18 PM
Anyone here have any new info to report... seems like we've running in cirlces all day.
Posted by KCLexGS, 12-23-2004, 09:24 PM
HiVelocity, update please.
Posted by KCLexGS, 12-23-2004, 09:38 PM
I hope silence dose not mean BAD news...
Posted by zcwd, 12-23-2004, 09:40 PM
HV's page is up now
Posted by KCLexGS, 12-23-2004, 09:43 PM
Their phone system is still down, and i cannot get any of their pages up = NO power :-(
Posted by odenx, 12-23-2004, 09:44 PM
Just spoke to Sandy B on there live chat, they said no update or ETA
Posted by zcwd, 12-23-2004, 09:47 PM
I spoke with Sam and he said things would be coming up soon. ?? Circles again.
Posted by vrstreet, 12-23-2004, 09:48 PM
HV is still down...we host 50 - 60 sites with HV, and this is killing us. We actually lost a client today, because we couldn't stay up for the holidays. Unless they can get back up (and stay back up) I'm going to lose more clients (not to mention losing potential clients because of the bad impression they get when our site is down.)
Posted by KCLexGS, 12-23-2004, 09:49 PM
We need to know if this thing will take another day. Some customers could have loaded their site somewhere else and changed their DNS and be online by now. We need to give customers some direction so they can make the right choice and minimize their losses. HiVelocity, update Please!
Posted by ceridius, 12-23-2004, 10:08 PM
Bascically it comes down to 20 batteries that we need to locate replacements for aftre they were blown this afternoon when we brought power back online, if we do not locate these batteries within the next 3 hours we will be relocating all servers in DC1 to DC2. I wish I had better news but this is the best I can offer.
Posted by mbgrafix, 12-23-2004, 10:15 PM
ONE OF MY CLIENTS IS PRETTY RIPPED.... it is very frustrating because when my clients ask, I really have nothing that I can tell them. I'm feeling pretty lame about now, like I've let my clients down big time! This could not have happened at a worse time!
Posted by zcwd, 12-23-2004, 10:17 PM
Why not have done that to start with and then put it back the other way when you had all the parts? Seems like a quick solid solutions for the customers sake at 10 this morning would have made all of our days easier. The solution paths today appear to be only in the interest of what's easiest for the datacenter and not what's the quickest solution to get paying customers back up. I appreciate the hard spot you guys are in but things seem always be handled the long way.
Posted by ceridius, 12-23-2004, 10:40 PM
Well we have located the batteries and they are being delivered and things will be back up in the next couple of hours.
Posted by nitra, 12-23-2004, 10:44 PM
Meanwhile my credit card gets billed again.
Posted by zcwd, 12-23-2004, 10:52 PM
Anyone know what caused the transformer problem today? Also, did anyone else get word from HV early today that this was a DOS attack?
Posted by cddinc, 12-23-2004, 10:54 PM
yah spoke with someone and they said the network was "hit" from an attack. No mention of transformer being down.
Posted by zcwd, 12-23-2004, 10:55 PM
Suppose there's any truth to that?
Posted by nitra, 12-23-2004, 10:56 PM
Guys there _was_ a fire in thier building. Several hosting companies had this issue. http://baynews9.com/content/36/2004/12/23/65939.html There is proof, that is thier address.
Posted by zcwd, 12-23-2004, 11:00 PM
Yes I saw your earlier post. (great work finding that by the way - I had a hunch something else was up) I am just evaluating the communication I have received from them today. If I was led to believe this was a DOS attack early in the day... why? To cover this up? How bad it realy was?
Posted by cddinc, 12-23-2004, 11:02 PM
Seems kind of silly to cover up. I must admit I would rather hear the transformer blew (one time event) rather than someone attacking the network (who knows how often this could happen).
Posted by zcwd, 12-23-2004, 11:06 PM
Agreed. I don't like to think I was mislead. I would like to hear the "real" story right away so I can tell my customers whats up and stand behind 100%. I also would like to know why the transformer went... you are correct in the assumption that they don't go very often and when they do go usually its something like a lightning strike or major electrical mishap.
Posted by zcwd, 12-23-2004, 11:15 PM
Here's a better article that answers some questions.... http://news.tbo.com/news/MGBJ41XR33E.html
Posted by ABCStore, 12-23-2004, 11:46 PM
Quote from the site - What batteries? Diesel generator does not need any... ABC P.S. I'm colocating at whatever DC and have had my server rebooted or just shut down countless times... P.P.S. my first hop: Pinging 69.46.23.1 with 32 bytes of data: Reply from 69.46.31.77: TTL expired in transit. Reply from 69.46.31.77: TTL expired in transit. Reply from 69.46.31.77: TTL expired in transit. Reply from 69.46.31.77: TTL expired in transit. as of 10:48PM EST
Posted by KCLexGS, 12-23-2004, 11:54 PM
HiVelocity, any updates? Are the batteries being installed already?
Posted by nitra, 12-23-2004, 11:56 PM
As someone who has worked for many years in a data center, here's how it works, power comes into the building, goes to the transformers then comes into the space where it then goes into the batteries, from the batteries to the equipment. When it's on diesel it follows the path, generator -> batteries-> equipment. Hope this helps you nitra
Posted by ceridius, 12-24-2004, 12:06 AM
Yes the batteries are being installed and energized as we speak , I do not have an exact ETA , but things are getting closer to where they were this after noon when we came back online completely for 90 min.
Posted by KCLexGS, 12-24-2004, 12:28 AM
OK, HV site is up, now what? Are all the server coming up now?
Posted by zcwd, 12-24-2004, 12:32 AM
HV's site has been for over an hour now
Posted by cbths1, 12-24-2004, 12:41 AM
HV site is back up? I guess it went back down again.
Posted by dicwsl, 12-24-2004, 12:42 AM
HV site is up but servers are down since last 14 hours. 13 Hours before Ive asked HV what was the trouble and any estimated time, they told me that was a network failure and that it will be sorted in 1 hour approx. Name it as you like, but no more lies, no more confusing info. If you cant set it up tell us, because Im losing more customers due to wrong info than the failure. This is not serious, not only by the down time, its not a serious company management, here is a copy of what HV posted about other server downtime troubles. -------------------------------------------------------------- That kind of downtime with no response or explanation is not acceptable. Regardless of what the problem is or whether it is going to fixed, they should have given you some type of reply or system wide notification as to what is going on. I hope that you have backups and your clients have backups of their own data, since it doesn't appear that you are going to be getting any answers from Mongo Hosting. Good Luck! _________________ - HiVelocity Hosting
Posted by zcwd, 12-24-2004, 01:01 AM
I have to agree. I can't tell you how many times I have heard "...in an hour" "In a few minutes" I even heard at one point from Sam today "It will be up by 6 EST" and at the time it was 5:56 EST! That has been a consistent issue for me as well. Give me real info - not a guess, a story, or spin. Straight talk works with our customers just the way it does with us... not a very wild concept.
Posted by zcwd, 12-24-2004, 01:06 AM
HiVelocity do you have any updates?
Posted by KCLexGS, 12-24-2004, 01:07 AM
HiVelocity, Update please?
Posted by hlesesne, 12-24-2004, 01:07 AM
We had things go down on hivelocity as well and then they were back up for a bit and then offline again... I did get a chance to chat with network support and the following was their reply: Hal: I was wondering if you guys were still having power issues? I am not able to get to any of our colocated servers - I think I can get to one of the dedicated servers we have with you, but none that are in our colo rack Bill V.: Transformer here has blown up. The main transformer that converts the high voltage from the power main to lower voltage that feeds the datacenter blew up.We have already installed a new transformer and more than half the datacentre is coming online and rest all the servers will be online very soon.We brought everything back up this afternoon after the transformer was replaced but then the batteries blewthe replacement batteries are being installed and energized & so we are going to power everything back up slowly once they are fully energized, so servers will be back online soon. I have never known HiVel to lie about any network problems and we have been hosting with them for more than 1 year. I trust that they team is doing everything they can to remedy the problem and will update if I find out differently. Shame to hear about the fire in the building (blown tranformer). I have been to the datacenter there and it must be bad considering the amount of precautions they seem to have... We endured hurricanes Charley and Francis down there without a blip... Anyway - my 2 cents are still in support of Hivelocity and their tireless support. Hal
Posted by zcwd, 12-24-2004, 01:14 AM
I don't think anyone is questioning HV's work on the actual problem. The issue for many of us is the handling of it before the started using the description you mentioned. That was and is the problem and has real factors with real amounts of time involved... something I can pass along to a client and they can relate to the magnitude. Early in the day this was not treated with detail it needed. 9:30 this morning was the time to give us the detail you mentioned not 7:30pm.
Posted by zcwd, 12-24-2004, 01:19 AM
HiVelocity any updates to pass along?
Posted by dicwsl, 12-24-2004, 01:22 AM
Well man, Im not saying HV are liars, every person can discover the word by himselft, Im just saying that as they judged other server lack of info, I ask them for real info... and Im going to explain it in 3 steps: 1) 1 hour of failure --- network problems, 1 hour estimated 2) 10 hours failure (chat works because its on other server) --- power failures --- 2 hour estimated --- same text of your post 3) Now, almost 15 hours --- my server doesnt work Im sure you get the point... and... Im not going to post anymore, Im not trying to start a discussion related to HV's ethics, this is what I feel when confidence is broken between a customer and a provider... just trying to ask HV for a real estimated time, so I can tell my customers a real time... but pls, dont want to hear again that it will be fixed in 1 hour
Posted by hlesesne, 12-24-2004, 01:23 AM
Understood. It would have been nice to know the magnatude of the problem from the beginning, but based on my previous experience with severe network trauma such as this - ETAs with any standard devation of more than 24 hours are pretty pointless. I can empathize with updates like "3 hours till the transformer is reinstalled"... once done "uh ohh 6 more hours for the unaccounted for recharging of the dc supply", etc... etc... I think that it was probably simply impossible to judge the amount of time it would take given the magnatude of the problem. Perhaps going forward some network status page off the actual Hivel network with updated etas, etc... would be exteremly beneficual - praying of course that it never has to be used. I am trying to picture the scene down in Tampa with smoke and no power, and having to figure out how to field the call center when more then likely the place was evac'd... Again my 2 cents which are probably worth less than that... I'm just glad things seem to be coming back up. I don't have anything to gain by lending my support of the hivel team, I just kinda feel for them right now... Christmas eve is probably not shaping up quite like they planned :-(. Hal
Posted by dicwsl, 12-24-2004, 01:35 AM
I know that I said I was not going to post, but... its better to write than wait Well, here we was exactly the same trouble and it was solved in 3-4 hours... estimated times are related to estimated, every body will understand a deviation, but ... you know, too many hours. BTW, Ive to agree that you are right, things seems to be going better as HV site is still up and that is a good new. Would be really good any feed back from HV like "things are going better" or "damn, not again, sorry"... it's 6:30 am here and would like to go to sleep and have a nice sleeping Ping ?... pong !
Posted by lingates, 12-24-2004, 01:38 AM
hey dicwsl, hivelocity.net is up for you?
Posted by zcwd, 12-24-2004, 01:39 AM
HiVelocity any updates to pass along?
Posted by dicwsl, 12-24-2004, 01:51 AM
yes, it is, remember it is hivelocity.net as .com doesnt work have talked with support right now and told me the same story, and that it will be fixed in 2-3 hours, I asked them for the truth, not a false ETA and they answered 4-5 hours so... go to sleep a bit, call your gf, dont drink coffe and pray a lot... good luck and happy Christmas
Posted by nitra, 12-24-2004, 02:16 AM
Redundancy- Built from the ground up utilizing all the latest redundancy features like: Diesel Generators, DC power to provide conditioned power to the IP floor. Our redundant Cisco routers utilize BGP4 protocol to provide redundant service through our multiple providers. All of our servers utilize RAID for maximum data availability. All systems are backed up through a daily, weekly and incremental offsite- backup system. Guess it wasn't built from the ground up very well. Nitra
Posted by nitra, 12-24-2004, 03:21 AM
[02:18] nitra1234: you guys going to talk to me or what? [02:22] HiVelocityH: what would you like me to say? [02:22] HiVelocityH: I do not have a new ETA [02:22] HiVelocityH: we have 15 electricians working on it
Posted by chet, 12-24-2004, 04:06 AM
What, the "ex-spamming" hivelocity isn't being honest or being a good hosting company??!?! Surprise, surprise. You give second chances to ability, not ethics. Last edited by chet; 12-24-2004 at 04:15 AM.
Posted by anon-e-mouse, 12-24-2004, 04:52 AM
nitra, I can understand you and others are frustrated, but expecting an update every five minutes and hassling them on IM won't make things happen any quicker.
Posted by nitra, 12-24-2004, 05:04 AM
I am not looking for an update every 5 minutes. I am looking for someone to talk to me so my credit card dosent get billed tonight. I have contacted then 3 times today. only 1 of those calls did i get treated with respect.
Posted by Whocares, 12-24-2004, 06:19 AM
I think servers are online now.
Posted by ceridius, 12-24-2004, 06:21 AM
Yes most servers should be coming back online.
Posted by Whocares, 12-24-2004, 06:23 AM
Again same problem will occur ?
Posted by ceridius, 12-24-2004, 06:27 AM
I want to say no , however please allow us some time to be sure before we say that. Better be safe than sorry.
Posted by AndreyLevshin, 12-24-2004, 09:24 AM
I've lost server by this issue. It is not started by kernel panic error! It is not first issue which end with the same result. I very disappointed of your service!
Posted by cddinc, 12-24-2004, 12:32 PM
our .net websites kind of choked on the windows server aswell, so if anyone has a .net application that got messed by the powerout then the easy fix is to delete all dll's and recompile the original application. AndreyLevshin -> Good luck with the panic error, that doesnt sound like a good thing
Posted by Kayvon, 12-24-2004, 12:36 PM
Could you guys (HV, Resellers) help me prepare a statement for the customers. I think we must now do damage control and gain customers trust that this was an unfortunate event and that they can allow us to continue host their most precious technology which is their Emails, website, Database, and online shopping. Please post suggestions, Thanks
Posted by ceridius, 12-24-2004, 12:39 PM
Well we do not have statement fully prepared but I will past a copy of my rough draft , maybe you can use some or all of it Hello Valued Hivelocity Customers, We deeply apologize for the outage of service yesterday. The outage was due to our Main DC transformer blowing. This is not due to a failure of our generator or battery power , it was the main transformer that converts the high voltage from the power main to lower voltage that feeds the datacenter floor. We have installed a new transformer and will be installing a larger transformer in the comming weeks. ETA for delivery on this component is 3 weeks. We know you were calling our phones during this time and getting no response. We apologize for the frustration. Unfortunately with no power in the office our phones were not ringing. Everything is coming back up now. If you are still experiencing any issues please contact us immediately. We are able to answer chat, email and phones at this time. We hope you have a very happy and safe Holiday. Please contact your account manager if you would like to discuss this in more detail
Posted by Kayvon, 12-24-2004, 12:44 PM
I wanted to say that I feel so much better that we are back up and running.... I had this choking feeling when we were down, I guess it comes from the fact that there is nothing I can do when things are in someone else's hands. I need a shrink...
Posted by ceridius, 12-24-2004, 12:47 PM
We all get that same feeling when something like that occurs.
Posted by Kayvon, 12-24-2004, 12:50 PM
Thanks, I bet you guys clocked some overtime, hah...
Posted by ceridius, 12-24-2004, 12:50 PM
LOL the clock is still running.
Posted by ceridius, 12-24-2004, 01:40 PM
On a side note , since we are obviously in violation of our uptime guarantee, all clients that were effected by this outage are entitled to a credit on their account. Please feel free to email sales@hivelocity.net and they will be happy to take care of that for you , please include the following information 1.) Name 2.) Account ID 3.) Best number and time to contact you. Thank you all very much for being clients of HiVelocity.net and we look forward to putting this incident behind us.
Posted by lingates, 12-24-2004, 01:46 PM
Hey, I think Hivelocity took care of things very well. I mean Nitra give Hivelocity here a break. I mean I know sales and everything were affected heavily etc, etc, but crap happens. For the thing that happened to Hivelocity, I think they handled the situation as best as possible and they updated us on everything that was happening. Have a great holidays everyone!
Posted by chet, 12-24-2004, 05:03 PM
Having a single point of failure in a system that you tout as being built from the ground up as being redundant? That is handling it well? To quote from their site:
Posted by dollabillz, 12-24-2004, 07:52 PM
"lingates" must be blind. Does everyone miss the fact that Hivelocity are fasle advertising and lying to us?? "Built from the ground up" "Deisel generators" WTF Can Hivelocity please address your false claims to this board?? Hello?
Posted by hlesesne, 12-25-2004, 12:27 AM
They do have cats for backup power and have used them from time to time... One case being in either Francis or Charley this year when Tampa cut power on purpose to prevent fires. They kicked in before any battery systems went offline. I think that the problem here was actually with the transformer or at least the step downs, not the backup systems itself... I suppose the solution could have been redundancy in those system, but I have never seen a dc with redundancy including multiple internal feeds for power, most are just grided to allow for multiple services to different areas of the dc, which it sounds like was still the case here... My knowledge of this is lacking and I am no electrician, but to call it false advertising I think would be taking it a bit far. I truely do believe HV runs a first class datacenter and could have supplied connectivity and power to the actual dc till the cows come home. They just suffered a catastrophe and I feel are fairing pretty well considering. Thanks, Hal
Posted by Kayvon, 12-28-2004, 09:44 PM
Looks like they are down again since 5pst today. Their site is offline...
Posted by neterizon, 12-28-2004, 10:02 PM
actually it was 6:50PM EST switched off and on every few minutes before that
Posted by Kayvon, 12-28-2004, 10:54 PM
They won't be able to keep satisfied customers if it goes like this for 3 more weeks...
Posted by ABCStore, 12-28-2004, 11:36 PM
Does this look familiar to you guys? Personally I'm sick and tired of seeing this on my FreeBSD server. If this happens again within 1 week I'm moving my equipment somewhere else...
Posted by KUNOFF, 01-25-2005, 04:06 PM
Here we go again. Down for a while now . . .
Posted by AndreyLevshin, 01-25-2005, 04:07 PM
Since 18:40 GMT Hivelocity.net completely unpingable. Again?
Posted by cathead, 01-25-2005, 04:13 PM
I also have no connectivity to my dedicated machine hosted with Hivelocity... has anyone heard anything from the company?
Posted by ABCStore, 01-25-2005, 04:31 PM
Neither have I. Too much headache lately. =(
Posted by europac, 01-25-2005, 04:56 PM
Seems to be becoming a common place happening - down on all systems for 2 hours this afternoon. Does anyone know of a host that stays up?
Posted by europac, 01-25-2005, 04:59 PM
Looks like it just came on again at 3.56PM EST - finally
Posted by Reseller-Center, 01-25-2005, 05:58 PM
Seems they don't have an explanation, or anything posted yet. http://www.hivelocity.net/Network/status.asp More than likely a routing hardware failure i would suspect.