Hivelocity Outage [merged]

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Hivelocity Outage [merged]

Posted by datingsoftwares, 10-13-2009, 09:08 PM
Server with Hivelocity was down 2 hours last night. Their helpdesk was down too! When it's up, one of the hard drives is missing. All websites are down because database failure due to the drive. My clients are very upset.

I have been using Hivelocity for years and the uptime is very inconsistent. Last months, my connection to my hub was disconnected without notice. I asked them and they told me I don't have any hub. This caused my database to go own because the local network ip changed. My friend with Hivelocity have same bad experience. I recommended him to Hivelocity years ago. Now I feel bad. I am also using softlayer which is lot better. But it's hard to move everything.

Now there's no one answering live chat
Posted by datingsoftwares, 10-13-2009, 11:11 PM
After 2.5 hours, still nobody's checking. I can see it via remote desktop not sure if this is how they treat Critical ticket. All sites remain down.
Posted by maithily, 10-14-2009, 12:43 AM
My server with hivelocity is also down for last 17 hours. I was informed that the hdd is down and advised to reload os. Even 9 hours after lodging the ticket to reload os, nothing has been done, no information, online support has no information. They say that the support ticket would give me reply.

I have sent a mail to HivelocityGM to but no reply till now...

My all the sites are down.

Don't know what to do...
Posted by datingsoftwares, 10-14-2009, 01:50 AM
Your case is even worst. Finally my server is up after 5.5 hours. I still appreciate their hard work. I will try to move out although it's a bit tough with all the ips, dns, files, and settings, and hope it won't happen again before that.
Posted by helloashok, 10-14-2009, 03:42 AM
I am unable to access one of my server in hivelocity as well hivelocity.net site, many of our customers are also complaining of site down in this server.

However I checked through proxy sites are up.

Anyone else facing problem in connecting to hivelocity server.
Posted by Scott.Mc, 10-14-2009, 04:02 AM
We also noticed a connectivity issue which just recovered a few minutes ago.
Posted by HivelocityGM, 10-14-2009, 11:00 AM

Quote:

Originally Posted by maithily

My server with hivelocity is also down for last 17 hours. I was informed that the hdd is down and advised to reload os. Even 9 hours after lodging the ticket to reload os, nothing has been done, no information, online support has no information. They say that the support ticket would give me reply.

I have sent a mail to HivelocityGM to but no reply till now...

My all the sites are down.

Don't know what to do...

I have looked through all the tickets and there is only one with your situation and the tickets are replied to. I have replied to any and all emails sent to me.
Posted by datingsoftwares, 10-15-2009, 10:39 AM
We provide application hosting and with the nature of this business, we always receive complaints. It's just how you deal with complaints and resolve issues.
Posted by TFArchive, 10-15-2009, 11:06 AM
Looks like they are down again. At 11:01 my ssh stopped responding and I"m getting 90% packet loss from my home connection and from my dormant SL server.
Posted by Scott.Mc, 10-15-2009, 11:07 AM
Yeah they are down again, third day in a row. This time it's effecting a large portion though.
Posted by twhiting9275, 10-15-2009, 11:11 AM
Definitely latent to the point of being down. Traceroutes FWIW , from inside the server, though it is taking a LOT of time to go through (due to latency):

Quote:

traceroute to google.com (74.125.53.100), 30 hops max, 40 byte packets
1 74.50.117.1 (74.50.117.1) 1.100 ms 2.952 ms 3.052 ms
2 69.46.31.97 (69.46.31.97) 11.988 ms * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 *
[Load: 1.90] [Time: 10:07:26]
traceroute microsoft.com
traceroute to microsoft.com (207.46.197.32), 30 hops max, 40 byte packets
1 74.50.117.1 (74.50.117.1) 2.081 ms 2.329 ms 4.045 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

Posted by VisiGod, 10-15-2009, 11:15 AM
I cannot reach them also
Posted by HivelocityGM, 10-15-2009, 11:18 AM

Quote:

Originally Posted by datingsoftwares

We provide application hosting and with the nature of this business, we always receive complaints. It's just how you deal with complaints and resolve issues.

As I mentioned in a previous post. The majority of the folks who have recently complained here about getting their ips null-routed or there servers pulled offline fit into 1 of 3 categories.

1. They blatantly abused our network and made no effort to correct it.
2. They blatantly abused our network and the only efforts they made were to and hide the abuse by moving things to different ips.
3. They simply lack the skill or effort it takes to manage a server and let their servers abuse perhaps without even knowing what was going on. But we certainly cant let ignorance be a reason for letting someone repeatedly spam.

That being said, we are not perfect and we have already and will continue to review our abuse policies and procedures. The last thing we want is to adversely affect someones business who is not blatantly abusing the network and willing to react accordingly and timely.

However, if someone purchases a self-managed server then they are accepting responsibility for managing that server, ensuring bad things are not going on and promptly responding to abuse issues. Not knowing how to manage a server, laziness and vacation are not excuses to let things run amuck.

I want to reiterate, we are reviewing our abuse procedures. Everyone deserves a fair amount of time to resolve an issue.
Posted by HivelocityGM, 10-15-2009, 11:21 AM
Hey Everyone, we are working on it. Something is going on with one of our routers. Give me a few minutes for an update.
Posted by petteyg359, 10-15-2009, 11:26 AM
Must be an important router, since it seems to have taken out _everything_ Thanks for update.
Posted by Arsenico, 10-15-2009, 11:37 AM
how much long will this take?
Posted by HivelocityGM, 10-15-2009, 11:47 AM
Everything should be back up now, just maybe a little slow. Everything will be back to normal in just a few minutes.
Posted by HivelocityGM, 10-15-2009, 11:55 AM
Everything should be back to normal now.
Posted by c0bra, 10-15-2009, 12:09 PM
It's not back to normal at all. We are seeing connectivity errors all over the place.

I cannot send backups to bqbackup the connection keeps dropping and timing out. I have just attempted a third rsync and once again the connection has frozen after 200mb of data.

Absolutely useless.
Posted by HivelocityGM, 10-15-2009, 12:12 PM

Quote:

Originally Posted by c0bra

It's not back to normal at all. We are seeing connectivity errors all over the place.

I cannot send backups to bqbackup the connection keeps dropping and timing out. I have just attempted a third rsync and once again the connection has frozen after 200mb of data.

Absolutely useless.

Send me a PM with your ip or contact information. I have a lot of people telling me and my techs all is back to normal. If you are not experiencing the same we need to check it out.
Posted by mainstreamnet, 10-15-2009, 12:49 PM
Still down for us. We have 1 server there which I think we are going to move later tonight. Way to much downtime lately.
Posted by HivelocityGM, 10-15-2009, 12:57 PM

Quote:

Originally Posted by mainstreamnet

Still down for us. We have 1 server there which I think we are going to move later tonight. Way to much downtime lately.

PM me your ip so I can get it looked into immediately.
Posted by mainstreamnet, 10-15-2009, 01:05 PM
Just came up a few minutes ago.
Posted by remotay, 10-15-2009, 03:18 PM
So frustrated with Hivelocity at the moment.

At first my server went down and I was told to run a Tracert because I couldn't access the WHM - and the tracert revealed routing issues on hivelocity's end. I submitted a ticket marked critical and after 20 minutes it was fixed.

5 minutes later the same exact issue - the same routing issue and my site goes down again.

It gets fixed in another 10 minutes and right around now it's 5:50 AM and I haven't slept yet so I tell the same support guy in the ticket I'll be heading out and I hope this issue doesn't happen again [Btw, I pay for 24x7 Ping monitoring on their silver managed service, so I figure if anything happens they'll at least reboot the server or look into it]

I wake up at 2PM and BAM my server is once again down - likely down for at least 4 hours. I reply to my critical ticket and no response in 45 mins so far. I was told by live chat they were looking into it. This is ridiculous! What's the point of paying for the 24/7 monitoring if they don't even do anything until I bring the issue to them?

Update; Over an hour on my critical support ticket. The previous 'techs are working on it' reply was from a live chat technician and still no luck.
To be fair though - I haven't had any issues with Hivelocity before this incident. My server is still down as I write this so I'm still extremely frustrated.
Posted by remotay, 10-15-2009, 03:56 PM
Now the live support button gets an error message. So frustrated at the moment and considering moving hosts if Hivelocity can't rectify this situation.
Posted by spiderxp, 10-15-2009, 04:02 PM

Quote:

Originally Posted by remotay

Now the live support button gets an error message. So frustrated at the moment and considering moving hosts if Hivelocity can't rectify this situation.

can i know plz your server specification
Posted by HivelocityGM, 10-15-2009, 06:25 PM

Quote:

Originally Posted by remotay

So frustrated with Hivelocity at the moment.

At first my server went down and I was told to run a Tracert because I couldn't access the WHM - and the tracert revealed routing issues on hivelocity's end. I submitted a ticket marked critical and after 20 minutes it was fixed.

5 minutes later the same exact issue - the same routing issue and my site goes down again.

It gets fixed in another 10 minutes and right around now it's 5:50 AM and I haven't slept yet so I tell the same support guy in the ticket I'll be heading out and I hope this issue doesn't happen again [Btw, I pay for 24x7 Ping monitoring on their silver managed service, so I figure if anything happens they'll at least reboot the server or look into it]

I wake up at 2PM and BAM my server is once again down - likely down for at least 4 hours. I reply to my critical ticket and no response in 45 mins so far. I was told by live chat they were looking into it. This is ridiculous! What's the point of paying for the 24/7 monitoring if they don't even do anything until I bring the issue to them?

Update; Over an hour on my critical support ticket. The previous 'techs are working on it' reply was from a live chat technician and still no luck.
To be fair though - I haven't had any issues with Hivelocity before this incident. My server is still down as I write this so I'm still extremely frustrated.

If you are still with any issue please email me directly. We did have a routing issue this morning for about 45 minutes due to a problem with one of the new router cards installed in preparation of Level 3. Everything was back to normal hours ago though. If you are still with an issue I would like to look into it personally. Please email me asap.
Posted by Scott.Mc, 10-15-2009, 06:45 PM

Quote:

Originally Posted by Arsenico

Yesterday a power issue, and today a network issue

The power issue caused lot of HDD failures and filesystems too
For example, one of my servers were down for almost 2 hours

Today a network issue (they said a router failed) made the whole network go down
Maybe, the techs are still working on the ISSUE, don't really know.

Was there a power issue yesterday? One of our systems went down and we assumed they rebooted it from the tickets. Normally it's relating to a switch issue which happens every other month without fail and we assumed they just rebooted it and fixed the switch issue afterwords but it could have been a power issue as that makes sense but they never said anything.
Posted by Arsenico, 10-15-2009, 06:57 PM

Quote:

Originally Posted by Scott.Mc

Was there a power issue yesterday? One of our systems went down and we assumed they rebooted it from the tickets. Normally it's relating to a switch issue which happens every other month without fail and we assumed they just rebooted it and fixed the switch issue afterwords but it could have been a power issue as that makes sense but they never said anything.

Sorry, before yesterday the monday: http://forums.hivelocity.net/network...-circuits.html

I don't know the day i live.
Yesterday, no problems, 0 troubles.

I'm still with HV, and i will, since i hope they'll fix every single problem.

2 of my servers rebooted too
Posted by stablehost, 10-18-2009, 02:01 PM
Once again, packet lost to my gateway.

This is getting old.

--- 96.31.94.129 ping statistics ---
34 packets transmitted, 23 received, 32% packet loss, time 33037ms
rtt min/avg/max/mdev = 5.573/19.062/248.514/49.420 ms
Posted by cristibighea, 10-18-2009, 02:06 PM
Responding over here, just a small service interruption?
Posted by Arsenico, 10-18-2009, 02:12 PM
i'm only experiencing problems on servers with ip 9x.xxx.xxx not others.

Ping statistics for 96.31.94.129:
Packets: Sent = 200, Received = 153, Lost = 47 (23% loss),
Approximate round trip times in milli-seconds:
Minimum = 190ms, Maximum = 353ms, Average = 218ms
Posted by mainstreamnet, 10-18-2009, 02:15 PM
Their site is down also. Going to cancel our last server with them as soon as their site is up.
Posted by spiderxp, 10-18-2009, 02:23 PM
Hivelocity again

im sorry to hear that i hope this company upgrade every thing and first service
Posted by stablehost, 10-18-2009, 02:23 PM
I've been with them many years, it has been going downhill fast

I put in a support ticket about an hour ago about it, no replies yet.

This is the second time in 2 weeks this has happened.
Posted by stablehost, 10-18-2009, 02:28 PM
Odd, now I can't even reach support.hivelocity.net.

Gonna be a good day
Posted by remotay, 10-18-2009, 02:30 PM
I Can't access my server or hivelocity.net homepage. Pretty ridiculous as I had this same issue just a few days ago and told them about it.

Tracert reveals that 69.46.31.98 is timing out which is on hivelocity's end.

Maybe I'm jumping the gun here as after I called they said they were working on it and it should be done within the next 10 minutes.
Posted by jordanriane, 10-18-2009, 02:34 PM
It's all down here for me (Chicago.) My servers are inaccessible with them and their main website is as well.

They seem to always have major issues with routing; Last time I showed them proof I was told it was temporary and resolved. Pfft.
Posted by Arsenico, 10-18-2009, 02:37 PM
nice, everything is down right now
Posted by petteyg359, 10-18-2009, 02:38 PM
I thought they fixed this yesterday (or day before?)... I'm in the middle of coding and my SSH suddenly hangs...
Posted by mistical, 10-18-2009, 02:44 PM
It's down for me as well.
Posted by mainstreamnet, 10-18-2009, 02:45 PM
Their site is up now.
Posted by stad021, 10-18-2009, 02:46 PM
Routing stuff is a little greek to me but here's what I see from 3 locations:

WORKING:

...
4 cr2.hs1tx.ip.att.net (12.123.212.214) 14.787 ms
5 cr1.dlstx.ip.att.net (12.122.28.157) 14.149 ms
6 ggr3.dlstx.ip.att.net (12.122.139.25) 13.272 ms
7 192.205.35.142 (192.205.35.142) 13.359 ms
8 ae-81-80.ebr1.Dallas1.Level3.net (4.69.145.178) 19.558 ms
9 ae-1-12.bar2.Tampa1.Level3.net (4.69.137.117) 62.042 ms
10 ae-0-11.bar1.Tampa1.Level3.net (4.69.137.109) 63.264 ms
11 HIVELOCITY.bar1.Tampa1.Level3.net (4.53.172.2) 58.902 ms
12 * 69.46.31.98 (69.46.31.98) 58.331 ms
13 xxxxx.xxx (66.xxx.xxx.xxx) 57.436 ms

WORKING:

...
3 209.220.16.125.ptr.us.xo.net (209.220.16.125) 12.958 ms
4 206.111.13.54.ptr.us.xo.net (206.111.13.54) 12.084 ms
5 WBS-CONNECT-LLC.ae0.409.ar2.TPA1.gblx.net (64.214.147.222) 41.249 ms
6 69.46.31.98 (69.46.31.98) 36.144 ms
7 xxxxx.xxx (66.xxx.xxx.xxx) 38.927 ms

FAILING:

...
6 56 ms po10-0.core01.ord03.atlas.cogentco.com [154.54.10.125]
7 58 ms te3-1.mpd01.ord03.atlas.cogentco.com [154.54.3.234]
8 60 ms vl3488.ccr01.ord01.atlas.cogentco.com [154.54.5.25]
9 95 ms te4-7.ccr01.atl01.atlas.cogentco.com [154.54.29.93]
10 97 ms te4-2.ccr01.jax01.atlas.cogentco.com [154.54.29.130]
11 119 ms te4-3.ccr01.tpa01.atlas.cogentco.com [154.54.29.138]
12 109 ms HiVelocityVenturesCorporation.demarc.cogentco.com [38.112.240.54]
13 101 ms 69.46.31.98
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.

Unfortunately, the third failing location is my home office so I can't get there from here, but my server is up and I am logged in with no interruptions from one of the first two working locations.

Anyone with better networking understanding know why the third traceroute would stop at 69.46.31.98 while the other two go past?

Thanks
Posted by stablehost, 10-18-2009, 02:55 PM
Our server is back now too.
Posted by EGC-Carlos, 10-18-2009, 02:56 PM
I have a dedicated server with them and the IP starts with a 9 and I'm able to ping it just fine with no packet loss. Sorry to hear so many bad reviews, I for one have been with them for about 8 months and so far so good, couple of issues here and there but nothing serious.
Posted by stad021, 10-18-2009, 03:05 PM
I can also reach from my own location again. Interesting I guess, now the last hop from all three is "trans3.hivelocity.net (69.46.31.106)" instead of 69.46.31.98.
Posted by Arsenico, 10-18-2009, 03:16 PM
<<snipped>>

i'm not getting 20-30% packet lost right now

Ping statistics for 96.31.94.xxx:
Packets: Sent = 200, Received = 198, Lost = 2 (1% loss),
Approximate round trip times in milli-seconds:
Minimum = 195ms, Maximum = 663ms, Average = 221ms

(170-200 ms is the normal for my country)
Posted by HivelocityGM, 10-18-2009, 03:20 PM
Everything should be back to normal now. Sorry for the issue everyone, we just had a problem with one of our routers. We added a new router module a couple of weeks ago in preparation for Level 3 being turned up. For 2 weeks we did not have any problems with it. We turned up Level 3 a few days ago and the module has has given us a few issues since. My have replacement modules and the old module that was working fine so whatever is causing the problem we will get it fixed once and for all today. If we have to replace anything on the router you will not be affected.
Posted by Arsenico, 10-18-2009, 03:30 PM

Quote:

Originally Posted by HivelocityGM

Everything should be back to normal now. Sorry for the issue everyone, we just had a problem with one of our routers. We added a new router module a couple of weeks ago in preparation for Level 3 being turned up. For 2 weeks we did not have any problems with it. We turned up Level 3 a few days ago and the module has has given us a few issues since. My have replacement modules and the old module that was working fine so whatever is causing the problem we will get it fixed once and for all today. If we have to replace anything on the router you will not be affected.

Please Steve, fix it A.S.A.P, i got like 8 SMS from my monitors, a few sms from customers and another Calls from other customers.

Posted by sirius, 10-19-2009, 09:47 AM
Thread cleaned.

Just a reminder....

Quote:

Originally Posted by Outage Forum Rules

This forum is provided to discuss current outage issues and to allow a way for customers and providers to communicate. Comments by non-customers will be removed. Reviews are only suitable in the appropriate main forum category.

Sirius