BuyVM Down?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > BuyVM Down?

Posted by WWS2010, 05-06-2011, 06:35 AM
Hi,

Anyone other noticed that buyvm is down? For me (from the Netherlands) i cant reach my VPS (205.185.126.x) nor manage.buyvm.net (72.52.127.128).

Does anyone have more info on this? Thanks!
Posted by cpservr, 05-06-2011, 06:42 AM
it's HE fremont facility, not only buyvm
Posted by JonFatino, 05-06-2011, 06:49 AM
HE FMT1
EGI
BuyVM
Frantech

All are down.
Posted by Trix, 05-06-2011, 06:57 AM
Do we know if it's a power or networking issue?
Posted by shearerc, 05-06-2011, 06:59 AM
All HE sites inaccesible at the moment.

I checked twitter of frantech, henet, egihosting - all no mention of what's going on at time of this post.
Posted by cpservr, 05-06-2011, 07:00 AM
check this: http://status.linode.com
Posted by JonFatino, 05-06-2011, 07:03 AM
http://www.webhostingtalk.com/showthread.php?t=1046194

PHP Code:

TraceRoute to 64.62.142.171 [bgp.he.net] Hop (ms) (ms) (ms) IP Address Host name 1 0 0 0 206.123.64.154 - 2 0 0 0 64.124.196.225 xe-4-2-0.er2.dfw2.us.above.net 3 3 2 33 64.125.26.213 xe-1-1-0.cr2.dfw2.us.above.net 4 5 5 5 64.125.26.134 xe-1-2-0.cr2.iah1.us.above.net 5 20 20 20 64.125.31.49 xe-1-1-0.mpr3.atl6.us.above.net 6 20 20 20 64.125.31.42 xe-0-0-0.mpr4.atl6.us.above.net 7 25 25 24 198.32.132.75 10gigabitethernet1-3.core1.atl1.he.net 8 46 46 46 184.105.213.113 10gigabitethernet3-2.core1.dal1.he.net 9 47 50 49 72.52.92.253 10gigabitethernet1-2.core1.phx1.he.net 10 63 55 62 72.52.92.249 10gigabitethernet2-2.core1.lax1.he.net 11 72 74 74 72.52.92.21 10gigabitethernet1-3.core1.pao1.he.net 12 Timed out Timed out Timed out - 13 Timed out Timed out Timed out - 14 Timed out Timed out Timed out - 15 Timed out Timed out Timed out - Trace aborted

Seems as entire FMT1 is down
Posted by craig joe, 05-06-2011, 07:08 AM
this could be a dissaster since many major providers hosted in there
Posted by markdrandall, 05-06-2011, 07:16 AM
It must be a power issue I'm guessing as their Network appears to be up everywhere else.... Although I am sure that they have UPS's and Backup Generators? (Unlike FDC for instance)
Posted by shearerc, 05-06-2011, 07:22 AM

Quote:

Originally Posted by Oli-Myshoutcast

Although I am sure that they have UPS's and Backup Generators? (Unlike FDC for instance)

For sure they have, but how many times have we seen the generators crapping out too...
Posted by markdrandall, 05-06-2011, 07:26 AM

Quote:

Originally Posted by shearerc

For sure they have, but how many times have we seen the generators crapping out too...

We only have 1 server there (Via EGI Hosting) and before this it has only had 1 power outage in 9 months and that lasted for about 5 minutes...

I don't get why companies like EGI host their actual websites on HE though, as that is where they co-locate so it would make sense to host their site somewhere else...
Posted by net, 05-06-2011, 07:32 AM
Wow, how long he.net has been down already?
Posted by WWS2010, 05-06-2011, 07:32 AM
Ah so the whole HE.net facility is down Didn't know that, thought it was BuyVM only.
Posted by freethought, 05-06-2011, 07:38 AM

Quote:

A utility failure is affecting much of Fremont, CA, USA, damaging some electrical equipment. We are currently working to restore power.

From @henet on Twitter
Posted by WWS2010, 05-06-2011, 07:52 AM
It seems that at least some part in the network is alive again, when i first got time-out on ping, i now get;

64.71.165.146: Destination Host Unreachable.
Posted by canar, 05-06-2011, 07:52 AM
hmm this could take a while
Posted by markdrandall, 05-06-2011, 07:55 AM
Are you all still having problems? Our Dedicated box came back online 10 mins ago:

7 43 ms 36 ms 39 ms 10gigabitethernet1-1.core1.lon1.he.net [195.66.224.21]
8 119 ms 119 ms 125 ms 10gigabitethernet6-3.core1.ash1.he.net [72.52.92.137]
9 193 ms 198 ms 198 ms 10gigabitethernet1-4.core1.pao1.he.net [72.52.92.29]
10 215 ms 194 ms 194 ms 10gigabitethernet1-2.core1.fmt1.he.net [66.160.158.241]
11 194 ms 194 ms 197 ms 64.71.129.70
12 196 ms 196 ms 192 ms 184.105.203.74

Trace complete.
Posted by Trix, 05-06-2011, 08:02 AM

Quote:

Originally Posted by Oli-Myshoutcast

Are you all still having problems? Our Dedicated box came back online 10 mins ago:

Mine isn't back up yet
Posted by craig joe, 05-06-2011, 08:03 AM

Quote:

Originally Posted by Oli-Myshoutcast

Are you all still having problems? Our Dedicated box came back online 10 mins ago:

it's still offline for me
Posted by WWS2010, 05-06-2011, 08:04 AM
Still having provlems.

My BuyVM VPS is still down. The buyvm.net site itself isn't up yet, seems that frantech hasn't restored yet.
Posted by shearerc, 05-06-2011, 08:06 AM
One by one servers are coming back up.

As far as I notice, BuyVM not back up yet.
Code:
traceroute to 205.185.xxxx (205.185.xxxx), 30 hops max, 40 byte packets
 1  xxxxxxx.in-addr.arpa.static.dsn1.net (69.xxxxx)  0.033 ms  0.045 ms  0.027 ms
 2  gige-g2-2.core1.pdx1.he.net (65.19.175.37)  2.284 ms  2.783 ms  2.777 ms
 3  10gigabitethernet7-1.core1.sjc2.he.net (72.52.92.13)  26.010 ms  25.938 ms  25.919 ms
 4  10gigabitethernet1-1.core1.fmt1.he.net (72.52.92.109)  25.211 ms  25.196 ms  25.706 ms
 5  te1-1.ve5.crr1.fmt1.egihosting.com (64.71.129.70)  22.867 ms  22.856 ms  22.838 ms
 6   (64.71.165.146)  15.965 ms  15.958 ms  15.942 ms
 7   (64.71.165.146)  2993.821 ms !H * *
Posted by Shatowolf, 05-06-2011, 08:15 AM

Quote:

henet Hurricane Electric : via Twitter

Power has been restored and servers are coming back online.

It will be fine,
Posted by craig joe, 05-06-2011, 08:16 AM

Quote:

From twitter :
Power has been restored and servers are coming back online.

well, it's gonna take few more hours since the vm need to do some fsck
Posted by Shatowolf, 05-06-2011, 08:38 AM

Quote:

From twitter :
henet Hurricane Electric
FMT2 datacenter remains unaffected. FMT1 experienced utility failure w/damaged power equip. Servers continuing to boot & come back online.

thank god, they update us, realy fast
Posted by WWS2010, 05-06-2011, 10:33 AM
I'm back online.
Posted by dlewis23, 05-06-2011, 10:45 AM
Well my VPS's are back up at BuyVM, but my MySQL databases are not. Is anyone else's Offloaded MySQL back up?
Posted by zidit, 05-06-2011, 11:16 AM
mine stil offline (buyvm)
Posted by WWS2010, 05-06-2011, 11:51 AM
Is Fran still asleep? I don't see any status updates on twitter or in the client area. I can tell that node 21 is up, but my 2nd VPS on node 18 is still down.
Posted by Shatowolf, 05-06-2011, 11:57 AM
Frantech team working on it.
Posted by DeltaAnime, 05-06-2011, 12:41 PM
I can confirm that there was a fairly large scale power outage today and the guys are hurrying to correct everything they can and checking for any physically damaged gear.

Sorry about this everyone, we're working quickly at it all

Francisco
Posted by craig joe, 05-06-2011, 02:01 PM
it's not your fault frans,
i'm glad my vm is back online
Posted by WWS2010, 05-07-2011, 05:36 AM
Got 1 VPS still down, thats 24 hours and counting. But is wasn't running anything important, so i'll just wait and see
Posted by cedricd, 05-07-2011, 05:37 AM

Quote:

Originally Posted by WWS2010

Got 1 VPS still down, thats 24 hours and counting. But is wasn't running anything important, so i'll just wait and see

All of mine have successfully come back up at this point; I'm sure yours won't be too longer (as sure you can be in these situations )!
Posted by araki, 05-07-2011, 06:08 AM
My VPS on node18 is still down.
Posted by Jacob2014, 05-07-2011, 08:57 AM

Quote:

Originally Posted by araki

My VPS on node18 is still down.

Same here node 18 appears to be down
Posted by craig joe, 05-07-2011, 09:02 AM
here the list :

Quote:

node02 (offline/fscking)
node05 (online/fsck done)
node06 (online/no fsck required)
node08 (online/fsck done)
node13 (online/no fsck required)
node14 (online/fsck done)
node15 (online/fsck done)
node16 (online/no fsck required)
node18 (offline/fscking)
node20 (online/ nofsck required)
node21 (online/fsck done)
node22 (offline/fscking)
node23 (online/no fsck needed)
node24 (online/no fsck needed)
node25 (online/fsck done)
node27 (online/no fsck needed)
node28 (online/no fsck needed)
node29 (online/no fsck needed)
node30 (online/no fsck needed)
node32 (online/no fsck needed)

https://my.frantech.ca/announcements.php?id=62
Posted by DeltaAnime, 05-07-2011, 10:39 AM

Quote:

Originally Posted by WWS2010

Got 1 VPS still down, thats 24 hours and counting. But is wasn't running anything important, so i'll just wait and see

Most nodes came up but were later diagnosed for needing a FSCK done alas

node18 is one of them, as is a couple others.

Thanks,

Francisco
Posted by quad3datwork, 05-07-2011, 07:17 PM
Anthony did a good job on the recovery. I had a VPS on node21 affected by this. He was kind and polite replying to my ticket.
Posted by DeltaAnime, 05-07-2011, 07:20 PM

Quote:

Originally Posted by quad3datwork

Anthony did a good job on the recovery. I had a VPS on node21 affected by this. He was kind and polite replying to my ticket.

Glad to hear it

We have a meeting with egihosting/HE on Monday to decide what's going to happen.

Thanks,

Francisco
Posted by WWS2010, 05-08-2011, 10:12 AM
My VPS on node 18 is back online again. It required a manual boot-up via manage.buyvm.net, however my other VPS on node 21 powered up automatically.

I'll send a ticket to have fran check if my VPS is in the automatic start queue or not. Don't want to boot it manually after the node reboots.