Anybody Know What's Going On With VolumeDrive Right Now?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Anybody Know What's Going On With VolumeDrive Right Now?

Posted by geeyay1, 01-22-2012, 05:45 AM
I have about 20 big sites hosted with them right now and my box is down.

I emailed them but then noticed that their site is down as well. anyone know what the deal is here?
Posted by JD0192, 01-22-2012, 06:19 AM
I know there was supposed to be maintainance on a couple of our servers, Just appears as "No route to host". There is also a 2 Hour time window between the Start and End of the maintainance.

I guess its a network / power issue, They are not responding to emails.
Posted by Matt R, 01-22-2012, 06:54 AM

Quote:

Originally Posted by JD0192

I guess its a network / power issue, They are not responding to emails.

Not sure. their DNS doesn't appear to be resolving. The IP address previously logged by domaintools.com as being theirs is currently resolving directly to a Cacti installation.

I'm guessing their websites DNS is messed up right now.
Posted by che1959, 01-22-2012, 07:16 AM
A quick check shows that they're hosting in Miami. Just a shot in the dark but they could be using the same data center that Burstnet is using....and they have been having issues in their Miami data center for about 6 hours so far.

Edited: And now burstnet is saying less than two hours until the issue is corrected(it's a fiber issue outside the data center)
Posted by xenbox, 01-22-2012, 07:18 AM
This message is to inform you of an upcoming scheduled maintenance which may result in the following server(s) temporarily going offline for up to 15 minutes: CLF-03

Scheduled Maintenance:
Start: 2012-01-22 08:00:00 UTC
End: 2012-01-22 10:00:00 UTC

--
-----------------------------------------------

Best Regards,
Simon E.

General: info[at]volumedrive.com
Support: support[at]volumedrive.com

VolumeDrive | Internet Technologies
Posted by Matt R, 01-22-2012, 07:20 AM

Quote:

Originally Posted by che1959

A quick check shows that they're hosting in Miami. Just a shot in the dark but they could be using the same data center that Burstnet is using....and they have been having issues in their Miami data center for about 6 hours so far.

AS46664 (Volumedrives ASN for their IP ranges) routes over the HostNOC network (BurstNET) and nLayer. So yes, they do use BurstNET as their primary upstream provider.

I don't see the IP addresses of 199.19.111.4/199.19.111.5 resolving currently. But since they're all on the same AS# and other IP's are resolving, it all depends on where the IP addresses are announced.

Since I don't know where they announce those IP's normally, I can't speak for it. I just know that their websites assigned IP address is resolving but their DNS servers are not
Posted by JD0192, 01-22-2012, 07:27 AM
The maintainance time was supposed to be resolved over 1 Hour ago.

Quote:

Originally Posted by xenbox

This message is to inform you of an upcoming scheduled maintenance which may result in the following server(s) temporarily going offline for up to 15 minutes: CLF-03

Scheduled Maintenance:
Start: 2012-01-22 08:00:00 UTC
End: 2012-01-22 10:00:00 UTC

--
-----------------------------------------------

Best Regards,
Simon E.

General: info[at]volumedrive.com
Support: support[at]volumedrive.com

VolumeDrive | Internet Technologies

Posted by JD0192, 01-22-2012, 07:31 AM
Oh well, I will make another coffee while volumedrive twiddle their thumbs.

Quote:

Service: ping monitor
Status: DOWN
Time: Sun Jan 22 11:29:07 2012 GMT
Previous state lasted: 85 days, 9 hours
Current state start time: Sun Jan 22 08:28:48 2012 GMT
Current state elapsed: 3 hours

Hong Kong (CN): PING CRITICAL - Packet loss = 100%
New York (US): PING CRITICAL - Packet loss = 100%
Moscow (RU): PING CRITICAL - Packet loss = 100%
Tokyo (JP): PING CRITICAL - Packet loss = 100%
Ireland (GB): PING CRITICAL - Packet loss = 100%
Las Vegas (US): PING CRITICAL - Packet loss = 100%
Milan (IT): CRITICAL - Host Unreachable
Dallas (US): PING CRITICAL - Packet loss = 100%
Frankfurt (DE): PING CRITICAL - Packet loss = 100%
Stockholm (SE): CRITICAL - Time to live exceeded
Seattle (US): CRITICAL - Time to live exceeded
Manchester (GB): PING CRITICAL - Packet loss = 100%
Karlstad (SE): PING CRITICAL - Packet loss = 100%
Madrid (ES): PING CRITICAL - Packet loss = 100%
Montreal (CA): CRITICAL - Host Unreachable
Singapore (SG): CRITICAL - Network Unreachable
Mumbai (IN): PING CRITICAL - Packet loss = 100%
Basel (CH): CRITICAL - Network Unreachable
Paraiba (BR): PING CRITICAL - Packet loss = 100%
Brisbane (AU): CRITICAL - Host Unreachable
London 2 (GB): PING CRITICAL - Packet loss = 100%
Cape Town (ZA): PING CRITICAL - Packet loss = 100%
Los Angeles (US): PING CRITICAL - Packet loss = 100%

Posted by xenbox, 01-22-2012, 07:41 AM

Quote:

Originally Posted by JD0192

The maintainance time was supposed to be resolved over 1 Hour ago.

ah my bad during time zone
Posted by xenbox, 01-22-2012, 07:46 AM
my server up now
Posted by JD0192, 01-22-2012, 07:53 AM

Quote:

Originally Posted by xenbox

my server up now

All resolved, But that is 3 Hours downtime. Not good at all.
Posted by xenbox, 01-22-2012, 07:56 AM
yeah at email they send its only take 15 minutes.
Posted by cd/home, 01-22-2012, 08:18 AM
It allways amazies me to see people who want to pay bottom dollar expect uptime of 100% 24/7... If you want enterprise uptime then use enterprise equipment on an enterprise network, etc
Posted by JD0192, 01-22-2012, 08:22 AM

Quote:

Originally Posted by cd/home

It allways amazies me to see people who want to pay bottom dollar expect uptime of 100% 24/7... If you want enterprise uptime then use enterprise equipment on an enterprise network, etc

I don't see anything on this thread that quotes "uptime of 100% , Nor that anyone expects a uptime guarentee", We just simply want prior notification which they did but the downtime exceeded the maintainance time frame.
Posted by Matt R, 01-22-2012, 08:24 AM

Quote:

Originally Posted by JD0192

I don't see anything on this thread that quotes "uptime of 100% , Nor that anyone expects a uptime guarentee", We just simply want prior notification which they did but the downtime exceeded the maintainance time frame.

Major providers always do something similar to the following:

They schedule 12am-4am as a maintenance period, but made sure to clearly notate that the work itself is only estimated to take no longer than 15-30 minutes and that packetloss may be intermittent during that time period.

That way, if something goes wrong, they did provide adequate prior notification.

Always overestimate.
Posted by JD0192, 01-22-2012, 08:28 AM
It's happened, again.. My emails do not even get delivered to their mail servers so I am assuming something went wrong in the maintainance.

Quote:

Service: ping monitor
Status: DOWN
Time: Sun Jan 22 12:24:57 2012 GMT
Previous state lasted: 39 minutes
Current state start time: Sun Jan 22 12:24:53 2012 GMT
Current state elapsed: 4 seconds

Hong Kong (CN): PING CRITICAL - Packet loss = 100%
New York (US): PING CRITICAL - Packet loss = 100%
Moscow (RU): OK
Tokyo (JP): OK
Ireland (GB): OK
Las Vegas (US): PING CRITICAL - Packet loss = 100%
Milan (IT): OK
Dallas (US): PING CRITICAL - Packet loss = 100%
Frankfurt (DE): OK
Stockholm (SE): CRITICAL - Time to live exceeded
Seattle (US): CRITICAL - Time to live exceeded
Manchester (GB): PING CRITICAL - Packet loss = 100%
Karlstad (SE): PING CRITICAL - Packet loss = 100%
Madrid (ES): PING CRITICAL - Packet loss = 100%
Montreal (CA): CRITICAL - Host Unreachable
Singapore (SG): OK
Mumbai (IN): PING CRITICAL - Packet loss = 100%
Basel (CH): CRITICAL - Network Unreachable
Paraiba (BR): PING CRITICAL - Packet loss = 100%
Brisbane (AU): OK
London 2 (GB): OK
Cape Town (ZA): PING CRITICAL - Packet loss = 100%
Los Angeles (US): OK

Quote:

Originally Posted by Matt R

Major providers always do something similar to the following:

They schedule 12am-4am as a maintenance period, but made sure to clearly notate that the work itself is only estimated to take no longer than 15-30 minutes and that packetloss may be intermittent during that time period.

That way, if something goes wrong, they did provide adequate prior notification.

Always overestimate.

Posted by heropage, 01-22-2012, 08:51 AM
mine are still down!
Posted by iexo, 01-22-2012, 08:51 AM
With their pricing it surprises me to hear they even connect the server to a port.
Posted by geeyay1, 01-22-2012, 08:54 AM

Quote:

Originally Posted by Matt R

Major providers always do something similar to the following:

They schedule 12am-4am as a maintenance period, but made sure to clearly notate that the work itself is only estimated to take no longer than 15-30 minutes and that packetloss may be intermittent during that time period.

That way, if something goes wrong, they did provide adequate prior notification.

Always overestimate.

My box has been down for over 4 hours now. Just got an email responding to a support query from earlier today about a totally unrelated issue and they didn't bother to mention anything about this.

I'm definitely not very pleased right now. Hope this gets sorted asap.
Posted by JD0192, 01-22-2012, 09:05 AM
Online again, Just got the notification. Lets see if it stays online...

Quote:

wget -O /dev/null http://cachefly.cachefly.net/100mb.test
--2012-01-22 08:03:28-- http://cachefly.cachefly.net/100mb.test
Resolving cachefly.cachefly.net... 205.234.175.175
Connecting to cachefly.cachefly.net|205.234.175.175|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/octet-stream]
Saving to: `/dev/null'

100%[======================================>] 104,857,600 11.0M/s in 9.3s

2012-01-22 08:03:38 (10.8 MB/s) - `/dev/null' saved [104857600/104857600]

Posted by volumedrive, 01-22-2012, 01:45 PM

Quote:

Originally Posted by Matt R

AS46664 (Volumedrives ASN for their IP ranges) routes over the HostNOC network (BurstNET) and nLayer. So yes, they do use BurstNET as their primary upstream provider.

I don't see the IP addresses of 199.19.111.4/199.19.111.5 resolving currently. But since they're all on the same AS# and other IP's are resolving, it all depends on where the IP addresses are announced.

Since I don't know where they announce those IP's normally, I can't speak for it. I just know that their websites assigned IP address is resolving but their DNS servers are not

Our Jacksonville data center (which this is about) peers directly with nLayer. We do not not have any affiliation with BurstNET here. Nonetheless, this scheduled maintenance this morning was planned and was part of a large scale IOS and router upgrade that was needed for future expansion and reliability. Took longer than expected. All services have been fully restored.
Posted by volumedrive, 01-22-2012, 01:46 PM

Quote:

Originally Posted by JD0192

Online again, Just got the notification. Lets see if it stays online...

Of course it will. Our upgrades and maintenance have been completed at this time. If you have any specific questions, please feel free to contact us.
Posted by volumedrive, 01-22-2012, 01:51 PM

Quote:

Originally Posted by JD0192

I don't see anything on this thread that quotes "uptime of 100% , Nor that anyone expects a uptime guarentee", We just simply want prior notification which they did but the downtime exceeded the maintainance time frame.

Maintenance did exceed the time frame. However, we did notify everyone of this very important upcoming maintenance window. Everything went well and all upgrades that were planned have been completed. We do not have any foreseeable scheduled maintenance in the near future.
Posted by markdrandall, 01-22-2012, 06:11 PM
And at least VD is back up unlike burst.net in Miami which has now been down for over 15 hours!
Posted by BurstNET, 01-22-2012, 06:17 PM

Quote:

Originally Posted by Oli-Myshoutcast

And at least VD is back up unlike burst.net in Miami which has now been down for over 15 hours!

Fiberlight took out an entire redundant fiber ring in Miami, with a botched maintenance they were doing, which we were not properly notified of even in the first place. Some genius there though it would be a good idea to work on two sides of a redundant fiber ring at the same time. Obviously we are livid, and they will have ramifications, but all we can do at the moment is wait for repair, which is underway still.
.
.
Posted by iTom, 01-22-2012, 06:19 PM

Quote:

Originally Posted by BurstNET

Fiberlight took out an entire redundant fiber ring in Miami, with a botched maintenance they were doing, which we were not properly notified of even in the first place. Some genius there though it would be a good idea to work on two sides of a redundant fiber ring at the same time. Obviously we are livid, and they will have ramifications, but all we can do at the moment is wait for repair, which is underway still.
.
.

Multi vendor/supplier might be a good idea in the future, where I work had a similar problem with a very large local telco recently (in the UK).
Posted by BurstNET, 01-22-2012, 06:35 PM

Quote:

Originally Posted by iTom

Multi vendor/supplier might be a good idea in the future, where I work had a similar problem with a very large local telco recently (in the UK).

A fully redundant/diverse dark fiber ring should be all that is needed for 100% uptime. No gear is going to fail from a single carrier, and we are talking about just two fibers running on two completely separate paths and entrance ways---and only 7 blocks between locations even. That is no different really than two unique providers each providing a unique pair on different paths, and about the same level of risk only.

There should never be an outage based on our design, and the only reason this occurred is due to intentional actions by our fiber provider and their totally screwing up here...and possibly opening themselves up to legal action due to just plain incompetence and poor decision making. I mean really, who in their right minds does maintenance on two sides of a ring at the exact same time (and then botches it to boot)? Morons...

...and on top of that, they never even notified us of the maintenance to proper contacts we have listed with them, so we could not even object to it in advance.

PS - We utilize multiple/redundant transit providers in all our POPs....that is not the issue here.
.
.
Posted by WinsNexus, 01-22-2012, 08:16 PM
Burst,

Will the servers be up by morning. A large percentage of our VPS servers are with you and unfortunately are in Miami. One is our development server and I pay 50+ an hour to 9 developers come morning. Tomorrow is going to cost me 5k for developers sitting around cruising Google. I have already paid our weekend warriors 2K today alone for nothing.
Posted by infracom2005, 01-22-2012, 08:20 PM
BurstNET - 17 hours offline, last update was 7 hours ago... Service is terrible
Posted by WinsNexus, 01-22-2012, 08:35 PM

Quote:

Originally Posted by infracom2005

BurstNET - 17 hours offline, last update was 7 hours ago... Service is terrible

I can understand the long period of downtime due to the fiber being cut. To repair fiber is a long process, and I really only know of one major fiber repair company in the southeast so more than likely we also have to take into consideration of travel time, plus repair.

Being that this is not really Burst's fault, but we do need to start looking at SLA's guaranteed to customers from Burst's end, which I am sure that Burst is looking into SLA and legal with their provider. I am sure that Burst will be given something for their trouble due to the issue - Cont. Next Paragraph.

As long as Burst understands that there is money being lost on both ends, which Burst's SLA agreement basically covers almost every aspect of service not being provided to the point they are not responsible for any type of outage. We will see if your comment does in fact hold weight in regards to the situation.
Posted by che1959, 01-22-2012, 09:28 PM
To Burstnet's credit, they have been updating all along.
Posted by WinsNexus, 01-23-2012, 01:40 AM
20.5 hours of downtime, servers are pingable with Burst - Miami. Response times are quite high (8 times higher 300 - 400 ms from NC), but at least on-line.
Posted by cresci, 01-23-2012, 07:52 AM

Quote:

Originally Posted by BurstNET

Fiberlight took out an entire redundant fiber ring in Miami, with a botched maintenance they were doing, which we were not properly notified of even in the first place. Some genius there though it would be a good idea to work on two sides of a redundant fiber ring at the same time. Obviously we are livid, and they will have ramifications, but all we can do at the moment is wait for repair, which is underway still.
.
.

Considering that they are providing service for 99% of the Southern Florida carriers as they have the only decent ring around... FPL Fibernet also does have that, but they have almost abusive pricing on transport services. I wonder all Cogent Miami is down also, as well as links between Miami and Atlanta?

Weird is they did an excellent job while swapping/rerouting the fiber out of Equinix Miami 1 (Metromall) with almost no downtime; yet this event was/is a complete disaster? They left almost every Miami datacenter except Terremark down or mostly down.
Posted by pubcrawler, 01-26-2012, 03:31 PM
Volumedrive, another dual DNS server on same network setup.

When are these providers going to learn to place DNS outside your network and to put ticketing, support and even your own website outside such too?

This outage showed that consolidation and monopoly like control over a regional market can and will inevitably lead to a massive outage.

Florida, another hosting market to steer clear of.
Posted by volumedrive, 01-26-2012, 05:36 PM

Quote:

Originally Posted by pubcrawler

Volumedrive, another dual DNS server on same network setup.

When are these providers going to learn to place DNS outside your network and to put ticketing, support and even your own website outside such too?

This outage showed that consolidation and monopoly like control over a regional market can and will inevitably lead to a massive outage.

Florida, another hosting market to steer clear of.

We have nothing to do with the above provider or outage. We opened a maintenance window with all of our Jacksonville, FL customers to let them know about the planned upgrades to our core routers at this facility. Everything is restored and we have no foreseeable planned maintenance coming up. What seems to be the issue here pubcrawler?