EuroVPS

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > EuroVPS

Posted by Shardhost, 01-19-2013, 10:02 AM
EuroVPS appears to be down. All of our services are offline and their website is down also
Posted by n!ghtmare, 01-19-2013, 10:13 AM
Down here too
Posted by n!ghtmare, 01-19-2013, 10:14 AM
And their phone number has no connection
Posted by Maak, 01-19-2013, 10:17 AM

Quote:

Originally Posted by n!ghtmare

And their phone number has no connection

Down worldwide
http://www.just-ping.com/index.php?v....com&c=&s=ping!
Posted by target, 01-19-2013, 10:20 AM
Weird, no news on twitter also announcing the problem.
Posted by n!ghtmare, 01-19-2013, 10:21 AM
I am definitely moving after they come back up. Been with them for around three months and had 5+ downtimes.
Posted by a_system, 01-19-2013, 10:23 AM
Down for me too
Posted by pedrojose, 01-19-2013, 10:27 AM
also down from here.. no twitter updates and they should have their helpdesk on a different network for cases like this.

i have checked leaseweb status page and not problem there. it seems to be just eurovps
Posted by Shardhost, 01-19-2013, 10:33 AM
I've managed to speak to livechat using googles webcache of their page. Support seemed unaware of a downtime. Downtime was confirmed after though and said hope to have it up shortly.
Posted by Maak, 01-19-2013, 10:38 AM

Quote:

Originally Posted by Shardhost

Support seemed unaware of a downtime.

Posted by Maak, 01-19-2013, 10:40 AM
OK ONline now!
Posted by pedrojose, 01-19-2013, 10:41 AM
They have just now updated their twitter with:

Update on Downtime: Major inbound DDoS. Will update with developments asap.
Posted by pedrojose, 01-19-2013, 10:45 AM
My monitoring tools reports that now its up again but it was a total downtime of 44 minutes.
Posted by Maak, 01-19-2013, 10:47 AM

Quote:

Originally Posted by pedrojose

My monitoring tools reports that now its up again but it was a total downtime of 44 minutes.

My VPS ok, but eurovps.com site it's down

http://www.just-ping.com/index.php?v...779&vhost=_&c=
Posted by a_system, 01-19-2013, 10:53 AM
my vps back online - client area not yet - but I surely prefer tis than the opposite...
Posted by target, 01-19-2013, 11:48 AM

Quote:

Originally Posted by n!ghtmare

I am definitely moving after they come back up. Been with them for around three months and had 5+ downtimes.

I tried them out a few months ago. The VPS did run smoothly and support reacted very fast. Only thing that bothered me and why i left was that their 'managed' option isn't really managed.

Quote:

Originally Posted by Shardhost

I've managed to speak to livechat using googles webcache of their page. Support seemed unaware of a downtime. Downtime was confirmed after though and said hope to have it up shortly.

That's very weird.
Posted by a_system, 01-19-2013, 12:22 PM
I have been with EUROVPS with a VPS3 for week now and I am:
- very happy with performance
- very happy with 2-level disk setup
- OK with support and communication (got silver option - too early to have clear view - I see they respond fast but won't get very deep in things that they consider out of the scope of your support plan)
- not happy with the lazyness of greek ISPs DNS (truely awful in some cases but of course not related to eurovps)

So I really wish EUROVPS would prove to be a steady ground for me and I am very worried about this incident and especially the "seemed unaware of a downtime". I would imagine bells ringing and alarms screaming in such a situation, wouldn't you?

Which brings me to another weird thing: my UptimeRobot monitors that I did setup a few days ago show the same "unaware" behavior (see attachment). Why would that happening? Is UptimeRobot some free crap?
Posted by target, 01-19-2013, 12:36 PM

Quote:

Originally Posted by a_system

I have been with EUROVPS with a VPS3 for week now and I am:
- very happy with performance
- very happy with 2-level disk setup
- OK with support and communication (got silver option - too early to have clear view - I see they respond fast but won't get very deep in things that they consider out of the scope of your support plan)

Response is fast but for a managed option their answers were sometimes at the lazy side. Always redirecting things, so at the end you still have to do it yourself.
Posted by pedrojose, 01-20-2013, 07:55 AM
Hi,

Does anyone received any kind of official information by email or other way from EuroVPS about what was the issue that caused the 44 minutes downtime yesterday?

And also the interesting part was when the user Shardhost said that when he talked with EuroVPS team it seemed that they where not aware of the problem and if we take the timeframe with attention they have only update their twitter few minutes Shardhost as talked with them by live chat.

Im a long time EuroVPS client and i think they are an excellent provider, but this lack of information and also the re-active way that it seem the problem was handle leaves me with a step back towards the future.
Posted by Shardhost, 01-20-2013, 09:47 AM

Quote:

Originally Posted by pedrojose

Hi,

Does anyone received any kind of official information by email or other way from EuroVPS about what was the issue that caused the 44 minutes downtime yesterday?

And also the interesting part was when the user Shardhost said that when he talked with EuroVPS team it seemed that they where not aware of the problem and if we take the timeframe with attention they have only update their twitter few minutes Shardhost as talked with them by live chat.

Im a long time EuroVPS client and i think they are an excellent provider, but this lack of information and also the re-active way that it seem the problem was handle leaves me with a step back towards the future.

While I was on Livechat, I also asked if an RFO would be issued. They said yes they always send them out or via twitter. I claimed a 20% SLA credit for the downtime.

Since the downtime my sites have been quite slow. Some page loads taking 12 seconds
Posted by pedrojose, 01-20-2013, 10:00 AM

Quote:

Originally Posted by Shardhost

While I was on Livechat, I also asked if an RFO would be issued. They said yes they always send them out or via twitter. I claimed a 20% SLA credit for the downtime.

Since the downtime my sites have been quite slow. Some page loads taking 12 seconds

@Shardhost: When you where on live chat, did you really have the feeling that their team was not aware of the problem? Because if yes, this is really very strange, because for example my simple Pingom monitoring too reported to me immediately.

Also regarding the RFO in the past they have always send it in a very short time.. but until now nothing. It seems that they still must be searching and trying to fully understand what happen.
Posted by Shardhost, 01-20-2013, 10:13 AM

Quote:

Originally Posted by pedrojose

@Shardhost: When you where on live chat, did you really have the feeling that their team was not aware of the problem? Because if yes, this is really very strange, because for example my simple Pingom monitoring too reported to me immediately.

Also regarding the RFO in the past they have always send it in a very short time.. but until now nothing. It seems that they still must be searching and trying to fully understand what happen.

I did not keep the chat log however it was:

Me telling them was downtime

Then telling me there shouldn't be although there might be some maintenance going on

Me telling them that all services were down including their website and I was using the google cache to use the live chat
Posted by Time4VPS, 01-20-2013, 10:19 AM

Quote:

Originally Posted by Shardhost

I did not keep the chat log however it was:

Me telling them was downtime

Then telling me there shouldn't be although there might be some maintenance going on

Me telling them that all services were down including their website and I was using the google cache to use the live chat

Maybe they using outsourced support. Nothing strange.
Posted by pedrojose, 01-20-2013, 10:28 AM

Quote:

Originally Posted by time4vps

Maybe they using outsourced support. Nothing strange.

No. EuroVPS clear says on their website that they do not outsource support.

They say and i quote: "We dont outsource our support to third parties" on their support info page at: http://www.eurovps.com/support
Posted by Shardhost, 01-20-2013, 10:35 AM
Either way. DDOS happens and the downtime I can suffer. They have credited as per their SLA and I await an RFO.
Posted by EuroVPS/Dennis, 01-21-2013, 05:43 PM
As promised, here is the reason for outage. Our initial tweet cited a DDoS attack as the cause of downtime. Although this is partly true, now that the dust has settled, we are now able to provide a more detailed uncensored explanation.

When we detected the attack, we dispatched one engineer to the datacenter to investigate the problem, due to heavy snow fall in Amsterdam it took approximately 23 minutes for the engineer to reach the datacenter. During this time, I was attempting to control the situation and inadvertently, due to a human error, created a network loop between our LAN and WAN network. This was definitely one of the most dreadful moments of my life.

Our administrative systems were affected due to this loop as they lost both paths to their iSCSI datastores, as both iSCSI networks were looped. We discovered that an earlier, undetected human error had resulted in a bridge between switches, and this became a cataclysmic issue when I was attempting to control the situation.

When the engineer arrived on site a few minutes later, he corrected the loop, and reloaded the routers. From this moment, internet traffic began to flow normally once again, however systems which were dependant on iSCSI (these are our own systems only, we do not use iSCSI for customer property) were severely affected.

As a result of this, EuroVPS has completed a full migration of it's corporate and administrative systems to Fibre Channel, and even in the case of any future network anomaly, loop, etc, such an event cannot reoccur.

We do not use iSCSI for any customer properties for various reasons, and as of now, we are also glad to have moved our own systems away from iSCSI. iSCSI is excellent if used properly, however if any human error is introduced and/or any networking is shared between it and other non-iSCSI systems, disaster can ensue.

We were aware of this deficiency and had already planned a maintenance window to migrate all of our systems, administration platforms, email, away from iSCSI and to Fibre Channel. This now, was completed during the last 36 hours.

Again, we do not use iSCSI for our customer servers, and no customers experienced any data loss or corruption due to this fault, however some customers were exposed to network downtime (public internet access) for up to 44 minutes.

The first racks started coming online 6 minutes after the reload, however the initial cause of the downtime was still ensuing, and therefore more work was needed to get other racks online. Within 40 minutes, all customer services were online.

Our administration systems however were offline for a slightly longer than our customers', which is why there were reports of not being able to login to the client area portal. This was because the migration towards Fibre Channel started immediately after our customers WAN & public VLANs were online.

We apologize for this very embarrassing mishap, and rest assured that we have taken corrective measures so this never happens again. Any clients that want to claim SLA compensation are of course welcome to request it by sending an email to administration@eurovps.com.

Best Regards,
Dennis M.
EuroVPS Sr. Network Engineer
Posted by pedrojose, 01-21-2013, 07:16 PM

Quote:

Originally Posted by EuroVPS/Dennis

As promised, here is the reason for outage. Our initial tweet cited a DDoS attack as the cause of downtime. Although this is partly true, now that the dust has settled, we are now able to provide a more detailed uncensored explanation.

When we detected the attack, we dispatched one engineer to the datacenter to investigate the problem, due to heavy snow fall in Amsterdam it took approximately 23 minutes for the engineer to reach the datacenter. During this time, I was attempting to control the situation and inadvertently, due to a human error, created a network loop between our LAN and WAN network. This was definitely one of the most dreadful moments of my life.

Our administrative systems were affected due to this loop as they lost both paths to their iSCSI datastores, as both iSCSI networks were looped. We discovered that an earlier, undetected human error had resulted in a bridge between switches, and this became a cataclysmic issue when I was attempting to control the situation.

When the engineer arrived on site a few minutes later, he corrected the loop, and reloaded the routers. From this moment, internet traffic began to flow normally once again, however systems which were dependant on iSCSI (these are our own systems only, we do not use iSCSI for customer property) were severely affected.

As a result of this, EuroVPS has completed a full migration of it's corporate and administrative systems to Fibre Channel, and even in the case of any future network anomaly, loop, etc, such an event cannot reoccur.

We do not use iSCSI for any customer properties for various reasons, and as of now, we are also glad to have moved our own systems away from iSCSI. iSCSI is excellent if used properly, however if any human error is introduced and/or any networking is shared between it and other non-iSCSI systems, disaster can ensue.

We were aware of this deficiency and had already planned a maintenance window to migrate all of our systems, administration platforms, email, away from iSCSI and to Fibre Channel. This now, was completed during the last 36 hours.

Again, we do not use iSCSI for our customer servers, and no customers experienced any data loss or corruption due to this fault, however some customers were exposed to network downtime (public internet access) for up to 44 minutes.

The first racks started coming online 6 minutes after the reload, however the initial cause of the downtime was still ensuing, and therefore more work was needed to get other racks online. Within 40 minutes, all customer services were online.

Our administration systems however were offline for a slightly longer than our customers', which is why there were reports of not being able to login to the client area portal. This was because the migration towards Fibre Channel started immediately after our customers WAN & public VLANs were online.

We apologize for this very embarrassing mishap, and rest assured that we have taken corrective measures so this never happens again. Any clients that want to claim SLA compensation are of course welcome to request it by sending an email to administration@eurovps.com.

Best Regards,
Dennis M.
EuroVPS Sr. Network Engineer

Its great that EuroVPS can be so clear about what happen.. but i think that before you posted to the public you should first send out the information to your clients (by Email / RFO) and only then to public announce.

Take me as an example, one of my clients read this before me and call me. I did know nothing at the moment...
Posted by pedrojose, 01-21-2013, 09:17 PM
already received direct mail from EuroVPS with RFO. things things happen.

anyway i will still keep my services at eurovps, as until now their services as always been very reliable
Posted by kpmedia, 01-22-2013, 12:42 AM

Quote:

Originally Posted by Shardhost

EuroVPS appears to be down. All of our services are offline and their website is down also

Yes. It was less than 45 minutes. I was asleep, and didn't even notice. It's the first downtime since 2009. Not bad, I'd say. (Nobody else cam say that.)

Quote:

Originally Posted by n!ghtmare

And their phone number has no connection

I have two phone numbers for them in my cell, and they have always worked. I've called them, they've called me, over the past ~7 years.

Quote:

Originally Posted by n!ghtmare

I am definitely moving after they come back up. Been with them for around three months and had 5+ downtimes.

You'll be missing out on an awesome host if you do.

Quote:

Originally Posted by time4vps

Maybe they using outsourced support. Nothing strange.

No.

Quote:

Originally Posted by Shardhost

Either way. DDOS happens and the downtime I can suffer. They have credited as per their SLA and I await an RFO.

Exactly. Down times are extremely rare, and I've not had one in several years. (That wasn't caused by the server, and some of those are PEBKAC errors; I have no issues admitting that!)

Quote:

Originally Posted by pedrojose

Its great that EuroVPS can be so clear about what happen.. but i think that before you posted to the public you should first send out the information to your clients (by Email / RFO) and only then to public announce.

They did. I got the email at 5:31 p.m. CST.
_______

This was under an hour, the first time in years, over the weekend (down time for most sites; i.e. not rush hours), and honestly the looping the servers is funny. If you got to screw up, at least make it humorous! I'm going to cut them slack on this one. Even at their worst, they're better than 99% of the host (and "hosts") out there.

.
.
Posted by target, 01-22-2013, 03:34 AM
At least they are upfront and not making up excuses. +1 for Eurovps.
Posted by layer0, 01-22-2013, 06:30 AM
That's a pretty transparent and professional explanation for the outage, which wasn't even that long in duration in the grand scheme of things. Kudos to them. Downtime happens to everybody, what counts is how you handle it.