Leaseweb cloud down -> A total disappointment

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Leaseweb cloud down -> A total disappointment

Posted by Valentin Mitoiu, 09-25-2013, 12:48 PM
I have an Express S cloud package in their NL site.

I have to say that this has turned out to be a total disappointment from the beginning!

At first the HDD speeds where very very low (30 mb/sec).

After I have opened a ticket with them complaining about performance issues, they just replied once to my ticket and after that the whole cloud became unavailable.

In this process they didn't even bother to reply to my ticket (opened for HOURS!!!!!) or send any type of notification.

Ofcourse the VPS is down and technical support is unavailable...

Aaaa...I asked if I can be moved to their Germany site in order to avoid this kind of issues and they said that I would have to open an account, pay for another cloud instance and migrate myself the VPS.

So, they mess their cloud and I have to move my stuff if I want a working cloud instance .

This is a total disappointment!!!!!!!!
Posted by iCode, 09-25-2013, 01:22 PM
I never really hear anything good about Leaseweb.

Hope your problems get sorted soon!
Posted by HaronMedia, 09-25-2013, 02:18 PM
Your ticket will be open for hours because you probably have the best effort SLA. I suggest subscribing to their NOC list, they've been posting updates about this whole day (http://leasewebnoc.com/en/networkstatus ).

They've had issues with the storage platform since two weeks ago. Thankfully we decided to consolidate all the services we had on a number of Express line VPS to a couple of dedicated servers, those are at least very stable at Leaseweb and the network is superb. We migrated out the last VPS literally minutes before it all collapsed last night.

This is the first time I've seen such an outage with the Express line for over a year we've had them.
Posted by LeaseWeb - Jemma, 09-25-2013, 03:39 PM
Dear Valentin Mitoiu,

We do apologize for the experience you have had with your VPS and the inconvenience this has caused you.

Regarding the current outage; the majority of VPS's are back online. There are some instances stuck in the re-start mode that are still experiencing problems. Our engineers have been hard at work to get this all fixed.

We do advise our customer to subscribe to our NOC notifications to be totally up to date with our Network Status: http://leasewebnoc.com/en/networksta...ud-platform-nl This will enable you to receive the latest notifications regarding our network. You are also able to activate these via your Self Service Centre.

Could you let us know via PM if your issue with the cloud has now been resolved or forward us the ticket number so that we can look into the matter.

This will also enable us to look into the matter of your move to Germany. Once again our apologies for the problems you have encountered. Once we receive your message we will make sure to contact you via mail to look into the possibilities and the assistance we can offer you in the move of your VPS to Germany if you still wish to do so.

Looking forward to receiving your message so we can assist you further.

Kind regards,
LeaseWeb Sales

Quote:

Originally Posted by Valentin Mitoiu

I have an Express S cloud package in their NL site.

I have to say that this has turned out to be a total disappointment from the beginning!

At first the HDD speeds where very very low (30 mb/sec).

After I have opened a ticket with them complaining about performance issues, they just replied once to my ticket and after that the whole cloud became unavailable.

In this process they didn't even bother to reply to my ticket (opened for HOURS!!!!!) or send any type of notification.

Ofcourse the VPS is down and technical support is unavailable...

Aaaa...I asked if I can be moved to their Germany site in order to avoid this kind of issues and they said that I would have to open an account, pay for another cloud instance and migrate myself the VPS.

So, they mess their cloud and I have to move my stuff if I want a working cloud instance .

This is a total disappointment!!!!!!!!

Posted by aeris, 09-25-2013, 04:12 PM
When this was going on I saw up to 30-40 *seconds* of I/O latency on the one instance I have on this, but it was still technically up.

At least they haven't been keeping quiet about it. I received a total of twelve emails from NOC about the issue these past thirteen hours.
Posted by AzaraT, 09-25-2013, 04:45 PM
My server with them is still down, hope it'll get fixed soon
Posted by christianmat, 09-25-2013, 05:30 PM
Hi,

Despite what the fellow LeaseWeb supporter claimed in the past thread (and what they state on their website) the majority of the LeaseWeb cloud is still down. LeaseWeb have stopped answering to new support tickets and replies to existing open ones. Our VPS has been down for more 19 hours at this point, and there is no time horizon as to when we can expect the clouds to go online again..

Anyone else expecting this on this forum? Any suggestions on how to get a VPS back online?

Thanks,
Christian
Posted by LeaseWeb - Jemma, 09-25-2013, 05:43 PM
Dear Christianmat,

We are sorry that you are also experiencing these problems.

Most of the VPS's are online again. There are a few left that are experiencing problems. We urge our customers NOT to restart their VPS's and not try to fix the filesystems. As you can imagine our engineers are focusing on getting this platform issue resolved.

We therefor use the NOC to keep our customers informed of our progress and when the issue is resolved. Please activate these notifications in the Self Service Centre or subscribe to the updates of the NOC: http://leasewebnoc.com/en/networksta...ud-platform-nl.

Please PM us your details.

Kind regards,
LeaseWeb Sales

Quote:

Originally Posted by christianmat

Hi,

Despite what the fellow LeaseWeb supporter claimed in the past thread (and what they state on their website) the majority of the LeaseWeb cloud is still down. LeaseWeb have stopped answering to new support tickets and replies to existing open ones. Our VPS has been down for more 19 hours at this point, and there is no time horizon as to when we can expect the clouds to go online again..

Anyone else expecting this on this forum? Any suggestions on how to get a VPS back online?

Thanks,
Christian

Posted by christianmat, 09-25-2013, 06:02 PM
I cannot PM you for some reason, but one of my open ticket IDs on LeaseWeb NL is 2803947.

See the attached screenshot (sorry for the irrelevant post other people).
Posted by VPSPowerUser, 09-25-2013, 11:40 PM
Its a real joke When I send support request I get this copy-paste:

Quote:

Dear Sir/Madam,

Some customers are still experiencing issues with their instances on the
LeaseWeb Cloud and Virtual Servers. You may not be able to view the
status in the SSC or receive an IP lease.

Engineers are investigating the issue at this moment.

Please be patient while we are working on this problem.

A small number of VPS's are still facing issues, we are working on
getting these back online.
Please do NOT restart your VPS or try fix your filesystem as we are
doing our utmost to solve this issue.

Visit http://leasewebnoc.com/en/networksta...loud-platform-
nl to track updates and progress.

We are sorry for the inconvenience.

Sincerely,

LeaseWeb
http://www.leasewebnoc.com

Data center naming system is explained here:
http://www.leaseweb.com/en/about-us/data-centers

When I check the recommended link I get this message

Quote:

Dear LeaseWeb Customer,

[UPDATE 17:00h CEST]
The Cloud platform is now stable again and is closely being monitored for any issues in the next days.
Should you encounter any issues with your VPS, please open a support ticket at support@leaseweb.com or via our Self Service Center.

We apologize for any inconvenience caused by the issues.

Sincerely,

LeaseWeb
http://www.leasewebnoc.com

Data center naming system is explained here: http://www.leaseweb.com/en/about-us/data-centers

So support tells to check page, page tells that you are fine or if not to contact support. Never ending circle.
Posted by christianmat, 09-26-2013, 02:39 AM
It's been 26 hours now, my VPS is still down. Nothing new to report on the status website.

And just to let everyone know: The LeaseWeb sales person on this forum did not contact me. She didn't even respond to my open ticket! It's also ironic how the support is so active on forums like these, but doesn't respond to any actual tickets.

Hello LeaseWeb! What's going on?!
Posted by neanton, 09-26-2013, 03:47 AM
Same problem here. VPS is down for more than 24 hours. My eCommerce business is loosing money. Waiting for resolution.
Posted by wswd, 09-26-2013, 04:15 AM
So much for clouds providing great uptime.
Posted by aeris, 09-26-2013, 04:23 AM

Quote:

Originally Posted by wswd

So much for clouds providing great uptime.

At this point, every single "cloud" I've ever tried has had severe downtime-inducing problems with its SAN at some point or another. Which is why I still greatly prefer dedicated servers with application-level redundancy. Not putting all your eggs in one basket, and all that.
Posted by VPSPowerUser, 09-26-2013, 05:21 AM

Quote:

Originally Posted by christianmat

It's been 26 hours now, my VPS is still down. Nothing new to report on the status website.

And just to let everyone know: The LeaseWeb sales person on this forum did not contact me. She didn't even respond to my open ticket! It's also ironic how the support is so active on forums like these, but doesn't respond to any actual tickets.

Hello LeaseWeb! What's going on?!

Same here.. Service down.. and nothing happens.
Posted by VPSPowerUser, 09-26-2013, 05:35 AM
It's really ironic. I have open support ticket, that my service is down. I have sent dozen of screenshots of console already. I got this response to my ticket:

Quote:

Dear LeaseWeb Customer,

The Virtual Server storage platform is stable, but we are seeing a
number of customers having trouble accessing or booting their instance.
Because we still have an open ticket registered to this issue we like
you to check if your instance is accessible right now. Please issue a
reboot command via the Self Service Center if it is not working.

Instances that are not functioning correctly after a reboot, we are
fixing on a case by case basis. If you are still encountering problems
we would like to ask you to open a new support ticket, or update this
one, so we can handle your issue as efficiently as possible.

If it is possible to see what message is shown on the console after a
failed reboot this would be helpful information to speed-up
troubleshooting. Please add that into the ticket created.

We apologize for any inconvenience you might have experienced and thank
you for patience in this matter.

Sincerely,

LeaseWeb

And ironically web based console even not functioning anymore Really bureaucratic approach.

Downtime 1 day and 6 hours for now
Posted by HaronMedia, 09-26-2013, 05:53 AM

Quote:

Originally Posted by aeris

At this point, every single "cloud" I've ever tried has had severe downtime-inducing problems with its SAN at some point or another. Which is why I still greatly prefer dedicated servers with application-level redundancy. Not putting all your eggs in one basket, and all that.

This. Absolutely this. Even with local discs you're still at the mercy of neighboring tenants and their disk churning preferences.

VPS and Cloud is really nothing more than glorified shared hosting. With traditional shared hosting you all share a single OS environment and have a directory in its filesystem. With the cloud you just go one level up, have your own kernel and share the hardware.

And nowadays it's easy to get comparatively powerful enough dedicated hardware for the price of a premium VPS.
Posted by LeaseWeb - Jemma, 09-26-2013, 05:53 AM
Dear all,

Our Virtual Server platform is stable, but we are seeing a number of customers having trouble accessing or booting their instance.

If your instance is not accessible right now, please issue a reboot command via the Self Service Center : https://secure.leaseweb.nl

Instances that are not functioning correctly after a reboot, we will be fixing them on a case by case basis. We would like to ask you to open a new support ticket if you are still encountering problems, so we can handle your issue as efficiently as possible.

Kind regards,
LeaseWeb
http://www.leasewebnoc.com
Posted by VPSPowerUser, 09-26-2013, 05:59 AM
Live chat with support:

Quote:

Originally Posted by bruijns:

[It should come online in the next 30 minutess. We are currently having issues with the console servers (these are the ones thet process the reboots). They are under an extremely heavy load at the moment. They should be fixed within the next 30 minutes.

Posted by pat010, 09-26-2013, 06:02 AM

Quote:

Originally Posted by VPSPowerUser

It's really ironic. I have open support ticket, that my service is down. I have sent dozen of screenshots of console already. I got this response to my ticket:

And ironically web based console even not functioning anymore Really bureaucratic approach.

Downtime 1 day and 6 hours for now

I have exactly the same problem as you. However i just been able to get into scc remote console.
The error im getting is :
ALERT: /dev/mapper/vg0-root does not exist. Dropping to a shell.
Posted by VPSPowerUser, 09-26-2013, 06:02 AM

Quote:

Originally Posted by LeaseWeb - Jemma

Dear all,

Our Virtual Server platform is stable, but we are seeing a number of customers having trouble accessing or booting their instance.

If your instance is not accessible right now, please issue a reboot command via the Self Service Center : https://secure.leaseweb.nl

Instances that are not functioning correctly after a reboot, we will be fixing them on a case by case basis. We would like to ask you to open a new support ticket if you are still encountering problems, so we can handle your issue as efficiently as possible.

Kind regards,
LeaseWeb
http://www.leasewebnoc.com

'
I have seen similar message since yesterday. How it can be stable, if its unable to work. Reboot command via the Self Service Center is useless. nothing happens.
Posted by iCode, 09-26-2013, 06:06 AM
It'd be interesting to see if they do actually come online within the next 30 minutes
Posted by VPSPowerUser, 09-26-2013, 06:08 AM

Quote:

Originally Posted by pat010

I have exactly the same problem as you. However i just been able to get into scc remote console.
The error im getting is :
ALERT: /dev/mapper/vg0-root does not exist. Dropping to a shell.

I had just a white empty page at console link. I have black empty console now. Nothing happens
Posted by pat010, 09-26-2013, 06:10 AM

Quote:

Originally Posted by iCode

It'd be interesting to see if they do actually come online within the next 30 minutes

Well, il post an update in 30 minutes. Ive been awake for 24 hours now checking my server every friggin 10 minutes anyway.

To be honest i have this weird feeling i might have lost all my data and they are unable to fix the issue

Hopefully im just being paranoid.
Posted by pat010, 09-26-2013, 06:12 AM

Quote:

Originally Posted by VPSPowerUser

I had just a white empty page at console link. I have black empty console now. Nothing happens

I get that to but if you wait 30 seconds and press a key with the focus on the black screen my console apears.

I would be curious to know what the error is on your part.
If its the same as mine..

Dont reboot though, they might be working on it right now. I HOPE!
Posted by VPSPowerUser, 09-26-2013, 06:16 AM

Quote:

Originally Posted by pat010

Well, il post an update in 30 minutes. Ive been awake for 24 hours now checking my server every friggin 10 minutes anyway.

To be honest i have this weird feeling i might have lost all my data and they are unable to fix the issue

Hopefully im just being paranoid.

Very realistic scenario.
Posted by VPSPowerUser, 09-26-2013, 06:21 AM

Quote:

Originally Posted by pat010

I get that to but if you wait 30 seconds and press a key with the focus on the black screen my console apears.

I would be curious to know what the error is on your part.
If its the same as mine..

Dont reboot though, they might be working on it right now. I HOPE!

I got web based console now. I have the same error all the time - file system with errors.

I guess, thats some kind bureaucratic game with all of us. Since the service went down - its shows file system with errors. Nothing has changed. This leads me to the conclusion that their data storage has burnt down etc and they are unable to restore it.

I have attache an image for you.
Posted by pat010, 09-26-2013, 06:27 AM
Seems we have the same issue. I wonder how many people have this problem as they state a few. I have a feeling it's allot more then a "few"

Have they replied to any tickets you made after the last NOC update of 11:40 ?
Posted by VPSPowerUser, 09-26-2013, 06:35 AM

Quote:

Originally Posted by pat010

Seems we have the same issue. I wonder how many people have this problem as they state a few. I have a feeling it's allot more then a "few"

Have they replied to any tickets you made after the last NOC update of 11:40 ?

No, they haven't. Anyhow I have tried live chat. They promised approx. 40 minutes ago, that service should be available within 30 minutes.

Quote:

[11:54 AM] j.bruijns: It should come online in the next 30 minutess. We are currently having issues with the console servers (these are the ones thet process the reboots). They are under an extremely heavy load at the moment. They should be fixed within the next 30 minutes.

As the error has been the same since the first moment, I suspect that they have lost their data storage and are unable to restore it.
Posted by aeris, 09-26-2013, 06:37 AM

Quote:

Originally Posted by VPSPowerUser

This leads me to the conclusion that their data storage has burnt down etc and they are unable to restore it.

FWIW, mine are currently up and have been since yesterday evening, with all data intact. My guess would be corrupt file systems on individual containers, which may or may not be recoverable.
Posted by pat010, 09-26-2013, 06:49 AM

Quote:

Originally Posted by aeris

FWIW, mine are currently up and have been since yesterday evening, with all data intact. My guess would be corrupt file systems on individual containers, which may or may not be recoverable.

Im happpy for you they were able to solve it.
Did you expirience the same errors as us?
Posted by aeris, 09-26-2013, 06:58 AM

Quote:

Originally Posted by pat010

Im happpy for you they were able to solve it.
Did you expirience the same errors as us?

One of my instances was offline for several hours, but came back online around the time they initially announced that the issues were resolved yesterday. The other had severe I/O delays, but was never entirely down.

Neither instance had any kind of corruption afterwards, and those that are offline now seem to be suffering from file system corruption on the instance itself, which will require manual intervention. Judging by the screenshots posted by @VPSPowerUser you may be able to run a fsck from the console and get it back online yourself, but there are no guarantees that it will be in a bootable state afterwards, so you may want to wait for LSW to get back to you if you have any important data.
Posted by VPSPowerUser, 09-26-2013, 07:02 AM

Quote:

Originally Posted by pat010

Im happpy for you they were able to solve it.
Did you expirience the same errors as us?

Its obvious that they are unable to manage the situation well. They have approx. 6200 support tickets opened since yesterday. Within last hour 255 tickets. Thats why they are asking to open new tickets, they are unable to manage these 6000 tickets. They don't care actually properly about old tickets for sure.
Posted by neanton, 09-26-2013, 07:03 AM
My issue is not the same. Here is what I see.

Yesterday my system showed GRUB loader with a kernel selection options. I do not want to loose my data...

BTW. They did reply to my ticket with a standard text like: "Please update this ticket or write a new one if you still experience a problem"
Posted by VPSPowerUser, 09-26-2013, 07:05 AM

Quote:

Originally Posted by neanton

My issue is not the same. Here is what I see.

Yesterday my system showed GRUB loader with a kernel selection options. I do not want to loose my data...

Please attach your screenshot
Posted by neanton, 09-26-2013, 07:13 AM
Forgot the screenshot
Posted by VPSPowerUser, 09-26-2013, 07:15 AM

Quote:

Originally Posted by neanton

Forgot the screenshot

In general you have the same issue actually.
Posted by VPSPowerUser, 09-26-2013, 07:25 AM
Seems I'm banned somehow for Live Chat support for now.
Posted by pat010, 09-26-2013, 07:41 AM
I gues the 30 minute time table they gave you was a bit optimistic huh, VPSPowerUser?

Anyway, im still down to. And i also got kicked from live chat.
Havent used it since yesterday though. So i dont think it's a "ban"

Either way, this is getting seriously messed up.

Worst case scenario, servers needs to be re-installed.
Will they be able to fetch my data though? Or is it also possible the data is lost forever ?

I do have backups, but we had a shitload of orders in the 24 hours before the **** hit the fan and my backup was sheduled an hour after everything went down.
Posted by VPSPowerUser, 09-26-2013, 07:48 AM

Quote:

Originally Posted by pat010

I gues the 30 minute time table they gave you was a bit optimistic huh, VPSPowerUser?

Anyway, im still down to. And i also got kicked from live chat.
Havent used it since yesterday though. So i dont think it's a "ban"

Either way, this is getting seriously messed up.

Worst case scenario, servers needs to be re-installed.
Will they be able to fetch my data though? Or is it also possible the data is lost forever ?

I do have backups, but we had a shitload of orders in the 24 hours before the **** hit the fan and my backup was sheduled an hour after everything went down.

Hard to tell about the availability of data. Anyhow the company isn't honest with us. I have asked about status of data in tickets, but they have ignored this question.

Live Chat shows than I'm in order, When comes my order, I'm kicked out.
Posted by Mikeambrose3, 09-26-2013, 07:55 AM
You should always have a backup you can easily deploy with another server/company if your primary server is offline. If leaseweb is having a hard time dealing with a mass amount of tickets, give them time. People complaining just delays them, and takes away their time to fix the issues. Even with a good amount of staff, 6,000+ tickets is a lot to handle within a 2 day period.
Posted by pat010, 09-26-2013, 08:11 AM

Quote:

Originally Posted by Mikeambrose3

You should always have a backup you can easily deploy with another server/company if your primary server is offline. If leaseweb is having a hard time dealing with a mass amount of tickets, give them time. People complaining just delays them, and takes away their time to fix the issues. Even with a good amount of staff, 6,000+ tickets is a lot to handle within a 2 day period.

Yes, this has been a wake up call for me. I never really had any important websites until i started a small bussines with suddenly grew extremely fast but sadly never really thought properly how to go about setting it up besides the basics. As soon as this is over im going to make some serious changes.

To bad i had to learn the hard way
Posted by VPSPowerUser, 09-26-2013, 08:18 AM

Quote:

Originally Posted by Mikeambrose3

You should always have a backup you can easily deploy with another server/company if your primary server is offline. If leaseweb is having a hard time dealing with a mass amount of tickets, give them time. People complaining just delays them, and takes away their time to fix the issues. Even with a good amount of staff, 6,000+ tickets is a lot to handle within a 2 day period.

If company will be honest, ti will help everyone. Both, the company and me. But unfortunately if company isn't fully open about the case and gives misleading information it also confuses me. They are telling that service is fine, and if you have still issues, send support ticket. I'm opening ticket, but no one takes care. If I contact from chat, they tell that the service will be available within 30 minutes. Nothing happens. If I'm trying again live chat, I'm kicked out. Who confuses who?
Posted by VPSPowerUser, 09-26-2013, 08:22 AM
They are on emergency now.

Quote:

Originally Posted by LeaseWeb

[Global, NL, DE, US, AMS-01, AMS-10, AMS-11, AMS-12, FRA-10, WDC-10] SSC emergency maintenance
September 26, 2013 14:15:00 (CEST)
Dear customer,

We will have to perform emergency maintenance on the Self Service Center platform. To be able to perform this the service will be disabled between 14:15 and 14:30.

All entities will be impacted.

Sorry for any inconvenience caused.

Sincerely,

LeaseWeb
http://www.leasewebnoc.com

Really, LeaseWeb, what the hell is going on? Some honest explanation available?
Posted by pat010, 09-26-2013, 08:28 AM
Perhaps to many people keep rebooting there vps while there trying to work on it. So they just shut down the ssc. I dont know.

BTW: Im absolutly no expert but isent it basicly impossible to loose data with raid 10? As i understand raid10 even if they cant get the server back up, the data should be safe?

I hope..
Posted by aeris, 09-26-2013, 08:36 AM

Quote:

Originally Posted by pat010

BTW: Im absolutly no expert but isent it basicly impossible to loose data with raid 10? As i understand raid10 even if they cant get the server back up, the data should be safe?

Not really. RAID-10 only protects against disk-level hardware faults, which, granted, is the most common cause of failure on platter-based storage. It adds resilience, but is in no way, shape or form a substitute for backups.
Posted by VPSPowerUser, 09-26-2013, 08:36 AM

Quote:

Originally Posted by pat010

Perhaps to many people keep rebooting there vps while there trying to work on it. So they just shut down the ssc. I dont know.

BTW: Im absolutly no expert but isent it basicly impossible to loose data with raid 10? As i understand raid10 even if they cant get the server back up, the data should be safe?

I hope..

It's untrue. There is no any warrant that if you use VPS that they take care about your data proper way and it's stored some fail safe system. Anyhow, if there is "hardware failure" they don't have your data or maybe even if they have, they are not willing to restore it, as it is too expensive to do it. So you, as the customer are always responsible whatever they do!!! Thats the hard truth about VPS and Cloud.
Posted by thehosterdude, 09-26-2013, 08:54 AM

Quote:

Originally Posted by aeris

At this point, every single "cloud" I've ever tried has had severe downtime-inducing problems with its SAN at some point or another. Which is why I still greatly prefer dedicated servers with application-level redundancy. Not putting all your eggs in one basket, and all that.

you couldn't have said it better . . .
Posted by pat010, 09-26-2013, 08:58 AM
IM BACK ONLINE

THANK GOD! And leaseweb for not giving up!

You guys back online to?
Posted by VPSPowerUser, 09-26-2013, 08:58 AM
Seems that LeaseWeb VPS is up now. What is the actual status is uncertain yet.

Quote:

LeaseWeb VPS is back UP (Host Is Reachable) (It was down for 33 hours, 29 minutes and 32 seconds).

Posted by pat010, 09-26-2013, 09:05 AM
ROFL....

So, i logged in super fast. Executed my backup script and BAM:

The system is going down for halt NOW!
Power button pressed

ROFL
Posted by VPSPowerUser, 09-26-2013, 09:10 AM

Quote:

Originally Posted by pat010

ROFL....

So, i logged in super fast. Executed my backup script and BAM:

The system is going down for halt NOW!
Power button pressed

ROFL

wtf? Maybe they haven't solved issue yet. I will keep hands away for an hour, to see how stable it is. Then I'm going to check the status of data.
Posted by neanton, 09-26-2013, 09:14 AM
Mine still does not boot..
Posted by pat010, 09-26-2013, 09:15 AM
I hear ya. But good news nonetheless. Atleast we know our data is still there. I dont know about you, but i couldent be more happy right now.

Il give it an hour before i do anything. Meanwhile im going to reply to what seems to be about 60+ email complaints from my customers. Yippie :/
Posted by VPSPowerUser, 09-26-2013, 09:19 AM

Quote:

Originally Posted by neanton

Mine still does not boot..

A least you have very big opportunity that you haven't lost your data

I recommend to open a new support ticket, don't forget to give root credentials as if they need those, they don't have to ask again you.
Posted by pat010, 09-26-2013, 09:38 AM
UPDATE:

Were not home yet...

Every command gives a BUS error.

Example:

sniff@butt:/home# ls
Bus error

Trying to stop/start a service results in seg faults.

/cry
Posted by VPSPowerUser, 09-26-2013, 09:47 AM

Quote:

Originally Posted by pat010

UPDATE:

Were not home yet...

Every command gives a BUS error.

Example:

sniff@butt:/home# ls
Bus error

Trying to stop/start a service results in seg faults.

/cry

I just opened a new ticket to get from them an official status report. I'm not doing anything on VPS until I get confirmation, that they have finished everything and I'm safe to use VPS.
Posted by ltlc, 09-26-2013, 09:54 AM
Our 3 virtual servers have been down since yesterday morning. Yesterday afternoon 1 came online shortly but kept giving disk i/o errors.

Now none of them will boot. All servers show the same error (in console access):

"No bootable device. Retrying in 60 seconds."
Posted by VPSPowerUser, 09-26-2013, 10:00 AM

Quote:

Originally Posted by ltlc

Our 3 virtual servers have been down since yesterday morning. Yesterday afternoon 1 came online shortly but kept giving disk i/o errors.

Now none of them will boot. All servers show the same error (in console access):

"No bootable device. Retrying in 60 seconds."

Since 2013-09-25 03:20:59 (London) VPS went went down. It's up now for 55 minutes. Still waiting for official confirmation that service is restored completely and we can continue with regular usage.

I recommend to close all your previous tickets and to open new one. Or try live chat. Provide screenshots from console and root access, so they can bring it up for you. Don't use it before you get official confirmation that they have done and you are welcome to use it as normal.

Please note, LeaseWeb gets approx. 250 support tickets per hour at the moment as I can see.
Posted by VPSPowerUser, 09-26-2013, 01:31 PM
Our VPS up for 4 Hours 29 Minutes for now. Anyhow, some slow responses are noticed.
Posted by ltlc, 09-26-2013, 01:37 PM
Mine are still down, with no change in 36h. LeaseWeb helpdesk can't give any more information over the phone and tickets are not being answered.

Since a lot of time has passed I am really doubting they still have the data of my servers, although the helpdesk has given me a 100% guarantee all servers will be restored.
Posted by pat010, 09-26-2013, 01:41 PM
I know it sucks but your probably safe. Seeing there getting about 250 tickets an hour allot of people are still down. And they have to fix the servers 1 by 1. With any luck you will be online soon. With a bit of bad luck you maybe last in line and have to wait for hours and hours.

The only good thing to take away from this is that your most probably safe and your servers will be restored. It's just a matter of when they get to your server.
Posted by VPSPowerUser, 09-26-2013, 01:45 PM

Quote:

Originally Posted by ltlc

Mine are still down, with no change in 36h. LeaseWeb helpdesk can't give any more information over the phone and tickets are not being answered.

Since a lot of time has passed I am really doubting they still have the data of my servers, although the helpdesk has given me a 100% guarantee all servers will be restored.

We had downtime from 2013-09-25 03:20:59 to 2013-09-26 12:59:42 = 1 Days 9 Hours. We have lost 2 business days. Actually almost 3 as we have to take care tomorrow for some related coincident events.

I haven't got any response to my ticket to clarify the actual status of VPS service to make sure that they have finished all restoration procedures. I guess they have serious lack of competent personnel to handle all these support request (approx. 250 per hour).
Posted by feastboy, 09-26-2013, 01:51 PM
lolz, down for 36 hour.

we're choosing cloud mainly for reliable, but this makes us to think twice before choosing cloud provider.

i understood, cloud is a marketing gimmick word now, so it's also not reliable.

guys will later release some another marketing word "Super cloud" or something to attract users..
Posted by VPSPowerUser, 09-26-2013, 01:55 PM

Quote:

Originally Posted by pat010

I know it sucks but your probably safe. Seeing there getting about 250 tickets an hour allot of people are still down. And they have to fix the servers 1 by 1. With any luck you will be online soon. With a bit of bad luck you maybe last in line and have to wait for hours and hours.

The only good thing to take away from this is that your most probably safe and your servers will be restored. It's just a matter of when they get to your server.

Seems that they are really restoring one by one each service. Sadly they aren't quite opened about actual issues. A bit less bureaucratic style of communication wouldn't harm. Suspicious waiting is worse than knowing the truth - they are restoring data one by one, and you have 50/50 chance or 100%, but with significant downtime (2-7 days).

Quote:

Dear Customer,

Some customers are still experiencing issues with their instances on the LeaseWeb Cloud and Virtual Servers.

[UPDATE 20130926-16:55]
We are still working on fixing individual instances on a one-by-one basis. Most customers are back online, but fixing the remaining instances is taking longer than expected. Implementing the fixes is taking place in a dedicated team, and in the mean time working on getting a good estimate of the amount of time required to fix these last instances to give a better indication on solve time.

We apologize for the impact this is having on your infrastructure, and are working hard and fast to get all customer instances online as soon as possible.

Sincerely,

LeaseWeb
http://www.leasewebnoc.com

Data center naming system is explained here: http://www.leaseweb.com/en/about-us/data-centers

Posted by VPSPowerUser, 09-26-2013, 02:00 PM

Quote:

Originally Posted by feastboy

lolz, down for 36 hour.

we're choosing cloud mainly for reliable, but this makes us to think twice before choosing cloud provider.

i understood, cloud is a marketing gimmick word now, so it's also not reliable.

guys will later release some another marketing word "Super cloud" or something to attract users..

You are correct, its obvious for me for now too, that Cloud and VPS are dangerous way to manage need of web connected server. Independent server (rented or owned) with full backup of entire server and data is a way to go.
Posted by feastboy, 09-26-2013, 02:01 PM
@VPSPowerUser
they don't have droplet like thing..?
put the droplet in another vps..boom restart, everything fine like that

because 36 hour ++++ counting is not acceptable, some time users lose lot of money, also business
Posted by VPSPowerUser, 09-26-2013, 02:16 PM
Just some speed checks:

Quote:

dd if=/dev/zero of=/tmp/output conv=fdatasync bs=384k count=1k; rm -f /tmp/output
1024+0 records in
1024+0 records out
402653184 bytes (403 MB) copied, 10.341 s, 38.9 MB/s

Actual write speed - 38.9 MB/s. Anyhow, some caching used, which gives with caching this output

Quote:

dd if=/dev/zero of=/tmp/output bs=384k count=1k; rm -f /tmp/output
1024+0 records in
1024+0 records out
402653184 bytes (403 MB) copied, 3.6748 s, 110 MB/s

Posted by VPSPowerUser, 09-26-2013, 02:36 PM

Quote:

Originally Posted by feastboy

@VPSPowerUser
they don't have droplet like thing..?
put the droplet in another vps..boom restart, everything fine like that

because 36 hour ++++ counting is not acceptable, some time users lose lot of money, also business

You couldn't "drop" anything, if its on failed data storage
Posted by webmajster, 09-26-2013, 03:06 PM
Some minutes ago good news arrived - ticket was answered and my cloud server is back online. There were some filesystem problems that I fixed in single mode.

All data seems intact.
Posted by wswd, 09-26-2013, 03:20 PM

Quote:

Originally Posted by LeaseWeb - Jemma

Dear all,

Our Virtual Server platform is stable...

It's stable, yet it has been down for over a day, and you're performing emergency maintenance on it?

Sounds stable!
Posted by VPSPowerUser, 09-26-2013, 03:21 PM

Quote:

Originally Posted by wswd

It's stable, yet it has been down for over a day, and you're performing emergency maintenance on it?

Sounds stable!

Well noticed!
Posted by VPSPowerUser, 09-26-2013, 05:58 PM
I got a message:

Quote:

Dear Sir/Madam,

We hereby would like to inform you about your VPS.

We are working hard to restore access to all servers. We have just
checked your VPS and it is currently back online and working as
expected.

We understand if you have questions about the outage and a reason for
outage will be provided shortly. However, we cannot provide you with
more information via this ticket, so please do not reply to this message
unless you still experience issues with your server.

Our apologies for the inconvenience caused.

Kind regards,

Sinan Polat
LeaseWeb - Customer Support

Posted by christianmat, 09-27-2013, 01:01 AM
It's been down now for 48 hrs. I'm literally close to just giving up. No response from support. No time horizon. Nothing. I simply thought LeaseWeb was a better company. By now, we've been forced to purchase another server with another company. And I don't think we'll renew with LeaseWeb. STAY AWAY from them. Clearly both an amateur hosting company and a lousy customer support!
Posted by VPSPowerUser, 09-27-2013, 03:46 AM

Quote:

Originally Posted by christianmat

It's been down now for 48 hrs. I'm literally close to just giving up. No response from support. No time horizon. Nothing. I simply thought LeaseWeb was a better company. By now, we've been forced to purchase another server with another company. And I don't think we'll renew with LeaseWeb. STAY AWAY from them. Clearly both an amateur hosting company and a lousy customer support!

Really sad to hear. The worst about the case is poor and non-open communication from company side. We don't like to be treated, we prefer honest communication. And thats where LeaseWeb fails at the moment for sure.

Seems that they don't have a valid disaster plan, especially how to keep good communication at the worst scenario.
Posted by neanton, 09-27-2013, 04:07 AM
WOOOOHHHHOOOO!!!

Mine VPS is back online. I received same ticket as VPSPowerUser!
All data is intact.
Time to make backups and prepare another failover VPS...

52h of outage...
Posted by LeaseWeb - Jemma, 09-27-2013, 04:08 AM
Dear all,

We have checked and fixed a couple of hundred more instances. Our support teams will continue to work on checking and fixing more instances.

Should you have issues with your instance, please try a full stop and start, as in a large amount of cases this fixes the problem.

Please update the tickets with correct server ID and the problem you are still facing. As all VPSs will be worked on by a case by case basis. Thank you.

Kind Regards,
LeaseWeb,
Posted by aeris, 09-27-2013, 10:36 AM
Well, so much for that. Yesterday both my instances started complaining about I/O issues, and while one of them restarted by itself and apparently fixed it automatically with fsck, the other is now stuck on "No boot device". Seeing as the latter never went offline throughout the failure, the corruption doesn't seem to be on the instance level, but rather on the backend storage itself.

(Stopping and starting did nothing as well, past making it being stuck in the "Stopping..." state.)
Posted by HaronMedia, 09-27-2013, 01:03 PM
Here's the official RFO http://blog.leaseweb.com/2013/09/27/...-cloud-outage/
Posted by ltlc, 09-28-2013, 09:35 AM
My last server is still offline. LeaseWeb told me by phone there is a great possibility all data is lost. Also they told me VPS servers are never being back-upped at their side.

All of their employees I spoke to before guaranteed me al data was still there, so either they lied or where deliberately mislead by LeaseWeb management. LeaseWeb is now trying to make me believe they are still working with a team to retrieve the data..

For me LeaseWeb stands for serious carelessness and lies now.

They have been a good supplier to us for years and I don't know what happened there, but we have had problems with colocated servers (cables are switched to the wrong network ports) and dedicated servers (with failing hard drives) this year. Now this. I guess it's time to move on..
Posted by HaronMedia, 09-28-2013, 02:21 PM

Quote:

Originally Posted by ltlc

I don't know what happened there

Did you read their official explanation? It's linked one post above.
Posted by ironfist, 09-28-2013, 04:17 PM
Eeeeh? So they designed a storage that totally breaks down at 70% capacity?
Posted by christianmat, 09-29-2013, 04:54 AM
After 5 days of down time, LeaseWeb finally replied to my ticket(s).

All data is lost. They offered me a reinstall.

Posted by feastboy, 09-30-2013, 10:03 AM
@christianmat
so they're not experoenced hosting company, you have backup or lost everything.......??
Posted by alainsteinmann, 09-30-2013, 12:32 PM
Do we know how many accounts were impacted ? The blog post doesn't tell so much...
Posted by ltlc, 09-30-2013, 08:12 PM
@christianmat
Unbelievable. So what would they mean by "Redundant storage for all data" on their website.

Mine is still down and no word from LeaseWeb…
Posted by christina21maria, 10-02-2013, 11:59 PM
I know that profile picture, she worked for me on odesk last year for a brief period of time