Softlayer Cloud down again

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Softlayer Cloud down again

Posted by gdtechind, 11-28-2009, 03:49 PM
Few weeks back, Softlayer had downtime for CloudLayer instances for two days.

I am wondering if anyone else is facing the trouble again ? I have just now noticed that the Cloud is down and we aren't able to connect for past 15 minutes.

I hope they won't start doing it as a monthly thing.
Posted by S2 Web Design, 11-28-2009, 04:35 PM
We just signed up for a CCI Public SAN solution yesterday to check it out and compare. There have been high I/O wait times noticed since signing up and we're on a RHEL 5 minimal install. Just an hour ago, we noticed the file system go into read-only status and the whole environment is now down without any communication from SoftLayer on what is happening.
Posted by gdtechind, 11-28-2009, 10:26 PM
yes, the same thing happened with my instance. First the read only error on databases.

After that the instance went down.
Posted by Scott.Mc, 11-29-2009, 12:24 AM
Out of interest what sort of read and write times are you getting from your instances? Ours have been terrible.
Posted by gdtechind, 11-29-2009, 04:11 AM
the instance was up after nearly 7 hours. Their support staff was doing some funny things and their responses were also strange.
Posted by CGotzmann, 11-29-2009, 01:17 PM
Cloud instances are not really meant for disk usage, IMO with the current technology.
They should be considered as general "storage" and for Processing/Memory resources.

Anything with heavy disk I/O is better kept on dedicated servers (IE: MySQL).
Posted by Scott.Mc, 11-29-2009, 02:11 PM

Quote:

Originally Posted by CGotzmann

Cloud instances are not really meant for disk usage, IMO with the current technology.
They should be considered as general "storage" and for Processing/Memory resources.

Anything with heavy disk I/O is better kept on dedicated servers (IE: MySQL).

Well that would depend on which "cloud" you use. For the record I fully agree, these types of "clouds" are designed for lots of smaller processing however many of them are being marketed that they will replace existing systems to users. Further many are nothing more than a VPS with a pretty interface.

Anyway the question I was asking is what sort of read/write speeds others receive from SL's cloud platform. I was struggling to write more than 9MB/s and simply given up processing the logs (IE days to process the logs rather than 1-2 hours). I have used many products but that is by far the worst of the lot.
Posted by barry[CoffeeSprout], 11-30-2009, 09:23 AM

Quote:

Originally Posted by Scott.Mc

Well that would depend on which "cloud" you use. For the record I fully agree, these types of "clouds" are designed for lots of smaller processing however many of them are being marketed that they will replace existing systems to users. Further many are nothing more than a VPS with a pretty interface.

Anyway the question I was asking is what sort of read/write speeds others receive from SL's cloud platform. I was struggling to write more than 9MB/s and simply given up processing the logs (IE days to process the logs rather than 1-2 hours). I have used many products but that is by far the worst of the lot.

Unfortunately I have to agree.
The product looks nice on paper and while I understand the limitations on IO, It's unacceptable that you can't even run a minimal MySQL config on them or have the filesystem turn read only from time to time.

I've run with it for a few months and then moved to other solutions.
Softlayer is great, but they really need to look at their cloud a bit more
Posted by eming, 11-30-2009, 11:41 AM
could it be this thing?

Quote:

.....

Emergency Data Center Maintenance [11/30/2009]
Date: Monday, November 30, 2009 (11/30/2009)
Services affected: Public Network
Devices affected: FCR01.DAL01, FCR02.DAL01, FCR03.DAL01, FCR04.DAL01, FCR01.WDC01, FCR01.SEA01

==================================================

The following notification is to inform you of an EMERGENCY MAINTENANCE event on the SoftLayer Technologies IP Network. Due to the severity, the time frame between this notice and the maintenance window has been shortened.

Event Times:

12:00 CST / 01:00 EST - FCR01.WDC01
12:30 CST - FCR01.DAL01
01:00 CST - FCR02.DAL01
01:30 CST - FCR03.DAL01
02:00 CST - FCR04.DAL01
02:30 CST / 12:30 PST - FCR01.SEA01

Summary:
SoftLayer Technologies in conjunction with Cisco Systems have identified a service impacting bug in the current IOS version running on this router. A vendor recommended upgrade has been provided as a resolution and needs to be implemented immediately.

Original Issue Reference:
With the introduction of 4-byte ASNs (Autonomous System Numbers), a newly devised method of injecting a specifically crafted prefix into the global route table allows for the BGP process to be reset on older IOS code.

http://www.cisco.com/warp/public/707...0729-bgp.shtml
https://forums.softlayer.com/showthread.php?t=4585

Customer Impact:
During this maintenance, customers will notice a complete loss of connectivity to their servers on the frontend network (public network). Backend network (private network) connectivity will NOT be impacted during this maintenance. While the upgrade duration is scheduled for 1 hour, we only expect between 15-20 minutes of downtime as the routers reload and fully converge. Again, this will NOT impact the backend network (private network) for customer servers.

A notice will also be posted in the portal prior to the maintenance, during the maintenance informing you of the progress, and after with completion details.

==================================================

If you have any problems with your connection to your services after this maintenance window, or if you have any questions regarding the maintenance at any point, please call us at 214-442-0603 or open a ticket in the customer portal.

We appreciate your patience during this maintenance and welcome any feedback you may have.

Sincerely,
SoftLayer Network Team
https://status.softlayer.com

Posted by Scott.Mc, 11-30-2009, 11:44 AM

Quote:

Originally Posted by eming

could it be this thing?

No that is the network outage (actually completed this morning), what we were talking about is fundamentally the product itself is useless with how slow the IO is. Even amazon is around 5x's the speed of it and you would think that their SAN with the number of disks (and in turn spindles) it would be much faster than it actually is.
Posted by atjeu, 11-30-2009, 04:15 PM
I heard amazon is using Citrix Xenserver for their whole cloud as well.
Posted by Scott.Mc, 11-30-2009, 05:46 PM

Quote:

Originally Posted by atjeu

I heard amazon is using Citrix Xenserver for their whole cloud as well.

Almost all are the same basic design, xen, san and a pretty interface.
Posted by S2 Web Design, 11-30-2009, 11:04 PM
For the 3rd time in 3 days of having my SoftLayer CCI instance, the filesystem has again gone read-only. I am not even running anything on it...only monitoring it because of this. I'm afraid I will have to be asking for my payment to be applied in full towards another solution with them. If that can't be done, then a refund. If that can't be done, I will have to issue a last attempt chargeback because this is ridiculous.
Posted by ibee, 12-01-2009, 12:15 AM
Can anyone suggest Google App Engine for cloud? we have a 1000 GB data website with having more than 15K unique visitors per day website.
Posted by gdtechind, 12-01-2009, 06:35 AM

Quote:

Originally Posted by S2 Web Design

For the 3rd time in 3 days of having my SoftLayer CCI instance, the filesystem has again gone read-only. I am not even running anything on it...only monitoring it because of this. I'm afraid I will have to be asking for my payment to be applied in full towards another solution with them. If that can't be done, then a refund. If that can't be done, I will have to issue a last attempt chargeback because this is ridiculous.

I am sure they will issue a refund as they have really made a mess this time.

Today, the downtime has already crossed 8 hours.