Cartika Websites Down. Reseller Alternatives?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Cartika Websites Down. Reseller Alternatives?

Posted by gbcactus, 08-10-2011, 02:09 PM
All of my websites on Cartika Hosting have been down for more than two hours and customers are upset. The data center in Dallas, Texas is down and there's no backup system. Does anyone have recommendations for a reseller host that has dependable backup systems to prevent websites from going down?

Thanks, Gary
Posted by trustedurl.com, 08-10-2011, 02:18 PM
Most reseller accounts do not provide automatic fail-over between data centers... you could setup a custom system for yourself, but that would be much more costly than the reseller account you have now.

Why is the noc down anyways?
Posted by gbcactus, 08-10-2011, 02:22 PM
See this post: Colo4 Service Entrance 2 issue

http://www.webhostingtalk.com/showthread.php?t=1072692
Posted by trustedurl.com, 08-10-2011, 02:24 PM
Hmm, just read that, not good at all, but this seems to be a rare problem. I guess you have to weight if the increased cost would be worth it; you would basically have to have failover setup with an active standby in a different noc.
Posted by gbcactus, 08-10-2011, 02:29 PM
One of my clients suggested Rackspace cloud hosting. I'll call them this morning.
Posted by trustedurl.com, 08-10-2011, 02:39 PM

Quote:

Originally Posted by gbcactus

One of my clients suggested Rackspace cloud hosting. I'll call them this morning.

I would research that thoroughly before jumping ship; I recall some issues with that option.
Posted by layer0, 08-10-2011, 02:51 PM

Quote:

Originally Posted by gbcactus

One of my clients suggested Rackspace cloud hosting. I'll call them this morning.

Do a search on this forum. Rackspace has had power issues of their own in the past too, and even an instance where they lost data on a fairly significant segment of cloud servers.

If you're going to start jumping ship over just one issue which is out of Cartika's control anyhow (just check the outage forum to see that many other providers are affected by the colo4 issue), you're going to be jumping ship from provider to provider for a long time, as these kinds of things can happen anywhere.
Posted by KMyers, 08-10-2011, 02:53 PM

Quote:

Originally Posted by gbcactus

One of my clients suggested Rackspace cloud hosting. I'll call them this morning.

Hello,
If this is the first time you have seen downtime, I would not jump ship. Downtime happens regardless of the setup/location. Moving sites can cause more of a headache at times. With this said, if downtime is a constant, then run (dont walk) to the nearest exit. Cartika is one of the most trustworthy hosts on WHT, so I would stick with them unless its everyday,.
Posted by gbcactus, 08-10-2011, 03:16 PM
This comment was posted on the Cartika forum by a Cartika staff member.

"Firstly, the fact that our primary network is up and several of our infrastructure is up indicates that we are indeed using A+B power. We just arent using it on all of our infrastructure - frankly, it is very expensive and people already complain that our $10 shared hosting is too expensive."

I'd like a host that has backup power. My sites have now been down for three hours and customers are mad.
Posted by layer0, 08-10-2011, 03:24 PM

Quote:

Originally Posted by gbcactus

This comment was posted on the Cartika forum by a Cartika staff member.

"Firstly, the fact that our primary network is up and several of our infrastructure is up indicates that we are indeed using A+B power. We just arent using it on all of our infrastructure - frankly, it is very expensive and people already complain that our $10 shared hosting is too expensive."

I'd like a host that has backup power. My sites have now been down for three hours and customers are mad.

I think you're misunderstanding what A/B power is vs. backup power.

Cartika and any host in Colo4 has backup power - that constitutes a UPS and generators. Currently there is an issue with a transfer switch from my understanding, causing an outage. But that doesn't mean there is no backup power.

Some customers in Colo4's facility go an extra mile and purchase A/B power which means there's power from two separate PDUs. But each individual unit is still backed by UPS and generators. A/B power is extremely expensive and you're going to be hard pressed to find too many hosts that are using this. And even if they are, it's possible they are impacted by issues like this (like some Colo4 customers with A/B power that are still down).

I understand you're frustrated, but I think you need to put this into perspective. Issues like this can happen almost anywhere. It's unfortunate that it happened at Colo4 and is affecting Cartika, but you should know that Colo4 is a reputable operation that has had very few issues over the last several years. We have a few systems on Colo4's network that have experienced 99.99%+ uptime over the years. This is the first time we're seeing an extended issue.
Posted by gbcactus, 08-10-2011, 03:28 PM
The Cartika website is up because it has A+B power. My sites are still down. Customers are mad. Some don't have email which is essential to their business. I need a more dependable host.
Posted by layer0, 08-10-2011, 03:37 PM

Quote:

Originally Posted by gbcactus

The Cartika website is up because it has A+B power. My sites are still down. Customers are mad. Some don't have email which is essential to their business. I need a more dependable host.

And some other Cartika servers with A/B power are still down, the same with some other providers. A/B power doesn't mean you can't possibly be affected by this issue.

Cartika has been communicating about the outage with their customers as best as they possibly can, and this problem is largely out of their control. They've been doing business with Colo4 for years who has been providing them with a stable facility throughout this time. They obviously couldn't have expected to experience an issue like that of today, but you should at least understand that it can happen with almost any provider.

Just because a host is having a single significant outage (the first in years) doesn't mean they are not dependable. Because it's not a matter of IF there's going to be an outage, it's a matter of WHEN. Nothing is 100%. Cartika is being completely 'dependable' about this in my book - they're doing everything in their power to make sure all of their customers are kept informed, and I have no doubt that they will work with Colo4 to prevent something like this from happening again.
Posted by trustedurl.com, 08-10-2011, 04:00 PM

Quote:

Originally Posted by layer0

as these kinds of things can happen anywhere.

Agreed, though I can definitely understand why it's frustrating.
Posted by Krishopper, 08-10-2011, 06:05 PM
Cartika is using Hsphere I believe. If you are looking for an Hsphere reseller so that you don't have a huge learning curve with something new, there are definitely options out there.

While complete failover may not be an option for them, its unfortunate that they don't have a disaster recovery plan to alleviate issues like this. All hosts should have a master escape plan for issues like that.
Posted by cartika-andrew, 08-10-2011, 08:08 PM

Quote:

Originally Posted by Krishopper

Cartika is using Hsphere I believe. If you are looking for an Hsphere reseller so that you don't have a huge learning curve with something new, there are definitely options out there.

While complete failover may not be an option for them, its unfortunate that they don't have a disaster recovery plan to alleviate issues like this. All hosts should have a master escape plan for issues like that.

Thank you for your feedback.

Would you mind outlining what plan would be reasonable to fail over within x hours when a datacenter loses power? I would be very interested in learning what solution could be delivered to end users...

We are launching plans which will allow customers with cloud servers to fail over between data centers in case of such failures, but, the price tag is in the $1000s per month. i am curious and quite interested to hear what solutions are possible for sub $100 shared reseller accounts that will allow for customers to fail over between datacenters in a matter of minutes in case of a failure at a data center level?
Posted by FRCorey, 08-10-2011, 09:10 PM
A+B power is not that expensive, maybe a few hundred dollars if you're buying your own servers, and another 10 dollars per moth for the extra port. Even if you deploy your own servers with single power supplies you can buy rack mount PDU's that allow a single plug to be powered by A+B power.
Posted by erickmiller, 08-10-2011, 09:23 PM

Quote:

"Firstly, the fact that our primary network is up and several of our infrastructure is up indicates that we are indeed using A+B power. We just arent using it on all of our infrastructure - frankly, it is very expensive and people already complain that our $10 shared hosting is too expensive."

I agree. Redundancy costs money, especially power. When people aren't willing to pay for it, they can't expect their systems to be up when major problems happen in the power infrastructure, especially with a critical single point of failure like an automatic transfer switch.

Eric
Posted by erickmiller, 08-10-2011, 09:35 PM

Quote:

Originally Posted by cartika-andrew

i am curious and quite interested to hear what solutions are possible for sub $100 shared reseller accounts that will allow for customers to fail over between datacenters in a matter of minutes in case of a failure at a data center level?

Me too. Redundancy costs money (power, network connectivity, DNS fail-over and/or BGP redirection, etc.), and most aren't willing to pay for, and thus support, the concepts around redundant systems. Many people (not all!) are generally willing to pay for a plan that is cheaper, simply because of the cost, not considering the risk they are taking, so it's not a great support system for redundancy.

Having options for redundancy at the infrastructure layer is a great thing, though. If anything, it shows what it actually costs to provide this type of service. Customers will at least be able to choose how much risk avoidance they are willing to pay for.

Eric
Posted by cartika-andrew, 08-10-2011, 10:42 PM

Quote:

Originally Posted by erickmiller

Me too. Redundancy costs money (power, network connectivity, DNS fail-over and/or BGP redirection, etc.), and most aren't willing to pay for, and thus support, the concepts around redundant systems. Many people (not all!) are generally willing to pay for a plan that is cheaper, simply because of the cost, not considering the risk they are taking, so it's not a great support system for redundancy.

Having options for redundancy at the infrastructure layer is a great thing, though. If anything, it shows what it actually costs to provide this type of service. Customers will at least be able to choose how much risk avoidance they are willing to pay for.

Eric

thank you Eric and an absolutely outstanding post. We recently launched a solution where users could distribute accounts across various data centers - but, we do not yet have a solution to allow accounts to seamlessly fail across datacenters. We do have a cloud based solution that will allow for that, but the price tag is steep. Consider that this facility has failed for the first time in 8 years and one has to wonder if it is worth the investment - end of the day though, people will complain with any outage and we understand that - but ask these same people to pay double or triple their existing rate so that once in every 8 years they can avoid an outage - and 99% of them would not be willing to pay the rate required.. its an interesting business we have chosen here
Posted by cartika-andrew, 08-10-2011, 10:46 PM

Quote:

Originally Posted by FRCorey

A+B power is not that expensive, maybe a few hundred dollars if you're buying your own servers, and another 10 dollars per moth for the extra port. Even if you deploy your own servers with single power supplies you can buy rack mount PDU's that allow a single plug to be powered by A+B power.

the cost for legitimate A+B power is not "a few hundred dollars"... or "10 dollars per month for the extra port"...

we have an entire facility that is completely A+B from the ground up - and the costs are a good 40-50% more over a 500-1000 sq ft cage footprint. not only on a one time setup, but also on a monthly recurring cost...
Posted by erickmiller, 08-10-2011, 10:58 PM

Quote:

Originally Posted by cartika-andrew

the cost for legitimate A+B power is not "a few hundred dollars"... or "10 dollars per month for the extra port"...

we have an entire facility that is completely A+B from the ground up - and the costs are a good 40-50% more over a 500-1000 sq ft cage footprint. not only on a one time setup, but also on a monthly recurring cost...

Absolutely right. Fully 2N (or even 2(N+1)) is very costly to implement properly (2x minus the building lease) and maintain enough extra capacity for a full fail-over while handling spikes.

The mis-information on WHT is baffling. Seems like a lot of people like to hear themselves talk, regardless of the accuracy of their statements. Good politicians?

Eric
Posted by erickmiller, 08-10-2011, 11:05 PM

Quote:

Originally Posted by cartika-andrew

end of the day though, people will complain with any outage and we understand that - but ask these same people to pay double or triple their existing rate so that once in every 8 years they can avoid an outage - and 99% of them would not be willing to pay the rate required.. its an interesting business we have chosen here

Yep, people definitely like complaining. IT is somewhat of a thankless job.

You're right, though, 99% (or more) would not pay for risk-avoidance for something that has a very low probability of occurring. It's human nature. I'm sure insurance companies have a lot of data on this. And that's all marketing people live for... to take advantage of this behavior. Somewhat self-defeating on the part of the customer.

Eric
Posted by RossH, 08-11-2011, 02:36 AM

Quote:

Originally Posted by erickmiller

Absolutely right. Fully 2N (or even 2(N+1)) is very costly to implement properly (2x minus the building lease) and maintain enough extra capacity for a full fail-over while handling spikes.

The mis-information on WHT is baffling. Seems like a lot of people like to hear themselves talk, regardless of the accuracy of their statements. Good politicians?

Eric

Andrew I would agree it is a thousand+ dollars to do A+B power but the question I must ask you is why haven't you provided a name server outside of colo4 when I myself and a few others have brought it to your attention for clients who use your name servers but a third party email provider. This was extremely cost effective and yet you refused to do it. Are you going to do it now to save those clients the frustration in the future?

This was the main reason I didn't chose you for my hosting.
Posted by cartika-andrew, 08-11-2011, 02:41 AM

Quote:

Originally Posted by RossH

Andrew I would agree it is a thousand+ dollars to do A+B power but the question I must ask you is why haven't you don't a name server outside of colo4 when I myself and a few others have brought it to your attention for clients who use your name servers but a third party email provider. This was extremely cost effective and yet you refused to do it. Are you going to do it now to save those clients the frustration in the future?

Ross - may I ask why you would mention this? throughout this process, our primary DNS has not been down. So, although you feel we could handle DNS better, we have had 100% DNS uptime for over 8 years. What exactly is your point here? which frustration would this have saved exactly Ross? Our DNS is on different power feeds (explicitly different power plants and are fed off of different networks) - our DNS has not gone down throughout this process - as I told you , the way we do DNS is actually just as reliable as spreading it out across different data centers... and likely more so considering the $100's of $1000's of dollars we have spent on securing our DNS servers via various hardware firewalls, DDOS appliances, etc... which are typically not available at your average DNS pop...

having said this - we are moving 50% of our DNS to our newly launched Toronto facility in the very near future - we are doing this to better service our newly launched facility - so, even though this would have made ZERO difference here - I think this will meet your expectations correct?
Posted by cartika-andrew, 08-11-2011, 02:43 AM

Quote:

This was the main reason I didn't chose you for my hosting.

thats too bad - I am not sure how many providers have had a 100% DNS uptime since that point... but, we are one of them..
Posted by RossH, 08-11-2011, 02:56 AM

Quote:

Originally Posted by cartika-andrew

Ross - may I ask why you would mention this? throughout this process, our primary DNS has not been down. So, although you feel we could handle DNS better, we have had 100% DNS uptime for over 8 years. What exactly is your point here? which frustration would this have saved exactly Ross? Our DNS is on different power feeds (explicitly different power plants and are fed off of different networks) - our DNS has not gone down throughout this process - as I told you , the way we do DNS is actually just as reliable as spreading it out across different data centers...

having said this - we are moving 50% of our DNS to our newly launched Toronto facility in the very near future - we are doing this to better service our newly launched facility - so, even though this would have made ZERO difference here - I think this will meet your expectations correct?

Andrew,

Let me first and foremost say I really respect you, you are truly client oriented and have a great grasp of the hosting industry. Every time I have talked to you I feel great knowing that you are there for your customers.

I am truly grateful for your service to clients as it is a shining example of how web hosts should be. With that being said I know that myself and other people have brought up personally to you that using a single data center for both of your name servers was a poor idea. You were warned about this specifically and I know the topic of using a third party email service was brought up. So YES it would have made a difference if you had name servers in multiple locations for your clients who were using third party email service or other services in this situation. If you are saying these exchanges didn't happen or if my assessment isn't correct and you wish to state so then so be it.

I am glad you are moving to correct the problem, this shows your resolve to clients.
Posted by cartika-andrew, 08-11-2011, 03:00 AM

Quote:

Originally Posted by RossH

Andrew,

Let me first and foremost say I really respect you, you are truly client oriented and have a great grasp of the hosting industry. Every time I have talked to you I feel great knowing that you are there for your customers.

I am truly grateful for your service to clients as it is a shining example of how web hosts should be. With that being said I know that myself and other people have brought up personally to you that using a single data center for both of your name servers was a poor idea which you seemed to shrug off. You were warned about this specifically and I know the topic of using a third party email service was brought up even if you were down situation was brought up. So YES it would have made a difference if you had name servers in multiple locations for your clients who were using third party email service or other services in this situation. If you are saying these exchanges didn't happen or if my assessment isn't correct and you wish to state so then so be it.

I am glad you are moving to correct the problem, this shows your resolved to clients.

Ross,

seriously man??? I am expecting a long and sincere apology here as you are creating issues for me on a day where I have enough issues to deal with.

Let me be clear - there has been ZERO DNS outage for our primary DNS services over this incident - so, your entire point is moot.

Why on earth would you bring up our DNS servers when, regardless of this outage at colo4 - our DNS has not gone down - what exactly is your point? all you are saying is that you think our DNS could be handled better and be more available? meanwhile, the datacenter has had a massive outage today and our DNS has NOT gone down.. so again, what is your point here?
Posted by RossH, 08-11-2011, 03:13 AM

Quote:

Originally Posted by cartika-andrew

Ross,

seriously man??? I am expecting a long and sincere apology here as you are creating issues for me on a day where I have enough issues to deal with.

Let me be clear - there has been ZERO DNS outage for our primary DNS services over this incident - so, your entire point is moot.

Why on earth would you bring up our DNS servers when, regardless of this outage at colo4 - our DNS has not gone down - what exactly is your point? all you are saying is that you think our DNS could be handled better and be more available? meanwhile, the datacenter has had a massive outage today and our DNS has NOT gone down.. so again, what is your point here?

Andrew,

If your dns has not gone down I sincerely apologize and retract all of my statements and feel truly horrible. Again I do not wish to hurt your reputation as a great hosting provider because you truly are, especially one I respect.

My point is you were using name servers in the same data center so clients using your hosting who may have been using outside resources (gmail) may have been completely down when they could have been somewhat functional had you diversified your dns infrastructure as people had suggested.

I don't know how you are saying zero dns downtime unless you added a third name server when both of your name servers were at colo4.

--removed--

At any event if you are saying this did not cause an outage for clients you are very lucky and I'm glad things worked out.
Posted by cartika-andrew, 08-11-2011, 03:19 AM
Holy crap Ross

can you PLEASE remove our PRIVATE LABEL DNS from a public forum.. what the heck man???

and yes - start profusely apologizing - neither of our DNS servers have been down today - not even for a single second, regardless of the outage at colo4..

please remove those DNS servers from this public forum ASAP - and yes, please apologize profusely - as our DNS servers have not been down today...

seriously Ross - its been a bad day - I seriously did not need this unnecessary stress from you today - especially when your point is completely moot - AGAIN - OUR DNS HAS NOT GONE DOWN FOR A SINGLE SECOND TODAY
Posted by cartika-andrew, 08-11-2011, 03:26 AM

Quote:

Originally Posted by RossH

I don't know how you are saying zero dns downtime unless you added a third name server when both of your name servers were at colo4.

As I previously told you Ross - our DNS setup is second to none - being in the same facility with our setup is more reliable then most that have their DNS in separate NOCs.. I do not want to debate this point with you right now as I have had a seriously bad day - but, I do expect an apology from you - as again - our DNS has not gone down today - not for a single second.. just because you do not understand how to deliver highly available DNS the way we deliver it, does not mean it is not possible - and it certainly doesnt mean that I need to deal with this today because you have a bone to pick and chose a very bad day to do so..

Ross - I hold no ill will - but, I am seriously not impressed with what you tried to do here and why... its just a bad day, I will be over this tomorrow - so, a simple apology will do for now and we can chat and resolve this at a later date...

thanks in advance..
Posted by RossH, 08-11-2011, 03:30 AM

Quote:

Originally Posted by cartika-andrew

Holy crap Ross

can you PLEASE remove our PRIVATE LABEL DNS from a public forum.. what the heck man???

and yes - start profusely apologizing - neither of our DNS servers have been down today - not even for a single second, regardless of the outage at colo4..

please remove those DNS servers from this public forum ASAP - and yes, please apologize profusely - as our DNS servers have not been down today...

seriously Ross - its been a bad day - I seriously did not need this unnecessary stress from you today - especially when your point is completely moot - AGAIN - OUR DNS HAS NOT GONE DOWN FOR A SINGLE SECOND TODAY

Andrew,

I apologize profusely then and have removed those from my post as per your request even though both name servers reside at colo4 which is what we are talking about here (geographic name server redundancy).

You are truly a great hosting company and have done a great job in supporting your clients.
Posted by cartika-andrew, 08-11-2011, 03:40 AM

Quote:

Originally Posted by RossH

Andrew,

I apologize profusely then and have removed those from my post as per your request even though both name servers reside at colo4 which is what we are talking about here (geographic name server redundancy).

You are truly a great hosting company and have done a great job in supporting your clients.

thanks Ross - and sorry - it has been - as I am sure you can imagine - a VERY bad day.

there is a reason our DNS did not go down - and there is a reason we chose to have our DNS in a single facility behind $500k of network and security hardware.

The risk on our part is that the entire DC disappears - this is a legitimate risk - and you are right about this. But, aside from the entire DC disappearing, we are more redundant and better protected then any geographically dispersed DNS.

The reason we did this is because we knew we were going to launch a 2nd facility in a different country (which we just launched). In the interim (over the last few years) - our setup at colo4 was more secure and more redundant then could have been accomplished at 2 different facilities. Frankly, we proved that here today. Our exposure was that colo4 disappeared - and we determined that was a lower risk then simple DNS attacks, or server failures or network failures (which we are better protected against at colo4 then we could have been at multiple facilities)

I hope this makes sense to you

anyway, I apologize for snapping at you and I very much appreciate your apology.. please understand - DNS aside - which has been rock solid today - we have obviously had a very bad day with our data center power issues.. so, I do apologize for reacting as I did - and I hope you understand and forgive me..

cheers
Andrew
Posted by RossH, 08-11-2011, 03:46 AM

Quote:

Originally Posted by cartika-andrew

As I previously told you Ross - our DNS setup is second to none - being in the same facility with our setup is more reliable then most that have their DNS in separate NOCs.. I do not want to debate this point with you right now as I have had a seriously bad day - but, I do expect an apology from you - as again - our DNS has not gone down today - not for a single second.. just because you do not understand how to deliver highly available DNS the way we deliver it, does not mean it is not possible - and it certainly doesnt mean that I need to deal with this today because you have a bone to pick and chose a very bad day to do so..

Ross - I hold no ill will - but, I am seriously not impressed with what you tried to do here and why... its just a bad day, I will be over this tomorrow - so, a simple apology will do for now and we can chat and resolve this at a later date...

thanks in advance..

Andrew,

I am more than willing to admit a mistake on my part as people can see in previous posts and am more than willing to give you credit for being a great hosting company. However you have seem to gone a bit far with my statement of not knowing how to deliver highly redundant dns considering I'm using anycast dual stack with dnssec on my domains and have sold other people on using anycast dns from service such as dyn.com.

After this accusation I would ask you which part of dns do you think I not understand while being a client of multiple anycast providers. This wasn't meant to be my pile on but my hopeful push for you to consider a name server in a geographically separate area. Again if no dns outages happened I'm sorry if that was the perception I gave however I hope this gives you the opportunity to diversify your name server infrastructure to your clients.

Andrew,

I apologize if this added to your load but I wished to add my comment here so you could better diversify your dns infrastructure for your clients benefit, if that has offended you then I'll say I'm sorry. I honestly wish you would just say "you know what that was a good move and we will add a third name server outside of colo4 for dns requests."

However I will apologize for bringing this all up, I am sorry Andrew personally and Cartika. I think you all run a great shop, I just want to see you be better.
Posted by cartika-andrew, 08-11-2011, 03:50 AM
Ross - please see my previous reply.. I obviously over-reacted here - but... to be fair, we performed where you thought we would fail..

its all good man.. You know I hold you in very high regard.. its just a bad day

thanks for your comments and please dont hold anything against me today
Posted by RossH, 08-11-2011, 03:54 AM

Quote:

Originally Posted by cartika-andrew

Ross - please see my previous reply.. I obviously over-reacted here - but... to be fair, we performed where you thought we would fail..

its all good man.. You know I hold you in very high regard.. its just a bad day

thanks for your comments and please dont hold anything against me today

Andrew,

I hold nothing against you since I hold you in such high regard, you are truly a shining light in this dim industry. Today has been a stressful day and I'm glad you performed to the expectations that were set, I would expect nothing less of you.

Thank you for all the hard work you guys do, all of us are truly glad you set the standard for professional hosting.
Posted by cartika-andrew, 08-11-2011, 04:02 AM
Thank you Ross - as per usual, you have demonstrated a level of class that is second to none. Again, I apologize for snapping at you..
Posted by Krishopper, 08-11-2011, 02:07 PM

Quote:

Originally Posted by erickmiller

Me too. Redundancy costs money (power, network connectivity, DNS fail-over and/or BGP redirection, etc.), and most aren't willing to pay for, and thus support, the concepts around redundant systems. Many people (not all!) are generally willing to pay for a plan that is cheaper, simply because of the cost, not considering the risk they are taking, so it's not a great support system for redundancy.

Having options for redundancy at the infrastructure layer is a great thing, though. If anything, it shows what it actually costs to provide this type of service. Customers will at least be able to choose how much risk avoidance they are willing to pay for.

Offering such service for shared hosting is a fine balance that has to be played correctly, both technically and financially. Obviously it isn't feasible to offer active/active failover between DC's for shared hosting accounts. However, it is in our business plan to ensure that there are disaster recovery plans for such an incident.

We have servers in multiple DC's across the US. It was easier for us to automate this with Hsphere than it is with cPanel, due to the way Hsphere was designed (single control panel server, and then clusters of servers configured within). But you need better than once-a-day backups. Asynchronous replication of customer data is most important, so we have the data ready to go at another DC should something happen. If there is an outage, and we do not have a clear understanding of when the outage will be resolved, or if it will not be resolved in a reasonable amount of time (I don't like going longer than 15 minutes, and even that is pushing it), accounts will be automatically created and restored on servers in another DC, and DNS will be automatically changed if we cannot quickly port the IP address(es) over.

There are a lot of other considerations (Customers hosting DNS somewhere else, TTL's with each domain's DNS, etc.) that need to be considered to actually make this work, but it's not a rocket science.

It's simply in the best interest of the customer, and your business, to make sure you don't depend on a single data-center for all of your infrastructure. Amazon's EC2 outage from several months ago is a prime example of this.
Posted by cartika-andrew, 08-11-2011, 03:11 PM
Hello Krishopper - very nice post and some really interesting ideas. Doing this for shared hosting accounts at a massive level is not really a feasible solution as far as we are concerned - but, I can see it working on smaller scales. I guess it just comes down to what people are willing to pay for. Offering a replicated solution like this would mean a greater then 100% price increase - and most of our customers would not pay for that, nor would they have signed up for our service to begin with at those price tags.

We take a different approach where users can spread accounts across fleets of servers and across different data centers (using hsphere as you mentioned). This allows risk to be spread. For example, with yesterdays power outage at our colo4 facility, it would be very unlikely that a reseller had all accounts down unless they were a smaller resellers with all accounts on a few servers and those servers happened to be the ones that were down. Most resellers, on average, would have had 25% of their accounts impacted (as that was the portion of our infrastructure that was down as a result of the power failure). moving forward with our Toronto facility, then only accounts at the impacted facility would be impacted. This does not make every account highly available at the data center outage level, but, it does spread and mitigate risk.

Quote:

Originally Posted by Krishopper

It's simply in the best interest of the customer, and your business, to make sure you don't depend on a single data-center for all of your infrastructure. Amazon's EC2 outage from several months ago is a prime example of this.

The Amazon example is an excellent one. Having said this, offering redundancy at the datacenter level is a different business. It is not one, as of today, that we or most providers offer. We do intend on launching such an offering for cloud customers between our 2 facilities, but, I assure you that 99% of customers will not purchase that option - it is just too expensive to justify something that happens once a decade or less (ie a data center level outage)

anyway, thanks for your post and your comments - very well stated and very accurate... obviously if you offer this specific service, its a nice, niche offering in shared hosting