Codero Down?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Codero Down?

Posted by tompOfAmerica, 03-15-2010, 10:20 AM
My web retail organization hosts its site on a dedicated server at Codero. We are down this morning and can't get Codero on the line. Anybody out there have any info?
Posted by Spudstr, 03-15-2010, 10:26 AM
We have a customer who also uses them, At first they blamed it on a failed network upgrade. Now they are saying it was a dc power outage.

This should be moved to outages forum.
Posted by jgoewert, 03-15-2010, 10:29 AM
It is off for me as well. Both of my dedicated servers can't be reached by IP address. Codero.com and serverportal.com are offline and no one is answering.

From my setup email:

Sales
Email Sales
Toll Free:
1.877.275.8763
Worldwide: 1.858.410.6929
Live Chat

Support
Contact Support
Toll Free:
1.877.275.8763
Worldwide:
1.858.410.6900
Live Chat

Billing
Contact Billing
Toll Free:
1.877.275.8763
Worldwide: 1.858.410.6900
Live Chat
Posted by Prajyot, 03-15-2010, 10:33 AM
sorry but both sites working fine for me
Posted by tompOfAmerica, 03-15-2010, 10:36 AM

Quote:

Originally Posted by Spudstr

We have a customer who also uses them, At first they blamed it on a failed network upgrade. Now they are saying it was a dc power outage.

This should be moved to outages forum.

Yep, I didn't know there was one. I was just scrambling around looking for a clue. If an admin wants to move this to the outages forum I'll follow it there.

Codero.com has now come up, telling us that it was a power outage as you say. I'm interested in following through with this to see if we can get a consistent story.

We're getting some realtime info at twitter:

twitter.com/coderonoc
Posted by Spudstr, 03-15-2010, 10:42 AM
I'm guessing by the looks of things they don't have a UPS.... which is odd.
Posted by net, 03-15-2010, 10:49 AM
Moved > Providers and Network Outages and Updates.
Posted by tompOfAmerica, 03-15-2010, 11:06 AM

Quote:

Originally Posted by Spudstr

I'm guessing by the looks of things they don't have a UPS.... which is odd.

I certainly hope they have a UPS! Why hire a data center to host your server if they all go down when there's a public power outage?

We're going to be interrogating them about exactly what's going on. They have told us that their power distribution unit is malfunctioning. Presumably they have both battery UPS and generator power. In the past we've had this happen and they said that the mechanism for firing up the generator failed.

We just got an update that said when the outage came on major breakers blew and electricians are fixing it now. So we are down for a bit longer. Oh goody.
Posted by liebwerks, 03-15-2010, 11:06 AM
Still down for me. codero.com and serverportal.com and my own websites. Going on 2 hrs now
Posted by Spudstr, 03-15-2010, 11:17 AM
Well, you can have all the redundancy in the world but if you only have one ATS and it blows when you failover you are SOL. The only way to survive that is to have a B power system with its own ATS on its own power feed and UPS/Genset.

When the ATS's snap down they hit hard I am surprised they don't snap more often then they do.
Posted by SinglehopDS, 03-15-2010, 11:32 AM
Seems like their site is back up. They did have power issues, which is extremely odd since one would expect them to be running power-redundant datacenters. Guess not.

Their front page site mentions the following:

"Our Phoenix Data Center experienced a power outage this morning. Power was restored, however, a PDU was affecting leaving some of the Data Center Still down. Electrical Engineers are working to restore the remaining Data Center.

Please contact us at 858.731.1650 if you have any questions"
Posted by rasputin, 03-15-2010, 11:32 AM
From the web site:

Quote:

Our Phoenix Data Center experienced a power outage this morning. Power was restored, however, a PDU was affecting leaving some of the Data Center Still down. Electrical Engineers are working to restore the remaining Data Center.

Please contact us at 858.731.1650 if you have any questions.

In my experience. $hit breaks. It happens to all of us, no matter how much you plan for it and no matter how much you invest to insure that it does not.
Posted by liebwerks, 03-15-2010, 11:51 AM
Unacceptable. I am still down and its almost 12p. There must be redundant everything at a dedicated hosting facility. $hit breaks and that is why you have backups of that $hit. Big thumbs down here for Codero.
Posted by energizedit, 03-15-2010, 11:52 AM
If you go to their webpage they have a popup that explains what is going on with their data center and a number to call for questions.
Posted by tompOfAmerica, 03-15-2010, 11:57 AM
Back up as of roughly 9:55 MDT.
Posted by rasputin, 03-15-2010, 11:59 AM

Quote:

Originally Posted by liebwerks

Unacceptable. I am still down and its almost 12p. There must be redundant everything at a dedicated hosting facility. $hit breaks and that is why you have backups of that $hit. Big thumbs down here for Codero.

Ironically even this forum was "broken" for a bit. Shortly after I posted I kept getting "database errors".

$hit breaks.

You want real redundancy, then ask yourself why you too are not redundant? Try placing a mirror of your site in another "redundant" facility.

It never ceases to amaze me how many people put all their eggs in one basket, and when that basket breaks, they cry about it.

I'm not trying to take sides here. And I am certainly not trying to demean you. All I am saying is that Murphy's Law applies. My loose interpretation being "$hit breaks."

As I recall, the Titanic was unsinkable right?
Posted by tompOfAmerica, 03-15-2010, 12:37 PM
We're on the trampoline--down, then up again, now down again.

Yeah, $hit happens. But hopefully it flushes in a reasonable time.
Posted by liebwerks, 03-15-2010, 03:55 PM
Its hard to say this but I sorta agree with you Rasputin. But I paid extra for this kick-butt dedicated server with redundant this and that so I didn't have to deal with the complexity of mirrored redundant servers. I thought I was in good hands. I was wrong.

In a perfect world where clients understood the value of spending extra $$$ (even 2x $$$) for a redundant system this would be doable. But mine are definitely not...

who can quantify how much was lost when retrocycle.com was not taking orders for 4 hrs today.

Oh, and I am down again at 3:35p east coast time.
Posted by Codero, 03-15-2010, 05:02 PM
We sincerely apologize for the recent outages and understand that such outages severely impact our customers businesses and are looking into the issues that have caused the outage. We are committed to our customers and will be conducting a full incident report which we will make available on our website once it is complete. In the mean time, our COO has provided preliminary information regarding today's interruption of services on our YouTube Channel CoderoTV. As new information comes in we will continue to provide updates via our Twitter accounts, blog, and YouTube.

Regards,

Shelby Garlock
Codero
Director, Marketing
marketing@codero.com
Posted by liebwerks, 03-15-2010, 05:36 PM
Nice explanation on YouTube.

But this is totally inexcusable. This cost me a lot of money and a lot of angry clients.
Posted by Codero, 03-15-2010, 05:41 PM
@liebwerks we agree this is unacceptable. We will continue to provide updates on our Twitter account as more information becomes available.
Posted by CodyRo, 03-15-2010, 11:20 PM

Quote:

Originally Posted by Codero

We sincerely apologize for the recent outages and understand that such outages severely impact our customers businesses and are looking into the issues that have caused the outage. We are committed to our customers and will be conducting a full incident report which we will make available on our website once it is complete. In the mean time, our COO has provided preliminary information regarding today's interruption of services on our YouTube Channel CoderoTV. As new information comes in we will continue to provide updates via our Twitter accounts, blog, and YouTube.

Regards,

Shelby Garlock
Codero
Director, Marketing
marketing@codero.com

Is it just me or the beginning of the video has a lady talking / saying it was really good / "this is how our customers are feeling"?

Just seemed odd :-\
Posted by rasputin, 03-16-2010, 10:44 AM

Quote:

Originally Posted by CodyRo

Is it just me or the beginning of the video has a lady talking / saying it was really good / "this is how our customers are feeling"?

Just seemed odd :-\

You're right.
Never caught that. Weird.