34 Peachtree Power Outage

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > 34 Peachtree Power Outage

Posted by UGVPS, 06-29-2013, 08:31 PM
My servers in 34 Peachtree are down, blahh. If anyone else in this facility is wondering why your atl servers are down, well this is why lol.

"There has been a power event at the facility resulting in the loss of all or part of the power to the DC floor."

More info on LET. Its possible 55 Marietta is also down. http://www.lowendtalk.com/discussion...o-atlanta-down
Posted by VortexServers, 06-29-2013, 08:40 PM
Yep down here as well.
Posted by ramnet, 06-29-2013, 08:43 PM
Same here.

1 hour and counting now.
Posted by RyanD, 06-29-2013, 08:55 PM
All,

That is correct, there is currently a power outage at 34 Peachtree impacting our operations at that site.

There were multiple utility failures and we had transitioned to generator after the first, it continued to bounce up and down after that initial fault.

While running on generator the generator experienced a fault and failed.

UPS Battery supply of 15 Minutes was subsequently exhausted after that point.

We are still currently without utility power and are working in parallel on the restoration of both Utility and Generator power.

Our helpdesk is online and available and will be posting further updates directly with clients in our helpdesk.

We have all hands on deck standing by to ensure a speedy recovery as soon as power is restored to the datacenter floor.

Thank you!
Posted by qps, 06-29-2013, 08:56 PM

Quote:

Originally Posted by UGVPS

Its possible 55 Marietta is also down.

55 Marietta (the building) is not impacted. It is possible that your specific provider within 55 Marietta may be having issues.
Posted by Gibby13, 06-29-2013, 09:24 PM
Down here too, is it whole building at 34 Peachtree?
Posted by techjr, 06-29-2013, 09:41 PM
I have services there but indirectly. Is this the first outage in a few years?
Posted by VortexServers, 06-29-2013, 09:49 PM
Just went back online over here.
Posted by ramnet, 06-29-2013, 10:13 PM
Down again
Posted by UGVPS, 06-29-2013, 10:18 PM

Quote:

Originally Posted by ramnet

Down again

Down again here aswell
Posted by VortexServers, 06-29-2013, 10:23 PM

Quote:

Originally Posted by ramnet

Down again

Ditto.....
Posted by mctDarren, 06-29-2013, 10:31 PM
Never even came back up.. sniff...
Posted by Gibby13, 06-29-2013, 10:43 PM
We were up temporarily as well, had an applogic grid controller in the middle of a recovery... Hope it survived.
Posted by Gibby13, 06-29-2013, 10:51 PM
We are back up again...
Posted by VortexServers, 06-29-2013, 10:56 PM

Quote:

Originally Posted by Gibby13

We are back up again...

We are back online again now as well. How did everything go with that recovery?
Posted by Gibby13, 06-29-2013, 11:35 PM
Everything recovered correctly and automatically and all customers are back online.

I love Applogic...
Posted by UGVPS, 06-30-2013, 06:17 AM
I thought the power outage was resolved a while ago =/ Ryan, can you comment? Has the power issue in 34 Peachtree been fully resolved at this time?
Posted by MattS, 06-30-2013, 06:41 AM
Should be resolved, my servers in that facility have been up since about 12 AM.
Posted by JonFatino, 06-30-2013, 08:48 AM
Everything has been up and online for about 9 hours now.
Posted by Vivid, 06-30-2013, 09:41 AM
Also have a few servers in Peacetree, been up since 11-12 EDT last night
Posted by Vivid, 06-30-2013, 09:57 AM
Looks like servers are down again, checked few other providers their and they don't seem to respond either
Posted by MattS, 06-30-2013, 09:59 AM
Yep, down again here too.
Posted by v3mediaworks, 06-30-2013, 10:07 AM
Here we go again!
Posted by mctDarren, 06-30-2013, 10:31 AM
vps.net, who had some clients down all night due to some kind of hardware issue, was just starting to bring up those cloud servers. Blargh.
Posted by JFOC, 06-30-2013, 10:37 AM

Quote:

Originally Posted by qps

55 Marietta (the building) is not impacted. It is possible that your specific provider within 55 Marietta may be having issues.

down here 55 Marietta since 38 mins ago
Posted by MattS, 06-30-2013, 10:55 AM

Quote:

Originally Posted by JFOC

down here 55 Marietta since 38 mins ago

Not having any issues myself in 55 Marietta.
Posted by sprintserve, 06-30-2013, 10:58 AM
It appears down again indeed.
Posted by JFOC, 06-30-2013, 11:04 AM

Quote:

Originally Posted by MattS

Not having any issues myself in 55 Marietta.

Then you're lucky
Posted by rob215x, 06-30-2013, 11:06 AM
Our dedicated server was down from 06/29/2013 7:43PM EST, then it was back up around midnight. This morning, 06/30/2013 9:53PM EST, it is down again and is still down.
Posted by Gogax | Simon, 06-30-2013, 11:10 AM
Another Power Failure... Another Hard Day
Posted by JFOC, 06-30-2013, 11:13 AM
There is currently issue with the power at the datacenter.

Electrician are on site and working on the utility power restoration. Primary
Generator have failed and a rollin generator ( ETA is 1 hour) is on its way to
the datacenter right now.

There is no currently a ETA on power restoration but we are working hard to
bring everyone back online as soon as possible.
--
Thank You,

Carlos Carluccio

Total Server Solutions
Posted by TheUruguayan, 06-30-2013, 11:31 AM
Just a update . Main power is back online. Network routing is online. Everything should be coming online shortly.
Posted by JFOC, 06-30-2013, 11:33 AM
Yes Its back online
Posted by sprintserve, 06-30-2013, 11:51 AM
Still not getting anything so far.
Posted by garysimat, 06-30-2013, 11:55 AM

Quote:

Originally Posted by sprintserve

Still not getting anything so far.

Each segment of the electrical infrastructure is being energized individually to help diagnose any issues. Several rows are online at this time, you should see additional rows come online as each section is inspected by our electrical engineers.
Posted by garysimat, 06-30-2013, 12:06 PM
The datacenter floor is presently 60% energized and we are pushing forward with energizing additional rows. We have electrical engineers and mechanics onsite diagnosing the generator and electrical system and an additional generator en route to the facility.
Posted by JonBiloh, 06-30-2013, 12:21 PM

Quote:

Originally Posted by garysimat

The datacenter floor is presently 60% energized and we are pushing forward with energizing additional rows. We have electrical engineers and mechanics onsite diagnosing the generator and electrical system and an additional generator en route to the facility.

What caused the outage this morning specifically?
Posted by garysimat, 06-30-2013, 12:31 PM

Quote:

Originally Posted by JonBiloh

What caused the outage this morning specifically?

The only information available right now is we lost city power for an extended period of time causing us to flip to generator power and the generator failed. We do load testing on the generator once a week so I am uncertain as to why this failed however we have mechanics onsite presently diagnosing the generator and we have an additional roll up generator that was just delivered which is being added to the electrical infrastructure immediately.
Posted by imtiax, 06-30-2013, 12:45 PM
UGVPS is still offline right now. Can't wait for it to be back.
Posted by garysimat, 06-30-2013, 12:49 PM

Quote:

Originally Posted by garysimat

The only information available right now is we lost city power for an extended period of time causing us to flip to generator power and the generator failed. We do load testing on the generator once a week so I am uncertain as to why this failed however we have mechanics onsite presently diagnosing the generator and we have an additional roll up generator that was just delivered which is being added to the electrical infrastructure immediately.

Additional information on the loss of city power - this was caused by a failing breaker on the utility feed.
Posted by mctDarren, 06-30-2013, 01:11 PM
vps.net back up on new generator power I assume
Posted by garysimat, 06-30-2013, 01:14 PM

Quote:

Originally Posted by serversphere

vps.net back up on new generator power I assume

Presently we are on utility power, the additional generator is being added right now and we will transfer to this just as soon as it is complete.
Posted by Gibby13, 06-30-2013, 01:24 PM
We are back up as of about 30 minutes ago.
Posted by UNIXy, 06-30-2013, 01:38 PM
What a mess this is Gary! Most facility is on utility power, however, a few remaining cabinets are "waiting" for the new diesel generator to be put "online"? Why are you oversubscribing power?

That's 5 2-hour outages in the last three months alone. I must call you out on this. What a mess! I'm a very patient man but this is unbelievable incompetence and greed. I'm very upset.
Posted by garysimat, 06-30-2013, 01:44 PM

Quote:

Originally Posted by UNIXy

What a mess this is Gary! Most facility is on utility power, however, a few remaining cabinets are "waiting" for the new diesel generator to be put "online"? Why are you oversubscribing power?

That's 5 2-hour outages in the last three months alone. I must call you out on this. What a mess! I'm a very patient man but this is unbelievable incompetence.

We are not oversubscribing power. As we have had faults in the power system the engineers do not want to energize that last portion until the generation is in place as to avoid any possible issues when getting closer to the 80% mark on that segment. We strictly abide by the 80% rule on all of our power systems the same way we enforce this to our clients.
Posted by UNIXy, 06-30-2013, 01:52 PM

Quote:

Originally Posted by garysimat

We are not oversubscribing power. As we have had faults in the power system the engineers do not want to energize that last portion until the generation is in place as to avoid any possible issues when getting closer to the 80% mark on that segment. We strictly abide by the 80% rule on all of our power systems the same way we enforce this to our clients.

NOT ACCEPTABLE. You've failed big time Gary. If you had enough power all systems would be up. You're still down. It's your job to worry about segment density.

I don't care if you're keeping it below 80, you're still DOWN.
Posted by garysimat, 06-30-2013, 02:00 PM

Quote:

Originally Posted by UNIXy

NOT ACCEPTABLE. You've failed big time Gary. If you had enough power all systems would be up. You're still down. It's your job to worry about segment density.

I don't care if you're keeping it below 80, you're still DOWN.

Joe - thank you for your feedback. Our segment density is just fine. Because of the faults happening the engineers - who approved the load - do not want to put more load on it until they can be 100% certain that what they believe to be the failing breaker is absolutely the issue. The breaker cannot be replaced without the generator in place. I understand your frustration but we do heavy planning with all of our engineers when designing our deployments. Issues do happen with any provider, as such today is just our day to be hit with it and we are working with each of our clients as quickly as possible to restore service. We appreciate your patience.
Posted by alpha, 06-30-2013, 02:09 PM
Hey guys,

This is what I recommended to Carlos last night in the live chat.

I think it would be a great idea if you guys could setup a system where customers are notified of outages actively. It seems like you guys were just passively communicating with customers that are opening tickets or reaching out via live chat.

As far as I know, there is no "status" page that we can check.

Would probably save you guys from quite a few "Why are my servers down?" tickets.
Posted by garysimat, 06-30-2013, 02:12 PM

Quote:

Originally Posted by alpha

Hey guys,

This is what I recommended to Carlos last night in the live chat.

I think it would be a great idea if you guys could setup a system where customers are notified of outages actively. It seems like you guys were just passively communicating with customers that are opening tickets or reaching out via live chat.

As far as I know, there is no "status" page that we can check.

Would probably save you guys from quite a few "Why are my servers down?" tickets.

Thank you for your feedback. We will have some internal discussion on this and see what we can accomodate.
Posted by ericgregory, 06-30-2013, 02:13 PM
Alpha,

Thanks for the suggestion. It's something that I will look into getting added for our customers across our sites. Something like Apple's system status on their site seems like a good format to show full system status across our locations.
Posted by Gibby13, 06-30-2013, 02:15 PM
We have been with Colo@ since February 2012, they are an awesome team to work with and we have never had any issues except this power issue the past 2 days. Their hands on support is second to none and very fast to respond, even on weekends to non-emergency requests. Gary and Ryan even setup a tour for us to be on a weekend before we signed up with them, then they also assisted with our move in during a weekend.

Thanks for keeping us updated Gary!!!
Posted by UNIXy, 06-30-2013, 02:19 PM

Quote:

Originally Posted by garysimat

Joe - thank you for your feedback. Our segment density is just fine. Because of the faults happening the engineers - who approved the load - do not want to put more load on it until they can be 100% certain that what they believe to be the failing breaker is absolutely the issue. The breaker cannot be replaced without the generator in place. I understand your frustration but we do heavy planning with all of our engineers when designing our deployments. Issues do happen with any provider, as such today is just our day to be hit with it and we are working with each of our clients as quickly as possible to restore service. We appreciate your patience.

It isn't feedback. It's fact. It's still DOWN.

Gary if the main breaker is going off at the utility's end it's because you've tripped it. It's an overload. Over capacity. There's no other way to paint it.

It's isn't just today. It's yesterday too. Not to mention issues I've experienced in the last couple of months. It's one too many issues.

It's still DOWN.
Posted by garysimat, 06-30-2013, 02:19 PM
Additional generator is done being cabled, engineers are starting it up to warm up and will transfer load to this shortly. Once the load is transferred they will begin investigating further.
Posted by sprintserve, 06-30-2013, 02:24 PM
Will we expect any downtime during this transfer?
Posted by garysimat, 06-30-2013, 02:25 PM

Quote:

Originally Posted by UNIXy

It isn't feedback. It's fact. It's still DOWN.

Gary if the main breaker is going off at the utility's end it's because you've tripped it. It's an overload. Over capacity. There's no other way to paint it.

It's isn't just today. It's yesterday too. Not to mention issues I've experienced in the last couple of months. It's one too many issues.

It's still DOWN.

Joe,

A breaker can be fault and not carry its rated load. As I have already mentioned, the engineers do not want to push the breaker anywhere past 70% at this time until they have the generator in place, which was just cabled.

The prior issue was a fiber cut on Zayo's network within 56 Marietta which affected over 1200 strands of fiber.
Posted by garysimat, 06-30-2013, 02:28 PM

Quote:

Originally Posted by sprintserve

Will we expect any downtime during this transfer?

There should be no downtime anticipated, we should be transferring in about 45 minutes.
Posted by UNIXy, 06-30-2013, 02:31 PM

Quote:

Originally Posted by garysimat

Joe,

A breaker can be fault and not carry its rated load. As I have already mentioned, the engineers do not want to push the breaker anywhere past 70% at this time until they have the generator in place, which was just cabled.
The prior issue was a fiber cut on Zayo's network within 56 Marietta which affected over 1200 strands of fiber.

In this case you don't go buying a portable GENERATOR to compensate for a defective breaker. Logic is the breaker needs to be replaced. This doesn't make sense...
Posted by sprintserve, 06-30-2013, 02:33 PM

Quote:

Originally Posted by UNIXy

In this case you don't go buying a portable GENERATOR to compensate for a defective breaker. Logic is the breaker needs to be replaced. This doesn't make sense...

I understand you are upset. We are too. Why don't you take this offline with them, ideally when the issues are fully resolved?
Posted by garysimat, 06-30-2013, 02:35 PM

Quote:

Originally Posted by UNIXy

In this case you don't go buying a portable GENERATOR to compensate for a defective breaker. Logic is the breaker needs to be replaced. This doesn't make sense...

Joe,

The generator is to cover the load while we replace the faulty breaker on the utility main. We are not going to take down all of the clients to replace this. As mentioned our primary generator had failed and we will not transfer load to this until it can be broken down and diagnosed, this is why we have an additional generator added and will be transferring load to it.
Posted by UNIXy, 06-30-2013, 02:38 PM

Quote:

Originally Posted by sprintserve

I understand you are upset. We are too. Why don't you take this offline with them, ideally when the issues are fully resolved?

I will keep reporting the issue here until my services are up. It's on topic. I'm upset because services have been partially powered up selectively. And because this is the drop that filled the glass. You might be up but not everyone is. Not to mention I owe more than this to my clients.
Posted by UNIXy, 06-30-2013, 03:09 PM

Quote:

Originally Posted by garysimat

The prior issue was a fiber cut on Zayo's network within 56 Marietta which affected over 1200 strands of fiber.

How does a fiber cut cause a complete BGP blackout? I know we've already gone through this episode.

I have never complained about a vendor in public until today. It takes a *lot* to get me upset. I didn't complain about the other 3 2-hour outages from a few months either.

What if this new generator fails... Based on what I've seen everything is possible at this point. Who are you going to "power up"?

And it's still DOWN.
Posted by UGVPS, 06-30-2013, 03:11 PM
Is 34 Peachtree fully resolved yet?
Posted by garysimat, 06-30-2013, 03:12 PM

Quote:

Originally Posted by UNIXy

How does a fiber cut cause a complete BGP blackout? I know we've already gone through this episode.

I have never complained about a vendor in public until today. It takes a *lot* to get me upset. I didn't complain about the other 3 2-hour outages from a few months either.

What if this new generator fails... Based on what I've seen everything is possible at this point. Who are you going to "power up"?

And it's still DOWN.

We will not be replying to you further within WHT as your clearly intent on arguing regardless of what is said. Please place your ticket through our official support options. Thank you.
Posted by UGVPS, 06-30-2013, 03:19 PM
Gary,

While unixy is upset hes upset with proper reason. Who wouldn't be? The outages have been very sporadic throughout the past 24 hours, and this can't be healthy for any server.
Posted by garysimat, 06-30-2013, 03:23 PM
All power has been fully restored to all rows at this time. The new generation is in place and on standby presently, we will be transferring to generator power while our engineers are investigating further.
Posted by UNIXy, 06-30-2013, 03:37 PM

Quote:

Originally Posted by garysimat

We will not be replying to you further within WHT as your clearly intent on arguing regardless of what is said. Please place your ticket through our official support options. Thank you.

What good is another ticket? Remember the last time I begged you to answer a ticket/question I had about one of the numerous outages I experienced? It took you 10 days to reply to the ticket only to to tell me to "place each question in individual tickets." Where would we be if it weren't for Ryan helping out... Ryan and Dmitry have been the only ones restoring my faith in your company. I'm not sure how long this can go.

Again, I'm upset because you have been neglecting me, your client. I hate this state of negativity and dread the day I have to go complain about a vendor in public. I'm human and only have so much patience.

It's up right now.

Thanks
Posted by RyanD, 06-30-2013, 04:11 PM

Quote:

Originally Posted by UNIXy

What good is another ticket? Remember the last time I begged you to answer a ticket/question I had about one of the numerous outages I experienced? It took you 10 days to reply to the ticket only to to tell me to "place each question in individual tickets." Where would we be if it weren't for Ryan helping out... Ryan and Dmitry have been the only ones restoring my faith in your company. I'm not sure how long this can go.

Again, I'm upset because you have been neglecting me, your client. I hate this state of negativity and dread the day I have to go complain about a vendor in public. I'm human and only have so much patience.

It's up right now.

Thanks

Joe,

Our full team is on-site and working to resolve these issues. To streamline the process people are focuses on specific recovery tasks. Myself, Dmitry, and the rest of the team are doing our best to ensure that the recovery is as smooth as possible, given the issues at hand.

Thank you,
Posted by UNIXy, 06-30-2013, 04:22 PM

Quote:

Originally Posted by RyanD

Joe,

Our full team is on-site and working to resolve these issues. To streamline the process people are focuses on specific recovery tasks. Myself, Dmitry, and the rest of the team are doing our best to ensure that the recovery is as smooth as possible, given the issues at hand.

Thank you,

Very grateful Ryan.

Thanks
Posted by sprintserve, 07-02-2013, 04:13 AM
We need a detailed report, including a timeline, of what went wrong at the data center on Saturday night and Sunday morning. Also, what changes are planned if at all, and if yes, what those are.

The sooner we can get this the better, as we do have clients threatening to leave.

We did ask this in a ticket but it got closed and it has been more than 48 hours since the first outage and clients are understandably concerned that the issue is being ducked. We are also having clients threatening to leave if they don't at least get this little bit of accountability. So please help us out a bit here.
Posted by ericgregory, 07-02-2013, 09:43 AM
Our team will be working on a detailed report on this incident and we will have a full RFO for our customers shortly. Our desire is to make certain all repairs are completed and everything is precisely as it should be prior to releasing any reports.

Thanks for being patient. It's greatly appreciated :-)
Posted by Micron21, 07-02-2013, 10:09 AM
As a fellow datacentre operator (in a land far away in Australia) I feel your pain of having a main breaker fail.

That being said Id love to know what brand breaker you are using and how many amps / volts is it pushing!

Within our facility we setup a double ATS configuration active / standby so we can replace a main breaker if it wants to fail .

We use Schneider electric logic 5 breakers

Kindest Regards

James
Posted by scottc, 07-02-2013, 07:29 PM
I am a reseller for a client of 34 Peachtree. Due to the extensive outages over the weekend I have to give my clients an explanation as to what happened (in addition to refunds). Does anyone have suggestions as to how to explain, in terms that bloggers and other small site owners can understand, that a data center which advertises "fully redundant power" had a long power outage? (That claim can be found on their blog at http://blog.coloat.com/?p=105 )

I'm also open to recommendations for other data centers. This is not our first power outage in a data center that claimed to have redundant power. How does one really know whether they have that capability or not?

Thanks for any advice you can give.

Scott
Posted by Rob T, 07-02-2013, 08:10 PM

Quote:

A breaker can be fault and not carry its rated load.

I can absolutely attest to that, as it happened in our facility last week. We experienced a breaker failure on a a piece of gear that was only at 50% load. It's a very difficult situation to diagnose - my sympathies go out to you guys.
Posted by CGotzmann, 07-02-2013, 08:18 PM

Quote:

Originally Posted by scottc

I am a reseller for a client of 34 Peachtree. Due to the extensive outages over the weekend I have to give my clients an explanation as to what happened (in addition to refunds). Does anyone have suggestions as to how to explain, in terms that bloggers and other small site owners can understand, that a data center which advertises "fully redundant power" had a long power outage? (That claim can be found on their blog at http://blog.coloat.com/?p=105 )

I'm also open to recommendations for other data centers. This is not our first power outage in a data center that claimed to have redundant power. How does one really know whether they have that capability or not?

Thanks for any advice you can give.

Scott

Scott,

Every datacenter will have downtime -- no matter what.
The only way to avoid this is to have production content spread between geographically diverse datacenters with some method of failover that is agnostic of either of the datacenters.
Your current provider can offer you this as it stands...
Posted by scottc, 07-02-2013, 08:47 PM

Quote:

Originally Posted by CGotzmann

Scott,
Every datacenter will have downtime -- no matter what.

I'm not looking for 100% uptime from a single data center. I am looking for a data center which offers what it advertises. Am I wrong to expect no lengthy power outages when a data center advertises "redundant power"?

I understand that switching between utility power and backup generators can cause short outages, and that we can live with.
Posted by CGotzmann, 07-02-2013, 08:49 PM
Unfortunately, yes. Redundant systems can fail - and you can have extended downtime until at least some form of temporary patch is in place.
At all datacenters, no matter what - there is always that possibility.
Posted by ericgregory, 07-03-2013, 12:19 AM
Details can't yet be provided here, but the problem that our facility experienced was extraordinarily unique. We do indeed have redundant systems, but as CGotzmann stated, even redundant systems can fail. It's certainly rare and provides a huge headache for our customers, their customers, and our staff. As mentioned before though, a detailed report is in the works. Reports take some time to research, write, and disseminate to clients. Rest assured though, we have full redundancy measures back in place and our engineers are working on a RFO regarding this incident.

Once the report is ready, we will send it to all effected customers. Our intention is not to keep anyone in the dark longer than necessary. Our first priority was, obviously, to restore full operation of our facility. The next priority is to build a report that details what went wrong, how it happened, and what steps were taken to correct the problem. Once a report has been assembled and checked for accuracy, we will then make sure our customers get a copy of the report.

As stated earlier, this was a very unique event, and we want to make sure that our customers know that we did everything in our power to restore operation as quickly, safely, and effectively as possible.

In the meantime, if you have any specific questions or need anything, please feel free to open a ticket with sales/support as that is the best way to get in touch with our staff.

Thanks for your patience.
Posted by CM2E, 07-03-2013, 10:46 AM
Thanks Eric,
I understand that power outages happen, redundant systems fail and to expect the unexpected. However, one question I do have is:

As we all know, 34 Peachtree lost power and the backup generator failed on Saturday and on Sunday. On Sunday we were notified that a temporary 'roll-in' generator had been requested. When did the temp generator get ordered, Sunday or Saturday?
Posted by ericgregory, 07-03-2013, 11:01 AM
Hi CM2E,

Everything will be discussed in more detail in the RFO when it's sent to customers. Also, in the meantime, if you have any specific needs, please don't hesitate to contact our staff via official correspondence channels.

Thanks for your patience in this.
Posted by CM2E, 07-03-2013, 11:08 AM
Thanks Eric - I look forward to seeing the RFO.
Posted by alanwoo, 07-03-2013, 11:43 AM

Quote:

Originally Posted by scottc

I'm not looking for 100% uptime from a single data center. I am looking for a data center which offers what it advertises. Am I wrong to expect no lengthy power outages when a data center advertises "redundant power"?

I understand that switching between utility power and backup generators can cause short outages, and that we can live with.

What is the point to have redundant power, but the power room is burned down.

http://www.computerworld.com/s/artic..._users_venting

But if you really want a recommendation, I would recommend "Equinix", I have been hosting with them for more than 8 years and I am impressed that they maintained 100% uptime on power and cooling.
Posted by sprintserve, 07-03-2013, 11:55 AM
There's certainly no lack of high profile incidents:

Amazon: http://aws.amazon.com/message/67457/
Rackspace: http://www.rackspace.com/blog/data-c...as-of-530-cst/
ThePlanet: http://www.datacenterknowledge.com/a...-major-outage/

The odds are on the side of a failure. The reason being that the redundant systems can work as planned a million times, and it just need 1 failure for there to be a downtime.
Posted by garysimat, 07-03-2013, 12:19 PM

Quote:

Originally Posted by alanwoo

What is the point to have redundant power, but the power room is burned down.

http://www.computerworld.com/s/artic..._users_venting

But if you really want a recommendation, I would recommend "Equinix", I have been hosting with them for more than 8 years and I am impressed that they maintained 100% uptime on power and cooling.

You are not exactly describing apples to apples in service nor is it reflected in pricing. I am not sure if you have compared our price points to Equinix or not. They are a great provider however, there are many great providers out there - most of which have had their own share of issues.

Quote:

Originally Posted by sprintserve

There's certainly no lack of high profile incidents:

Amazon: http://aws.amazon.com/message/67457/
Rackspace: http://www.rackspace.com/blog/data-c...as-of-530-cst/
ThePlanet: http://www.datacenterknowledge.com/a...-major-outage/

The odds are on the side of a failure. The reason being that the redundant systems can work as planned a million times, and it just need 1 failure for there to be a downtime.

Thank you for your understanding and for pointing this out. We are certainly working hard to correct anything we can on our side. As this is our first power failure event in 5 years, hopefully these corrections and the upgrade to 2N capability (which is currently in progress) will help us far surpass this 5 year uptime record we have to beat.
Posted by Jag, 07-12-2013, 05:09 PM

Quote:

Originally Posted by UNIXy

It isn't feedback. It's fact. It's still DOWN.

Gary if the main breaker is going off at the utility's end it's because you've tripped it. It's an overload. Over capacity. There's no other way to paint it.

It's isn't just today. It's yesterday too. Not to mention issues I've experienced in the last couple of months. It's one too many issues.

It's still DOWN.

And again, and with talking about not wanting to push past 70%, tells us your already close to, at or over that if you have to even consider that number and overall load. This does sound like an oversubscription of power and would make sense why we were contacted about a circuit, your auditing power.

The fact remains, the dc is being support by one generator, no backup generator, and you might not have the capacity in that one to support whats live, hence another generator failure today.

Even if you restore power, your still on one generator. Is there a failover, one, two, any?

This fault and design flaw should not be covered up or tip toed around, it should be fixed.
Posted by Jag, 07-12-2013, 05:14 PM

Quote:

Originally Posted by sprintserve

There's certainly no lack of high profile incidents:

Amazon: http://aws.amazon.com/message/67457/
Rackspace: http://www.rackspace.com/blog/data-c...as-of-530-cst/
ThePlanet: http://www.datacenterknowledge.com/a...-major-outage/

The odds are on the side of a failure. The reason being that the redundant systems can work as planned a million times, and it just need 1 failure for there to be a downtime.

It's disppointing to say the least, and repeating the failure today makes it worse. And sometimes in this industry, we all know **** happens.

I just can not stress the lack of an official place for all clients of this facility to visit in an instant to talk, know whats going on, and be in communication with the facility, even a place to direct clients to when they are bombarding us. We should not be turning to wht to discuss this but there is no other outlet for the facilities clients. Something you guys should consider to stem of the confusion for us all.
Posted by JonBiloh, 07-12-2013, 05:16 PM

Quote:

Originally Posted by Jag

It's disppointing to say the least, and repeating the failure today makes it worse. And sometimes in this industry, we all know **** happens.

I just can not stress the lack of an official place for all clients of this facility to visit in an instant to talk, know whats going on, and be in communication with the facility, even a place to direct clients to when they are bombarding us. We should not be turning to wht to discuss this but there is no other outlet for the facilities clients. Something you guys should consider to stem of the confusion for us all.

This no longer falls under the "**** happens" blanket. Now we're looking at a trend that destroy confidence. Not to mention the potential ugly truth as to the entire source of these issues.
Posted by sprintserve, 07-12-2013, 05:25 PM
unfortunately for it to happen again, for the 3rd time, doesn't quite fall in those once off anymore.
Posted by WindsOfChange, 07-12-2013, 11:53 PM
It's great to see loyal customers standing with DC. I understand you investment is too large to do anything but plead, hope and request.

But for future hedging our bets is the only solution!
Posted by ramnet, 07-13-2013, 12:22 AM

Quote:

Originally Posted by sprintserve

unfortunately for it to happen again, for the 3rd time, doesn't quite fall in those once off anymore.

Well, this is the same ongoing issue for the last 2 weeks. They have been operating under a temporary hacked together power setup since their primary generator went kaput at the end of June.

Once they get their new generator installed I fully expect the power reliability and stability that we've come to expect from Colo@ to return.
Posted by garysimat, 07-13-2013, 03:54 PM

Quote:

Originally Posted by ramnet

Once they get their new generator installed I fully expect the power reliability and stability that we've come to expect from Colo@ to return.

Thank you for the confidence Robert and I can assure you this will be the case. Additionally we will be capable of delivering 2N power to those customers who require an even higher level of uptime once our expansion is complete that which was part of last months announcement. We strive to provide the best service possible while maintaining a cost effective pricing model for the client.
Posted by Jag, 07-15-2013, 09:46 PM

Quote:

Originally Posted by garysimat

Thank you for the confidence Robert and I can assure you this will be the case. Additionally we will be capable of delivering 2N power to those customers who require an even higher level of uptime once our expansion is complete that which was part of last months announcement. We strive to provide the best service possible while maintaining a cost effective pricing model for the client.

What about all the clients that have been paying for 2N and B feeds that also lost complete power. Did you sell a non-existent service? The one generator for the whole facility is a fault.
Posted by CGotzmann, 07-15-2013, 09:47 PM

Quote:

Originally Posted by Jag

What about all the clients that have been paying for 2N and B feeds that also lost complete power. Did you sell a non-existent service? The one generator for the whole facility is a fault.

Just because you have a B feed circuit, does not mean it is a fully divergent power system. Who said it was? I doubt it says that anywhere in the service these clients may or may not have been sold.
Most of the time it is fed from a separate panel and UPS system, but not by default a different generator and utility power feed (even companies like coresite do not do this by default unless you specifically ask for it, and then it costs a lot more for that privilege).
Posted by Jag, 07-15-2013, 09:48 PM

Quote:

Originally Posted by CGotzmann

Just because you have a B feed circuit, does not mean it is a fully divergent power system. Who said it was? I doubt it says that anywhere in the service these clients may or may not have been sold.

Why sell me a B feed that won't work, we have been paying for them?
Posted by garysimat, 07-15-2013, 09:51 PM

Quote:

Originally Posted by Jag

What about all the clients that have been paying for 2N and B feeds that also lost complete power. Did you sell a non-existent service? The one generator for the whole facility is a fault.

It is not a design fault, that is the service that is sold and anyone with redundant circuits were clearly advised that it was multi UPS on a single generator/utility until the next expansion was completed in which 2N would be available.

Technically, even if you had true 2N power all of your racks would have been offline as you only have 1 psu going to your switching.