CoreXchange Connection Datacenter down?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > CoreXchange Connection Datacenter down?

Posted by mkern, 01-16-2013, 06:39 PM
It looks like the entire DC could be down (Harry Hines). I haven't had any luck reaching them. Has anyone heard anything?
Posted by net, 01-16-2013, 06:44 PM
It is ok now.

Seems a power issue since all our servers were rebooted.
Posted by mkern, 01-16-2013, 06:44 PM
It appears to have been power related. I talked to support briefly and then things started to recover. Anyone have root cause?
Posted by kris1351, 01-16-2013, 07:14 PM
It was an A side power outage from what we can tell. All of our dual powered items lost one side for about 20 minutes. If you have dual power supplies and dual bandwidth feeds you did not notice the outage (assuming you aren't overloaded). They said they will release an official RFO on it, but all seems fine now.
Posted by Dathorn-Andrew, 01-16-2013, 07:22 PM

Quote:

Originally Posted by kris1351

It was an A side power outage from what we can tell. All of our dual powered items lost one side for about 20 minutes. If you have dual power supplies and dual bandwidth feeds you did not notice the outage (assuming you aren't overloaded). They said they will release an official RFO on it, but all seems fine now.

We saw the same thing. We have A & B power and did not lose power but looking at our UPS graphs and alerts I can see that one of the feeds did indeed lose power. Sorry to those not so fortunate.
Posted by mkern, 01-16-2013, 07:40 PM

Quote:

Originally Posted by kris1351

It was an A side power outage from what we can tell. All of our dual powered items lost one side for about 20 minutes. If you have dual power supplies and dual bandwidth feeds you did not notice the outage (assuming you aren't overloaded). They said they will release an official RFO on it, but all seems fine now.

Yes the issue lasted 22 minutes and power restored after that.
Posted by gordonrp, 01-16-2013, 09:15 PM
2nd power issue in 12 months (first killed A/C), not cool.
Posted by CN-Jeremy, 01-16-2013, 09:38 PM
Seems like the power issues always happen to the other side of the DC as my Coloed servers have not been affected by either power outage.

I think gordon is adding to many servers at once
Posted by net, 01-16-2013, 09:47 PM

Quote:

Originally Posted by gordonrp

2nd power issue in 12 months (first killed A/C), not cool.

What did you do Gordon? Hehe

I agree, this is the 2nd time. We should bring this up to them!
Posted by RobertClarke, 01-16-2013, 09:52 PM
I would like to point out that I was blocked from livechat after inquiring about this downtime, as I am indeed in the same DC as Incero.
Posted by gordonrp, 01-16-2013, 10:09 PM

Quote:

Originally Posted by BundleByte

I would like to point out that I was blocked from livechat after inquiring about this downtime, as I am indeed in the same DC as Incero.

Yes we blocked you from our live chat because YOU'RE NOT OUR CUSTOMER, you're not on our servers. I honestly don't know why you like to waste our sales rep's time. Grow up man.
Posted by RobertClarke, 01-16-2013, 10:11 PM

Quote:

Originally Posted by gordonrp

Yes we blocked you from our live chat because YOU'RE NOT OUR CUSTOMER, you're not on our servers. I honestly don't know why you like to waste our sales rep's time. Grow up man.

At the time, I wasn't 100% on what the issue in CoreXChange was, thus I was getting the opinion of another host in the DC. Being fairly new to the DC, I also asked if this is a common occurrence. Not sure why that counts as wasting your rep's time, as you only had 2 people talking to him at the same time.
Posted by gordonrp, 01-16-2013, 10:20 PM

Quote:

Originally Posted by BundleByte

At the time, I wasn't 100% on what the issue in CoreXChange was, thus I was getting the opinion of another host in the DC. Being fairly new to the DC, I also asked if this is a common occurrence. Not sure why that counts as wasting your rep's time, as you only had 2 people talking to him at the same time.

We've had to block you on skype, twitter, and facebook in the past. We felt your chat was intended to be rude / waste our time especially you retweeting our outage notification on twitter. If you meant it another way, then fine. However kindly leave us alone, don't contact us further (we've asked you this before).
Posted by RobertClarke, 01-16-2013, 10:38 PM

Quote:

Originally Posted by gordonrp

We've had to block you on skype, twitter, and facebook in the past. We felt your chat was intended to be rude / waste our time especially you retweeting our outage notification on twitter. If you meant it another way, then fine. However kindly leave us alone, don't contact us further (we've asked you this before).

At no point was I rude during the conversation. I retweeted your outage tweet, because at the time, I was tweeting out a ton of "we are down, hold tight" tweets, which you can see looking through my timeline.
Posted by lxer, 01-17-2013, 12:40 PM
We were affected by this outage yesterday. We entered a ticket into their system asking for an explanation and haven't received anything yet. Very disappointed in the outage happening at all, and then the lack of communication is very unsettling. Starting to wonder if leaving colo4 was wise.
Posted by mkern, 01-17-2013, 12:45 PM

Quote:

Originally Posted by lxer

We were affected by this outage yesterday. We entered a ticket into their system asking for an explanation and haven't received anything yet. Very disappointed in the outage happening at all, and then the lack of communication is very unsettling. Starting to wonder if leaving colo4 was wise.

We had to put a call into our sales rep to get answers. He confirmed it was an A side power issue and put in a request for us to get a copy of the RFO. We haven't seen anything yet. While this is the first issue we have experienced in a year, it was handled very poorly.
Posted by gordonrp, 01-17-2013, 02:02 PM

Quote:

Originally Posted by lxer

...Starting to wonder if leaving colo4 was wise.

I wouldn't go that far just yet. How they resolve this and the updates they provide on their resolution (I don't mean "hey power is up now!" I mean a detailed overview of works/checks going to be completed by outside vendors rather than inside staff). Everything will fail at some point, whats important now is how they work to restore confidence.
Posted by mkern, 01-17-2013, 02:07 PM

Quote:

Originally Posted by gordonrp

I wouldn't go that far just yet. How they resolve this and the updates they provide on their resolution (I don't mean "hey power is up now!" I mean a detailed overview of works/checks going to be completed by outside vendors rather than inside staff). Everything will fail at some point, whats important now is how they work to restore confidence.

I agree completely with this. Failures will happen and most of us accept that. However, how things are handled during these issues speaks volumes. The notification process needs to improve.
Posted by kris1351, 01-17-2013, 02:18 PM
We have had cabinets in that facility since there was hardly anything in it over 2 years ago. This is the second issue we have seen in this entire time. The cooling issue was the only real issue I see as it caused our routers to shut down, but our servers stayed running. There was nothing we could do to prevent that issue as we were dual fed and have 2 core 7606s running redundant. They just hit the threshold and safely shut down.

As for the issue yesterday, this is why you invest in A&B power for important things. Something malfunctioned seriously and we are waiting on the RFO. UPS/Generator should have kicked in and I know the facility has run on generator a couple of times in the years we have been there. For facilities it is one of the better experiences we have had honestly.

RFOs with our other facilities usually take 72 hours to get out as they have to check SLA and legal terms before they just blast things out. While they are a very affordable colo they aren't from a budget quick fix type background, they run things very efficiently and professionally.
Posted by gordonrp, 01-17-2013, 03:23 PM

Quote:

Originally Posted by kris1351

While they are a very affordable colo they aren't from a budget quick fix type background, they run things very efficiently and professionally.

Sure, agreed. And if they fix the issue then great. However this is the 2nd power outage (the HVAC you mentioned) in a year. 2 times is forgivable, a third will be unreasonable. Lets hope they have some outside vendors do a real audit of the situation. I believe in the past they've used inside staff for electrical.
Posted by Rob T, 01-17-2013, 03:36 PM
There's a simple solution for anyone complaining about their gear going down - pay for redundant power. When you signed up for primary power only, you did so accepting the risk that comes along with having a non-redundant aspect to your service.
Posted by mkern, 01-17-2013, 03:43 PM

Quote:

Originally Posted by Rob T

There's a simple solution for anyone complaining about their gear going down - pay for redundant power. When you signed up for primary power only, you did so accepting the risk that comes along with having a non-redundant aspect to your service.

I don't disagree with what you are saying, however the issue to me here isn't the fact the gear went down its how they handled reporting they had an issue at all. We shouldn't have to wait for an RFO or jump on WHT digging for answers. They should be better equipped to handle these types of issues.
Posted by gordonrp, 01-17-2013, 03:44 PM

Quote:

Originally Posted by Rob T

There's a simple solution for anyone complaining about their gear going down - pay for redundant power. When you signed up for primary power only, you did so accepting the risk that comes along with having a non-redundant aspect to your service.

Rob, the issue is this is the 2nd power issue in 12 months. At no DC is that acceptable. Many of us do use redundant power, we're not ignorant. Reporting & handling of issues needs to improve.

EDIT: outage threads are for customers only.
Posted by Ionity, 01-17-2013, 04:04 PM

Quote:

Originally Posted by gordonrp

a third will be unreasonable.

I think technically this is the third?

Are you counting this one below? It might not have taken down your gear on 7/29, so maybe you didn't get this notice?

Quote:

Dear CoreXchange Customer,

Please see below for CoreXchange's Reason For Outage (RFO).

Date: 07/29/2012

Time: 11:15 CDT

Data Center: DLLSTX03 (8600 Harry Hines Blvd)

Description:

On July 29th 2012 at approximately 11:15am CDT, Power Distribution Units
2B and 3B at Data Center DLLSTX03 (The CoreXchange Connection) experienced
a phase sequence error which resulted in a momentary power brownout to
customer equipment. CoreXchange Facilities Engineers were dispatched to
diagnose the source of the issue. CoreXchange Facilities Engineers determined
the root cause as a capacitor failure in UPS1. The effected equipment was
repaired and all services returned to normal.

We apologize for the inconvenience that this has caused you. Our support
staff is ready to assist you with reboots or any other remote hands and eyes
services that you may need. If you have any questions or need assistance,
please contact our Network Operations Center. For the fastest response
please open a ticket via our customer portal at
http://corecontrol.corexchange.com or email support at support@corexchange.com.
We can also be reached via phone at 214-442-1111 option 3.

CoreXchange NOC
support@corexchange.com
214-442-1111 option 3

Posted by gordonrp, 01-17-2013, 04:30 PM

Quote:

Originally Posted by Ionity

I think technically this is the third?

Are you counting this one below? It might not have taken down your gear on 7/29, so maybe you didn't get this notice?

I stand corrected. I don't think we got hit by that one, not sure. I remember us chatting about something at the time.
Posted by Purevoltage, 01-17-2013, 09:47 PM
Picking up a second feed is always a good option for people who really care about downtime, however it's never fun having to deal with issues like this.

****@pv264:~# uptime
19:52:12 up 224 days, 2:34, 1 user, load average: 1.24, 1.20, 1.23

Everyone wants to at least know about issues like this from that provider even just a quick email sent out. This is mostly so everyone can check on their hardware to make sure everything is back up and running.

They have been a very good provider for the price hopefully this is the last power issue they have.
Posted by lxer, 01-18-2013, 10:02 PM
They have finally sent out the RFO.
Posted by AgServer, 01-19-2013, 12:26 AM
Really wish their switches were on A/B power so that a power issue doesn't take out the network.