GIP Networks Down - Plano - Knowledgebase

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > GIP Networks Down - Plano

GIP Networks Down - Plano

Posted by speckl, 03-13-2013, 11:12 AM
I am sick and tired of this crap. Fix your power, fix your network.
Posted by kris1351, 03-13-2013, 11:14 AM
Stinks to hear that, they started out so good and we toured the facility. Was thinking about putting a remote setup just for offsite DNS/Backups/CDN pop since it is close to my office. They only had one power feed when we toured, did they update that?
Posted by speckl, 03-13-2013, 11:16 AM
Yes. They just upgraded to N+1. Clearly, it isn't working.
Posted by kris1351, 03-13-2013, 11:17 AM
N+1 can mean a lot of things, did they get a second utility pull or just put in new UPS/Generator setups?
Posted by rjtx667, 03-13-2013, 01:40 PM
They have a new service entrance from a diverse sub station with a new transformer/ ATS/ Generator.

I am not sure what happened today, but I will be finding out.
Posted by speckl, 03-13-2013, 03:07 PM
Yes, they apparently did power maintenance at 10AM CST this morning. Something went bad and caused the entire DC to go down.

This is the second power outage in a years time that I've noticed.
Multiple network issues since then too.
Posted by rey, 03-14-2013, 01:09 PM
Speckl,

We hear you and I understand that no amount of apologies will be enough. If I was in your position, I would be very upset too and the least we can do is to fully disclose yesterday's incident. I also understand that you would like to stay anonymous and I'd be glad to share the letter we sent to all our immediate customers yesterday, just in case you have not received one from us.

Quote:

Dear Customer,

First, we would like to thank many of you who were very patience and understanding in this event. Your generosity and kindness are true source of encouragements for us.

After receiving many calls, these are the questions that we received the most:

1. Why the upgrade?

2. Why today?

3. What went wrong?

4. Will this happen again?

5. What is the status of this power upgrade?

So ...

1. Why the upgrade?

After what happened today, some of us may think "Is this upgrade even worth it?" If I knew that this would happen, my answer is absolutely not. Although we have spent over 1 million dollar and more than 8 months for this project, your uptime is still more important than all of these. As a matter of fact, we would not even start this project if we didn't get an assurance from our vendor that this upgrade can be achieved without service interruption. Upgrading power infrastructure from N+1 to 2(N+1) is an expensive and exhaustive project, but because we believe in safe guarding your business and to give you a better infrastructure and peace of mind by having 4 power sources for any of your power circuit without increasing the price on your service, we decided to move on.

2. Why today?

Why not at night or why not during the weekend or off hours? These are the questions we also asked our vendors. Last Saturday, we issued a maintenance window as we merged the new power grid to the current system. The paralleling was a success and the last thing needed to be done was to synchronize the bypass line of UPS A to that of UPS B. Therefore, since merging both UPS with synchronized bypass would be very easy compared to asynchronous bypass, we were assured by both our electricians and UPS technicians that this will be a very simple task with no risk and can be done at any given time without issues.

3. What went wrong?

In the process of bringing both UPS up from the bypass mode, the critical load dropped for a split second while both UPS were online. After sorting through every steps that were done today and Saturday, we suspect that a temporary jumper that was used for the UPS to operate when the bypass was not synchronized, was also being today while the UPS bypass is synchronized, might cause the UPS to reset and dropped the load for a split second. However, to clear all speculation, we have sent the UPS data to the vendor for analysis.

4. Will this happen again?

Very unlikely, due to the fact that we cannot recreate the issue in the first place after the temporary jumper was removed and all bypass is synchronized. Since it was a human error on the UPS side and has nothing to do with the equipment itself, we do not foresee that this issue will reoccur. To verify, we tested both UPS after the incident in all possible scenarios and everything work flawlessly as expected just as they have always been in the past.

5. What is the status of this power upgrade?

On a positive note, this power upgrade project is now complete and all your power circuits are now protected and conditioned by redundant UPS, fed by 2 active power grids, each with its own backup generator. I was only hoping that I could announce the completion of this project in a very different way.

All our staffs are standing by to assist you in every way possible. Please do not hesitate to contact us for any questions or concerns. We are truly grateful for your support.

Sincerely,

Reyner Natahamidjaja
President
214-432-0419 x505
reyner@gipnetworks.com
Global IP Networks, Inc.

If it would help, I'd be happy to send you a formal letter on our company letterhead that you can use for your customer to an anonymous email of your choice. Please let me know if there is anything I can do to work with you and explain to your customers that the fault is not on your end, and to give them assurance of what was done to make sure this issue will not ever reoccur.

Sincerely,