GIPNetworks / GlobalCon Down

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > GIPNetworks / GlobalCon Down

Posted by Calibur747, 06-17-2011, 04:59 PM
As of 3:50pm Central Standard Time, gipnetworks.com is offline as is a server we have at their colo.
Posted by Calibur747, 06-17-2011, 05:20 PM
Currently 4:20pm CST, still down. This is the half hour mark...
Posted by matrixdatatech, 06-17-2011, 05:21 PM
Odd. From my FIOS connection at home, I get this:

traceroute to heap.pbp.net (67.210.225.171), 64 hops max, 52 byte packets
1 trafficjam.home.pbp.net (192.168.1.1) 0.749 ms 0.452 ms 0.422 ms
2 l100.dllstx-vfttp-77.verizon-gni.net (71.97.84.1) 8.497 ms 6.833 ms 7.645 ms
3 67.210.225.171 (67.210.225.171) 7.329 ms !H 7.295 ms !H *

From other web based traceroute tools? They all time out. Tried one at princeton, sprint, socket.net.. they all fail. They work when I test to something like cmu.edu, though.

How strange.

EDIT: socket.net just came back.

traceroute to 67.210.225.171 (67.210.225.171), 30 hops max, 40 byte packets
1 fw1.como.socket.net (216.106.88.148) 0.507 ms 0.374 ms 0.488 ms
2 MLX1.CLMAMOXA.SOCKET.NET (64.85.192.205) 1.590 ms 1.580 ms 6.082 ms
3 gw2.kscymo.socket.net (216.106.31.133) 7.598 ms 7.708 ms 7.512 ms
4 gi1-1.ccr01.stl03.atlas.cogentco.com (38.104.162.53) 14.728 ms * 5.769 ms
5 * * *
6 * *
Posted by Calibur747, 06-17-2011, 05:23 PM
Interesting, still down for everything I have access to.
Posted by Calibur747, 06-17-2011, 05:26 PM
Attempted to ping atlast.cogentco and am getting nothing there. Could be a DNS issue with Cogent? You'd think GIP was multi-homed..
Posted by matrixdatatech, 06-17-2011, 05:28 PM
another odd traceroute..

FROM www.sdsc.edu TO 67.210.225.171 at Fri Jun 17 14:26:11 PDT 2011.
traceroute: Warning: Multiple interfaces found; using 132.249.21.58 @ bge0:1
traceroute to 67.210.225.171 (67.210.225.171), 30 hops max, 40 byte packets
1 lightning.sdsc.edu (132.249.20.6) 0.910 ms 0.543 ms 0.593 ms
2 dolphin.sdsc.edu (132.249.31.17) 18.952 ms 1.341 ms 0.482 ms
3

Same trace utility works to other hosts.
Posted by Calibur747, 06-17-2011, 05:32 PM
Things are starting to come back online...slowly. Getting hit or miss on each packet.
Posted by freethought, 06-18-2011, 04:42 AM
GIP (AS7819) appears to be multihomed onto AS12179 (Internap) and AS6461 (AboveNet) as well as a couple of paths via AS26769 (Bandcon).

At 20:50 UTC/GMT yesterday all BGP announcements from AS7819 disappeared from the global routing table, making them completely unreachable. This is why traceroutes were dying so early; as each provider's core network simply had no routing information about where to deliver the traffic.

BGP re-announcements started at about 21:30 UTC/GMT. It must be a large scale network failure to completely isolate GIP from the internet for 40 minutes.
Posted by matrixdatatech, 06-21-2011, 01:13 AM
Looks like it was a power outage.

Ironically, when I was trying to post more information at the time of the outage, WHT was acting like IT was totally offline too.

So I went and got some tacos. By the time I was finished, I was getting traffic from my server again.

The email from GIP:

"On Friday 06/17/2011 (today), about 3:30 pm CST, we lost power from the grid. Here is the time line:

- 3:30pm CST we lost power from the grid, UPS bridge the load to the generator

- 3:43pm CST we lost the generator power due to generator over heat

- 3:53pm CST we lost the power from the UPS, which brought our data center down

- 4:26pm CST power is restored by running the generator after it cooled down

As of now, we are running by generator power as we wait for the power grid to restore.

The reason of the generator issue is still not known, specially after being maintained (regular PM) by qualified generator engineer three days ago on 6/14/2011 with very good report. At the time of testing, the entire facility is running at 51% of the generator capacity.

According to our generator technician, it could have been that the generator was over heated due to high humidty and 104 degrees temperature outside. So, from 3:43pm to 4:26pm we did everything we can to get the generator cooled. As soon as the generator cooled down, we manually start it up with all the doors open and it has been running continuously since then.

Some of you may asked about our 2nd generator which will be installed this month. As of now, our 2nd generator is currently in our vendor's warehouse in Forth Worth, ready to be delivered and installed. The only thing holding it is the city permit that is pending. Therefore, we are renting another generator which will be here within an hour at the time of writing as a backup in case of prolonged power outage.

Thank you for your patience in this unfortunate event. If you have any questions, please do not hesitate to write us a support ticket or to call us. We are standing by.
"

A friend at the Plano Fire Department also reported that there were many heat related power outages in the area, and the heat index was about 109 in many areas. really humid too.
Posted by Calibur747, 06-21-2011, 10:26 AM
I always enjoy a good report. It is interesting that they do this for GIP but not their hosting division, of which was also offline ( we have a few old dedicated boxes there too ). Oh well, sounds like it was a hot one and nothing overly mission critical!

Back to the ol' eggs in one basket avoidance.