HE Routing Issue

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > HE Routing Issue

Posted by dmst, 04-10-2011, 05:35 PM
Hurricane Electric is dropping packets at 30-50% in their core. Here's a current mtr into my network in FMT2 showing packet loss starting between DEN and LAX.

My traceroute [v0.71]
zaphod (0.0.0.0) Sun Apr 10 16:33:38 2011
Resolver: Received error response 2. (server failure)er of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. uiuc-mathdep-net.gw.uiuc.edu 0.0% 1309 0.4 3.0 0.3 232.3 11.6
2. t-core1.gw.uiuc.edu 0.0% 1309 0.3 0.4 0.3 39.2 3.1
3. t-exit1.gw.uiuc.edu 0.0% 1309 0.3 2.0 0.2 80.5 8.2
4. t-fw1.gw.uiuc.edu 0.0% 1309 0.7 0.6 0.5 25.2 0.7
5. t-exite1.gw.uiuc.edu 0.0% 1309 0.8 2.4 0.7 86.0 8.2
6. t-exite2.gw.uiuc.edu 0.0% 1308 0.9 3.1 0.8 87.3 10.0
7. t-dmzb.gw.uiuc.edu 0.0% 1308 1.0 1.0 1.0 20.3 1.0
8. ur2rtr-uiuc.ex.ui-iccn.org 0.0% 1308 1.0 2.5 0.9 23.2 3.2
9. t-710rtr.ix.ui-iccn.org 0.0% 1308 7.8 3.7 3.2 15.9 1.6
10. i2-cps.ex.ui-iccn.org 4.9% 1308 149.8 85.4 3.7 288.4 61.4
11. 10gigabitethernet4-1.core1.chi1. 0.0% 1308 4.7 7.0 4.5 25.2 3.4
12. 10gigabitethernet3-2.core1.den1. 0.0% 1308 57.4 57.7 57.0 95.6 1.5
13. 10gigabitethernet1-4.core1.lax2. 37.0% 1308 83.1 75.0 69.2 93.3 4.0
14. 10gigabitethernet2-1.core1.lax1. 42.0% 1308 72.2 74.6 69.2 101.3 4.1
15. 10gigabitethernet2-2.core1.fmt2. 39.4% 1308 84.6 75.5 69.6 107.3 4.6

I've noticed a few other users reporting similar issues on twitter.

I'll update this as I learn more.
Posted by DeltaAnime, 04-10-2011, 05:38 PM
We're seeing the issue on their San Jose link.

It's very hit/miss as their PAO links seem just fine

Francisco
Posted by cpservr, 04-10-2011, 05:39 PM
yeah me too. and my main site is going down now. the good side is, now i know my load balancing/failover is not configured correctly.
Posted by dmst, 04-10-2011, 05:48 PM
Seattle->SJC->Palo Alto->Freemont 2
Host Loss% Last Avg Best Wrst StDev
1. vl101-d1.acc.sea2.hopone.net 0.0% 0.5 0.5 0.4 1.9 0.2
2. gec1.core2.sea2.hopone.net 0.0% 0.4 0.3 0.3 0.4 0.0
3. ge6-1.core1.sea1.hopone.net 0.0% 1.4 1.6 1.4 4.6 0.5
4. 10gigabitethernet1-3.core1.sea1.he.net 0.0% 1.1 2.4 1.0 12.2 2.7
5. 10gigabitethernet9-1.core1.sjc2.he.net 32.8% 26.0 28.4 25.9 36.7 3.4
6. 10gigabitethernet3-2.core1.pao1.he.net 55.2% 30.2 28.6 25.8 37.6 3.5
7. 10gigabitethernet1-4.core1.fmt2.he.net 40.9% 26.7 29.4 26.5 37.7 3.6

Ashburn->Palo Alto->Freemont 2

Packets Pings
Host Loss% Last Avg Best Wrst StDev
1. h-vl202-d3.acc.dca2.hopone.net 0.0% 0.8 0.9 0.2 45.0 4.0
2. gec13.core1.dca2.hopone.net 0.0% 0.2 0.2 0.1 1.2 0.1
3. ge1-2.core1.iad1.hopone.net 0.0% 1.9 1.5 1.4 2.2 0.1
4. 10gigabitethernet2-2.core1.ash1.he.net 0.0% 3.7 6.3 2.6 16.0 3.5
5. 10gigabitethernet1-4.core1.pao1.he.net 0.0% 73.9 77.3 73.8 89.7 3.6
6. 10gigabitethernet1-4.core1.fmt2.he.net 0.0% 82.0 85.8 81.6 126.2 4.9
Posted by DeltaAnime, 04-10-2011, 06:04 PM
Things look to be back to normal now.

I'll be ticketing HE for an RFO to find out what exactly broke.

Francisco
Posted by Orien, 04-10-2011, 06:08 PM
Yep, I was seeing packet loss in San Jose as well, but back to normal now.
Posted by dmst, 04-10-2011, 06:14 PM
Move along.
Move along.
Posted by stablehost, 04-10-2011, 06:19 PM
*raises his hand* us too!
Posted by (Stephen), 04-12-2011, 08:16 AM
It is back one of our clients hitting an HE route incoming from across the pacific here. Anyone else seeing it back to bad responses again?
Posted by dmst, 04-12-2011, 02:24 PM
Today I received the following RFO:

At approximately 2:10pm (PDT) on Sunday, April 10th, an outward facing
interface on our Equinix San Jose router experienced an issue. The issue
manifested itself as intermittent packet loss. An engineer was alerted
and the router was reloaded to correct the condition.

We are continuing to explore possible causes.

+---------------- H U R R I C A N E - E L E C T R I C ----------------+
|Alex Broque Wholesale IPv4 and IPv6 Transit 510 580 4100|
| Internet Backbone & Colocation AS6939|
|Network Engineer & Operations Lead http://he.net|
+---------------------------------------------------------------------+