level3.net problems - too much packet loss - Knowledgebase

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > level3.net problems - too much packet loss

level3.net problems - too much packet loss

Posted by sharmaine1111, 03-08-2010, 10:40 AM
Ive seen so many servers not only from burst datacenter experiencing packet loss. Most if not all are servers going through level3.net

Is level3.net having problems? Are their services are not up to par?
Posted by sharmaine1111, 03-08-2010, 10:41 AM
Even WHT has packet loss!

Quote:

TraceRoute to 69.20.126.12 [webhostingtalk.com]
Hop (ms) (ms) (ms) IP Address Host name
1 11 6 31 72.249.128.109 -
2 27 64 52 8.9.232.73 xe-5-3-0.edge3.dallas1.level3.net
3 44 Timed out 89 4.69.145.116 ae-73-70.ebr3.dallas1.level3.net
4 56 82 Timed out 4.69.134.22 ae-7-7.ebr3.atlanta2.level3.net
5 55 56 53 4.69.132.86 ae-2-2.ebr1.washington1.level3.net
6 49 53 53 4.69.134.130 ae-61-61.csw1.washington1.level3.net
7 64 49 105 4.68.17.18 ae-1-69.edge1.washington4.level3.net
8 64 52 46 4.53.112.46 rackspace-m.edge1.washington4.level3.net
9 58 62 63 69.20.1.40 vlan903.core3.iad1.rackspace.com
10 59 48 47 69.20.3.11 aggr102a.iad1.rackspace.com
11 52 53 56 69.20.126.12 -

Trace complete

Posted by FastServ, 03-08-2010, 11:23 AM
In fact according to your trace, WHT has zero packet loss.

Don't assume just because burst.net has packetloss the whole internet does!
Posted by sharmaine1111, 03-08-2010, 11:33 AM
What do you mean zero packet loss? Have you really checked it?

Have you checked the "Timed out"??

Quote:

TraceRoute to 69.20.126.12 [webhostingtalk.com]
Hop (ms) (ms) (ms) IP Address Host name
1 11 6 31 72.249.128.109 -
2 27 64 52 8.9.232.73 xe-5-3-0.edge3.dallas1.level3.net
3 44 * 89 4.69.145.116 ae-73-70.ebr3.dallas1.level3.net
4 56 82 * 4.69.134.22 ae-7-7.ebr3.atlanta2.level3.net
5 55 56 53 4.69.132.86 ae-2-2.ebr1.washington1.level3.net
6 49 53 53 4.69.134.130 ae-61-61.csw1.washington1.level3.net
7 64 49 105 4.68.17.18 ae-1-69.edge1.washington4.level3.net
8 64 52 46 4.53.112.46 rackspace-m.edge1.washington4.level3.net
9 58 62 63 69.20.1.40 vlan903.core3.iad1.rackspace.com
10 59 48 47 69.20.3.11 aggr102a.iad1.rackspace.com
11 52 53 56 69.20.126.12 -

Trace complete

There are packet loss here:

ae-73-70.ebr3.dallas1.level3.net
ae-7-7.ebr3.atlanta2.level3.net

Please read carefully before you comment

Quote:

Originally Posted by FastServ

In fact according to your trace, WHT has zero packet loss.

Don't assume just because burst.net has packetloss the whole internet does!

And please, just because you see Time Out instead of *, dont assume it does not have any packet loss! *rolleyes*

And I have seen several servers with packet loss too w/c is not from burst:

Quote:

19 495 ms 485 ms * ae-2-2.ebr2.Frankfurt1.Level3.net [4.69.132.138]
21 * 497 ms * ae-2-2.ebr2.Chicago2.Level3.net [4.69.132.69]
22 499 ms * 519 ms ae-1-100.ebr1.Chicago2.Level3.net [4.69.132.113]

23 * 515 ms 504 ms ae-3-3.ebr2.Denver1.Level3.net [4.69.132.61]
24 495 ms * 496 ms ae-2-2.ebr2.Seattle1.Level3.net [4.69.132.53]
25 * * 496 ms ae-23-52.car3.Seattle1.Level3.net [4.68.105.36]

26 * 496 ms 506 ms metapeer.car3.Seattle1.Level3.net [4.71.152.18]

Duh!
Posted by qualityhostings, 03-08-2010, 12:49 PM
3 timeouts?

http://screencap.net/f/32a6555d.png

http://screencap.net/f/aceee858.png

</>
Posted by FastServ, 03-08-2010, 12:49 PM
By policy, routers drop ICMP packets at certain hops. It doesn't mean there are routing issues. In your trace:

8 64 52 46 4.53.112.46 rackspace-m.edge1.washington4.level3.net
9 58 62 63 69.20.1.40 vlan903.core3.iad1.rackspace.com
10 59 48 47 69.20.3.11 aggr102a.iad1.rackspace.com
11 52 53 56 69.20.126.12 -

There is zero loss on the rackspace/level3 peer nor inside rackspace network. All that matters is the final hop (destination), and your trace shows zero loss there.

Any competent network engineer is going to back me up in my statement that your traceroutes show nothing out of the ordinary.
Posted by MaB, 03-08-2010, 02:29 PM

Quote:

Originally Posted by FastServ

By policy, routers drop ICMP packets at certain hops. It doesn't mean there are routing issues. In your trace:
...

Any competent network engineer is going to back me up in my statement that your traceroutes show nothing out of the ordinary.

Absolutely correct. If there were *true* packet loss, it would logically have to follow down into the other steps of the traceroute... how could there be packet loss at hops 4 and 5, but none at hop 6? Packets don't just magically re-appear.
Posted by tomydurden, 03-08-2010, 04:02 PM

Quote:

Originally Posted by sharmaine1111

Ive seen so many servers not only from burst datacenter experiencing packet loss. Most if not all are servers going through level3.net

Is level3.net having problems? Are their services are not up to par?

Depending on your testing methods, you may be receiving a false positive. If you see packet loss in the path, but none at the end point, then you're likely seeing rate limiting and not packet loss.

Many Tier1 bandwidth providers and hosts set ratelimiting for packets which are destined to a router. This doesn't affect packets which are going through the router.

I know that Level3 does this on many of their Customer Access Routers, which is why I was interested in seeing the trace mentioned via Twitter.
Posted by railto, 03-08-2010, 04:13 PM

Quote:

Originally Posted by MaB

Absolutely correct. If there were *true* packet loss, it would logically have to follow down into the other steps of the traceroute... how could there be packet loss at hops 4 and 5, but none at hop 6? Packets don't just magically re-appear.

couldnt agree more
Posted by BurstJoeM, 03-08-2010, 08:19 PM
This is correct, unless the timeouts show up throughout the path of the traceroute, the issue is ICMP ratelimiting on the interface.

The other thing to look at is the reverse path from back to where your originating the traceroute. If you traceback to 72.249.128.109 you'll see this as the path:

traceroute 72.249.128.109
1 as45577.mel03.vic.intervolve.com.au (122.100.10.2) 0.327 ms
2 as45577.mel01.vic.intervolve.com.au (114.111.137.177) 0.894 ms
3 as45577.intervolve.vocusconnect.net.au (114.31.197.73) 1.655 ms
4 ge-1-0-6.cor01.syd03.nsw.VOCUS.net.au (114.31.192.82) 13.345 ms
5 ge-0-1-3-126.bdr01.sjc01.ca.VOCUS.net.au (114.31.192.101) 165.508 ms
6 206.223.143.88 165.248 ms
7 lax009-sje007-805-cr2.lax009.internap.net (66.79.147.6) 183.280 ms
8 lax009-phx007-832-cr1.phx007.internap.net (66.79.147.182) 183.207 ms
9 cr2-cr1.phx007.internap.net (66.79.147.174) 183.373 ms
10 dal005-phx007-833-cr1.dal005.internap.net (66.79.147.177) 207.671 ms
11 dal005-dalext1-846-core3.ext1.dal.internap.net (66.79.145.98) 207.537 ms
12 border4.pc1-bbnnet1.ext1.dal.pnap.net (216.52.191.30) 207.856 ms
13 colo4dallas-5.border4.ext1.dal.pnap.net (216.52.189.10) 208.462 ms
14 *
15 *
16 colo4dallas-5.border4.ext1.dal.pnap.net (216.52.189.10) 208.815 ms !H

So the question is, is the packetloss cause by level3 or pnap?

Have you spoke to level3? What do you see if you run a constant ping to the questionable hops? Do you loose packets then?
Posted by FastServ, 03-09-2010, 11:20 AM
To make matters even more interesting some networks rate limit or block TTL=1 packets which can hose the entire traceroute or make certain hops dissapear
Posted by emsjs1, 03-09-2010, 12:08 PM
I am hosted by a datacenter I won't name and yesterday I absolutely saw several time outs up to the last packet for level 3. I don't want to make a fuss about my datacenter so I don't want to paste the traceroute here but I was in the idea that level3 had problems. That was from hop 21 up to the last hop, it is constantly showing time outs

I reported this to the datacenter and told me that problem started with the level3 hop
Posted by FastServ, 03-09-2010, 12:36 PM

Quote:

Originally Posted by emsjs1

I am hosted by a datacenter I won't name and yesterday I absolutely saw several time outs up to the last packet for level 3. I don't want to make a fuss about my datacenter so I don't want to paste the traceroute here but I was in the idea that level3 had problems. That was from hop 21 up to the last hop, it is constantly showing time outs

I reported this to the datacenter and told me that problem started with the level3 hop

If you're seeing packet loss from a certain hop onward, then there is likely a problem. But you can't be sure where it is unless you get a traceroute back in the opposite direction.