Connectivity Issues BurstNet

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Connectivity Issues BurstNet

Posted by LV-Matt, 05-23-2011, 02:41 PM
Anyone else having some connectivity issues inside BurstNet.

Quote:

5 xe-11-1-0-rt001.the.as13285.net (62.24.240.6) 98.669 ms
xe-11-3-0-rt002.man.as13285.net (62.24.240.129) 40.204 ms
xe-11-1-0-rt001.sov.as13285.net (62.24.240.14) 43.626 ms
6 xe-11-1-0-rt001.the.as13285.net (62.24.240.6) 43.935 ms
xe-10-3-0-scr001.sov.as13285.net (78.144.1.132) 48.683 ms
host-78-144-0-200.as13285.net (78.144.0.200) 45.032 ms
7 host-78-144-0-206.as13285.net (78.144.0.206) 46.768 ms
xe-11-0-0-scr001.log.as13285.net (78.144.0.245) 48.026 ms
host-78-144-0-204.as13285.net (78.144.0.204) 45.379 ms
8 highwinds-pp-thn.as13285.net (78.144.3.42) 52.898 ms
xe-10-2-0-scr010.thn.as13285.net (78.144.1.2) 47.528 ms
highwinds-pp-thn.as13285.net (78.144.3.42) 62.339 ms
9 4-1.r1.ny.hwng.net (69.16.190.209) 120.319 ms
highwinds-pp-thn.as13285.net (78.144.3.42) 45.636 ms
4-1.r1.ny.hwng.net (69.16.190.209) 124.049 ms
10 e8-1.r1.dc.hwng.net (209.197.0.41) 127.016 ms
4-1.r1.dc.hwng.net (69.16.190.18) 123.906 ms
4-1.r1.ny.hwng.net (69.16.190.209) 125.742 ms
11 209.197.18.18 (209.197.18.18) 169.241 ms
e8-1.r1.dc.hwng.net (209.197.0.41) 126.207 ms
209.197.18.18 (209.197.18.18) 125.486 ms
12 209.197.18.18 (209.197.18.18) 124.772 ms * 123.351 ms
13 * 209.197.18.22 (209.197.18.22) 137.296 ms 130.089 ms
14 ec0-61.bf2803.sctn01.hostnoc.net (96.9.184.61) 125.274 ms 132.050 ms *
15 * * *
16 * * *
17 * * *

This is affecting all my nodes inside BurstNet.
Posted by teck, 05-23-2011, 03:10 PM
I tested it from 2 locations and it seems okay to me. Pings go through without any packet loss. Did you open a ticket cause they will be here in 3 minutes asking for a ticket number.
Posted by LV-Matt, 05-23-2011, 03:13 PM

Quote:

Originally Posted by teck

I tested it from 2 locations and it seems okay to me. Pings go through without any packet loss. Did you open a ticket cause they will be here in 3 minutes asking for a ticket number.

I am trying to it seems to be intermittent to me.
Posted by BurstNET_CSM, 05-23-2011, 03:14 PM
We are experiencing an issue with one of our 10G circuits between Scranton and Philadelphia.

We are aware and are looking into the issue with the circuit provider, but have no ETA at this time.

You may experience some packet loss and delay while we work on this issue.

Thank You

P.S. it was four minutes
Posted by LV-Matt, 05-23-2011, 03:15 PM

Quote:

Originally Posted by BurstNET_CSM

We are experiencing an issue with one of our 10G circuits between Scranton and Philadelphia.

We are aware and are looking into the issue with the circuit provider, but have no ETA at this time.

You may experience some packet loss and delay while we work on this issue.

Thank You

Thanks, much appreciated.
Posted by RedTechie, 05-23-2011, 07:52 PM
I am also having a bunch of issues accessing burst.net's network.

But my issues seem to be coming from another hop. The 209.197.18.22 hop seems to be the problem. It has very high ping spikes and some packet loss. I'm not sure if this is related or not. Some confirmation would be nice. Thanks!

Quote:

Tracing route to *** [***]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms *** [***]
2 5 ms 7 ms 5 ms 96.103.92.1
3 6 ms 5 ms 5 ms ge-2-43-ur01.killingworth.ct.hartford.comcast.net [68.85.187.37]
4 7 ms 6 ms 8 ms te-9-1-ur01.clinton.ct.hartford.comcast.net [68.87.182.42]
5 18 ms 13 ms 10 ms te-0-10-0-2-ar01.chartford.ct.hartford.comcast.net [68.85.162.190]
6 12 ms 12 ms 12 ms pos-1-5-0-0-ar01.needham.ma.boston.comcast.net [68.85.162.73]
7 20 ms 19 ms 19 ms pos-2-1-0-0-cr01.newyork.ny.ibone.comcast.net [68.86.95.29]
8 21 ms 18 ms 20 ms pos-1-3-0-0-pe01.111eighthave.ny.ibone.comcast.net [68.86.86.190]
9 28 ms 25 ms 24 ms 66.208.228.146
10 25 ms 25 ms 23 ms 4-1.r1.dc.hwng.net [69.16.190.18]
11 26 ms 25 ms 24 ms 209.197.18.18
12 115 ms 44 ms 219 ms 209.197.18.22
13 82 ms 29 ms 41 ms xe1-01.agg02.sctn01.hostnoc.net [64.191.116.242]
14 31 ms 29 ms 30 ms ec1-15.f2801.sctn01.hostnoc.net [96.9.156.66]
15 31 ms 31 ms 29 ms *** [***]

Trace complete.

Posted by Patrick, 05-23-2011, 11:20 PM
Ditto. Seeing a lot of problems at xe1-01.agg01.sctn01.hostnoc.net ... been going on for a few hours now. Sigh.
Posted by RedTechie, 05-24-2011, 01:00 AM
This is absolutely crazy... It's been like this all day. In addition to the massive packet loss sessions; we must of been completely dropped from our servers 30 times now!

I have a support ticket going but they keep saying nothing is wrong. Even after giving them tons of trace routes, pathping's, and ping's all from different sources.

Very very frustrating especially when your trying to get new clients setup on the same servers!
Posted by BurstNET_CSM, 05-24-2011, 08:34 AM
As we said, there is a problem with one of the 10G circuits that connects Scranton to Philly. It's our provider's issue and there's nothing we can do but remind them that this is affecting our customers.

Right now it seems it is dropping the connection every four hours or so, which is causing timeouts and packet loss. We ARE doing what we can (As noted above, however, it's nothing we can actually fix ourselves).
Posted by RedTechie, 05-24-2011, 09:15 AM
BurstNET_CSM: Thanks for the update on the issue! This makes me feel slightly better that you are aware of this problem. As the tech's in the ticket I have created keep telling me there is no problem.

Question: To verify, the 209.197.18.22 hop is related to this particular connection issue?
Posted by BurstNET_CSM, 05-24-2011, 09:37 AM

Quote:

Originally Posted by RedTechie

BurstNET_CSM: Thanks for the update on the issue! This makes me feel slightly better that you are aware of this problem. As the tech's in the ticket I have created keep telling me there is no problem.

Can you PM me the ticket where you were told this, please?

Quote:

Originally Posted by RedTechie

Question: To verify, the 209.197.18.22 hop is related to this particular connection issue?

I will ask out networking team.
Posted by RedTechie, 05-24-2011, 09:48 AM
BurstNET_CSM: The forum keeps saying I don't have permission to send you a PM. The ticket ID is 2694323.
Posted by BurstNET_CSM, 05-24-2011, 10:48 AM
It's because you have 0 posts. You need 10 posts to be able to send PM's.

Thanks.
Posted by RedTechie, 05-24-2011, 03:56 PM
BurstNET_CSM: Were you able to verify that the hop in my previous post is involved with this problem?
Posted by RedTechie, 05-24-2011, 06:46 PM
I believe there is a routing loop going on.

I pinged the hop before 209.197.18.22 which is 209.197.18.18 I got the following:

Quote:

Reply from 209.197.18.22: TTL expired in transit

So we increased the hop count for this IP to 255...and what do you know I got a result with very high latency (normal latency should be around 30ms for me):

Quote:

ping -i 255 209.197.18.18

Pinging 209.197.18.18 with 32 bytes of data:
Reply from 209.197.18.18: bytes=32 time=554ms TTL=245
Reply from 209.197.18.18: bytes=32 time=575ms TTL=245
Reply from 209.197.18.18: bytes=32 time=556ms TTL=245
Reply from 209.197.18.18: bytes=32 time=559ms TTL=245

Ping statistics for 209.197.18.18:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 554ms, Maximum = 575ms, Average = 561ms

We then tried to test for a routing loop using "tracert -w 500 209.197.18.18" we got the following:

Quote:

1 <1 ms <1 ms <1 ms *** [***]
2 7 ms 5 ms 16 ms ***
3 8 ms 9 ms 11 ms *** [***]
4 12 ms 15 ms 12 ms nrth-bb-1a-as6-0.network.virginmedia.net [212.43.163.217]
5 15 ms 14 ms 32 ms popl-bb-1b-as4-0.network.virginmedia.net [213.105.64.18]
6 35 ms 15 ms 73 ms popl-tmr-2-ae5-0.network.virginmedia.net [213.105.159.6]
7 18 ms 14 ms 14 ms tele-ic-2-as0-0.network.virginmedia.net [62.253.184.6]
8 17 ms 13 ms 13 ms 134-14-250-212.static.virginmedia.com [212.250.14.134]
9 88 ms 81 ms * 4-1.r1.ny.hwng.net [69.16.190.209]
10 152 ms 94 ms 87 ms 4-1.r1.dc.hwng.net [69.16.190.18]
11 92 ms 129 ms * 209.197.18.18
12 95 ms 95 ms 232 ms 209.197.18.22
13 99 ms 99 ms * 209.197.18.18
14 119 ms 100 ms 153 ms 209.197.18.17
15 101 ms 101 ms * 209.197.18.18
16 104 ms 104 ms 119 ms 209.197.18.22
17 108 ms 110 ms * 209.197.18.18
18 132 ms 111 ms 109 ms 209.197.18.17
19 117 ms 109 ms * 209.197.18.18
20 113 ms 113 ms 113 ms 209.197.18.22
21 118 ms 118 ms * 209.197.18.18
22 120 ms 124 ms 155 ms 209.197.18.17
23 118 ms 119 ms * 209.197.18.18
24 358 ms 122 ms * 209.197.18.22
25 140 ms 127 ms 125 ms 209.197.18.18

Although this last test seems to be intermittent where it sometimes produces this looping result.
Posted by RedTechie, 05-24-2011, 07:07 PM
Sorry for the multi-posts. I would edit my previous post but the forum won't allow me to do that either.

Here is more proof that there is a routing loop:

Quote:

ping 209.197.18.18 -i 11

Pinging 209.197.18.18 with 32 bytes of data:
Reply from 209.197.18.18: bytes=32 time=25ms TTL=245
Reply from 209.197.18.18: bytes=32 time=24ms TTL=245
Reply from 209.197.18.18: bytes=32 time=27ms TTL=245
Reply from 209.197.18.18: bytes=32 time=24ms TTL=245

Ping statistics for 209.197.18.18:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 24ms, Maximum = 27ms, Average = 25ms

ping 209.197.18.18 -i 12

Pinging 209.197.18.18 with 32 bytes of data:
Reply from 209.197.18.22: TTL expired in transit.
Reply from 209.197.18.22: TTL expired in transit.
Reply from 209.197.18.22: TTL expired in transit.
Reply from 209.197.18.22: TTL expired in transit.

Ping statistics for 209.197.18.18:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

ping 209.197.18.18 -i 13

Pinging 209.197.18.18 with 32 bytes of data:
Reply from 209.197.18.18: bytes=32 time=33ms TTL=245
Reply from 209.197.18.18: bytes=32 time=41ms TTL=245
Reply from 209.197.18.18: bytes=32 time=33ms TTL=245
Reply from 209.197.18.18: bytes=32 time=34ms TTL=245

Ping statistics for 209.197.18.18:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 33ms, Maximum = 41ms, Average = 35ms

ping 209.197.18.18 -i 14

Pinging 209.197.18.18 with 32 bytes of data:
Reply from 209.197.18.17: TTL expired in transit.
Reply from 209.197.18.17: TTL expired in transit.
Reply from 209.197.18.17: TTL expired in transit.
Reply from 209.197.18.17: TTL expired in transit.

Ping statistics for 209.197.18.18:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Notice that every other hop gives me a TTL expired. Out of the hops that give me the "TTL expired" I get a reply from either 209.197.18.22 OR 209.197.18.17 stating that the TTL has expired.