PDA

View Full Version : Latency/congestion Issues

Erik
03-20-2007, 09:01 AM
As of late, I've been having issues with packet loss/drop on TCP/UDP traffic to my dedicated box as indicated below. These issues usually arise during NA's peek time between 6-11pm EST. Feedback is appreciated

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~

Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

1 7 ms 7 ms 7 ms 172.30.192.1
2 7 ms 84 ms 107 ms mart-gw.chi-mart.il.cable.rcn.net [207.229.191.1
29]
3 7 ms 7 ms 7 ms ge1-0-2.core2.chsl.il.rcn.net [207.172.19.61]
4 6 ms 7 ms 7 ms ge5-0.border1.eqnx.il.rcn.net [207.172.19.15]
5 131 ms 134 ms 133 ms eqix.ge-0-3-0.ord1.nlayer.net [206.223.119.61]
6 128 ms 123 ms 123 ms 60.po1.ar1.ord1.us.nlayer.net [69.31.111.130]
7 * * * Request timed out.
8 116 ms 118 ms 113 ms 201.216-86-145.nozonenet.com [216.86.145.201]

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~

Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.0.1
2 30 ms 7 ms 76 ms 10.65.112.1
3 12 ms 11 ms 38 ms d226-9-249.home.cgocable.net [24.226.9.249]
4 35 ms 24 ms 15 ms h64-187-46-221.gtcust.grouptelecom.net [64.187.4
6.221]
5 44 ms 71 ms 12 ms GE4-0.WANB-TOROON.IP.GROUPTELECOM.NET [216.18.63
.5]
6 52 ms 24 ms 113 ms 66.59.191.110
7 25 ms 52 ms 34 ms eqix.ge-0-3-0.ord1.nlayer.net [206.223.119.61]
8 25 ms 77 ms 82 ms 60.po1.ar1.ord1.us.nlayer.net [69.31.111.130]
9 * * * Request timed out.
10 30 ms 30 ms 40 ms 201.216-86-145.nozonenet.com [216.86.145.201]

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~

Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

1 <1 ms 4294967291 ms 4294967292 ms 192.168.1.1
2 20 ms 16 ms 16 ms 69.155.200.1
3 24 ms 19 ms 16 ms 10.254.254.5
4 21 ms 21 ms 21 ms 70.129.237.149
5 21 ms 49 ms 24 ms bb2-p7-3.rcsntx.sbcglobal.net [151.164.41.188]
6 31 ms 24 ms 59 ms ex1-p12-0.eqdltx.sbcglobal.net [151.164.40.29]
7 28 ms 46 ms 26 ms 151.164.251.150
8 23 ms 79 ms 36 ms dpr1-ge-6-0-0.dallasequinix.savvis.net [204.70.1
94.33]
9 91 ms 24 ms 21 ms dcr2-so-5-2-0.Dallas.savvis.net [204.70.194.66]

10 22 ms 69 ms 23 ms dcr2-so-6-0-0.dallas.savvis.net [204.70.192.50]

11 49 ms 104 ms 93 ms dcr1-so-5-0-0.chicago.savvis.net [204.70.192.45]

12 91 ms 51 ms 61 ms ber2-pos-1-0-0.Chicago.savvis.net [208.175.10.98
]
13 52 ms 46 ms 59 ms ber2-tenge-3-1.chicagoequinix.savvis.net [204.70
.196.26]
14 60 ms 50 ms 71 ms 208.174.225.202
15 * 47 ms * 208.174.225.202
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~

Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms dslrouter [192.168.1.1]
2 11 ms 10 ms 10 ms 10.1.38.1
3 11 ms 10 ms 10 ms so-0-1-1-0.CORE-RTR2.RES.verizon-gni.net [130.81
.9.21]
4 13 ms 11 ms 47 ms so-6-0-0-0.BB-RTR2.RES.verizon-gni.net [130.81.2
0.18]
5 11 ms 11 ms 11 ms so-7-0-0-0.ASH-PEER-RTR2.verizon-gni.net [130.81
.17.179]
6 11 ms 11 ms 11 ms 130.81.15.14
7 12 ms 12 ms 12 ms xe-4-0-0.cr1.iad1.us.nlayer.net [69.31.31.137]
8 19 ms 18 ms 18 ms so-3-2-0.cr1.nyc3.us.nlayer.net [69.22.142.101]

9 37 ms 36 ms 37 ms so-2-1-0.cr1.ord1.us.nlayer.net [69.22.142.106]

10 36 ms 37 ms 36 ms 60.po1.ar1.ord1.us.nlayer.net [69.31.111.130]
11 * * * Request timed out.
12 36 ms 37 ms 37 ms 201.216-86-145.nozonenet.com [216.86.145.201]

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Justec
03-21-2007, 08:07 AM
I also go through nlayer and pinging higher than usual.

I used to ping like 40 ms over a year ago, but I think my ISP did something to screw that up. But I was still getting like 70-80ms, now im at about 100ms (90ms on monday). Also Im pretty sure the number of hops used to be about 12.

Miami -> SF
Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

1 28 ms 27 ms 27 ms 65.14.252.30
2 27 ms 27 ms 27 ms 65.14.252.30
3 29 ms 27 ms 28 ms 65.14.254.29
4 60 ms 59 ms 59 ms ixc01mia-ge-0-3-0.bellsouth.net [205.152.145.181]
5 57 ms 57 ms 57 ms axr01mia-so-1-3-2.bellsouth.net [65.83.237.168]
6 59 ms 59 ms 59 ms axr00bct-so-0-0-0.bellsouth.net [65.83.236.10]
7 59 ms * 58 ms AXR01BCT-1-0-0.bellsouth.net [65.83.236.55]
8 46 ms 45 ms 45 ms pxr00mia-2-0-0.bellsouth.net [65.83.236.18]
9 45 ms 45 ms 45 ms axr01asm-ge-5-1-0.bellsouth.net [65.83.236.187]
10 47 ms 46 ms 45 ms axr00aep-so-0-0-0.bellsouth.net [65.83.238.40]
11 61 ms 59 ms 59 ms pxr00ash-so-3-1-0.bellsouth.net [65.83.236.208]
12 59 ms 57 ms 57 ms g1-17.ar1.iad1.us.nlayer.net [69.31.30.21]
13 59 ms 59 ms 59 ms xe-4-0-0.cr1.iad1.us.nlayer.net [69.31.31.137]
14 79 ms 79 ms 79 ms so-3-2-0.cr1.nyc3.us.nlayer.net [69.22.142.101]
15 100 ms 99 ms 99 ms so-2-1-0.cr1.ord1.us.nlayer.net [69.22.142.106]
16 99 ms 99 ms 99 ms 60.po1.ar1.ord1.us.nlayer.net [69.31.111.130]
17 * * * Request timed out.
18 97 ms 97 ms 97 ms 201.216-86-145.nozonenet.com [216.86.145.201]

Trace complete.


SF -> Miami
traceroute to justec.us (67.35.87.15), 30 hops max, 46 byte packets
1 241.216-86-149.nozonenet.com (216.86.149.241) 0.302 ms 0.268 ms 0.240 ms
2 109.ae0.cr1.ord1.us.nlayer.net (69.31.111.57) 0.489 ms 0.402 ms *
3 60.ae0.cr1.ord1.us.nlayer.net (69.31.111.129) 0.513 ms 0.411 ms 0.394 ms
4 so-1-1-0.cr1.nyc3.us.nlayer.net (69.22.142.105) 20.398 ms 20.298 ms *
5 69.31.94.46 (69.31.94.46) 20.435 ms 20.408 ms 20.347 ms
6 xer01chi-pos-1-0.bellsouth.net (65.83.236.165) 66.626 ms 66.369 ms 66.805 ms
MPLS Label=808 CoS=0 TTL=127 S=0
7 pxr00chi-so-2-0-0.bellsouth.net (65.83.236.120) 40.987 ms 40.932 ms 41.029 ms
MPLS Label=123216 CoS=0 TTL=127 S=0
8 axr00asm-so-1-1-2.bellsouth.net (65.83.236.203) 70.483 ms 70.998 ms 70.557 ms
MPLS Label=116064 CoS=0 TTL=127 S=0
9 axr01bct-so-0-0-0.bellsouth.net (65.83.236.19) 70.571 ms 70.438 ms 71.422 ms
MPLS Label=130336 CoS=0 TTL=127 S=0
10 AXR00BCT-1-0-0.bellsouth.net (65.83.236.54) 71.129 ms 72.372 ms 71.163 ms
MPLS Label=176496 CoS=0 TTL=127 S=0
11 axr01mia-so-1-1-0.bellsouth.net (65.83.236.11) 70.546 ms 70.495 ms 70.446 ms
MPLS Label=105136 CoS=0 TTL=127 S=0
12 ixc01mia-pos-5-0.bellsouth.net (65.83.237.15) 69.688 ms 69.755 ms 69.780 ms
13 her01mia-ge-1-1.bellsouth.net (205.152.145.182) 70.952 ms 70.882 ms 71.009 ms
14 65.14.254.30 (65.14.254.30) 71.758 ms 71.662 ms 71.897 ms
15 adsl-****.mia.bellsouth.net () 98.516 ms 97.467 ms 97.593 ms
Karl
03-21-2007, 11:27 AM
Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

1 7 ms 7 ms 7 ms 172.30.192.1
2 7 ms 84 ms 107 ms mart-gw.chi-mart.il.cable.rcn.net [207.229.191.1
29]
3 7 ms 7 ms 7 ms ge1-0-2.core2.chsl.il.rcn.net [207.172.19.61]
4 6 ms 7 ms 7 ms ge5-0.border1.eqnx.il.rcn.net [207.172.19.15]
5 131 ms 134 ms 133 ms eqix.ge-0-3-0.ord1.nlayer.net [206.223.119.61]
6 128 ms 123 ms 123 ms 60.po1.ar1.ord1.us.nlayer.net [69.31.111.130]
7 * * * Request timed out.
8 116 ms 118 ms 113 ms 201.216-86-145.nozonenet.com [216.86.145.201]


That seems to have been temporary congestion between nLayer and RCN. We do not see that commonly, but we are currently working on establishing a more direct peering relationship with RCN which should be set in the next month or two.


Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.0.1
2 30 ms 7 ms 76 ms 10.65.112.1
3 12 ms 11 ms 38 ms d226-9-249.home.cgocable.net [24.226.9.249]
4 35 ms 24 ms 15 ms h64-187-46-221.gtcust.grouptelecom.net [64.187.4
6.221]
5 44 ms 71 ms 12 ms GE4-0.WANB-TOROON.IP.GROUPTELECOM.NET [216.18.63
.5]
6 52 ms 24 ms 113 ms 66.59.191.110
7 25 ms 52 ms 34 ms eqix.ge-0-3-0.ord1.nlayer.net [206.223.119.61]
8 25 ms 77 ms 82 ms 60.po1.ar1.ord1.us.nlayer.net [69.31.111.130]
9 * * * Request timed out.
10 30 ms 30 ms 40 ms 201.216-86-145.nozonenet.com [216.86.145.201]


Here the route looks good, there is no packet loss on the final hop, which is the only place it really matters, unless it shows throughout the whole route, etc. It actually looks very good considering the 2nd and 5th hops fluctuate as much as they do.


Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

1 <1 ms 4294967291 ms 4294967292 ms 192.168.1.1
2 20 ms 16 ms 16 ms 69.155.200.1
3 24 ms 19 ms 16 ms 10.254.254.5
4 21 ms 21 ms 21 ms 70.129.237.149
5 21 ms 49 ms 24 ms bb2-p7-3.rcsntx.sbcglobal.net [151.164.41.188]
6 31 ms 24 ms 59 ms ex1-p12-0.eqdltx.sbcglobal.net [151.164.40.29]
7 28 ms 46 ms 26 ms 151.164.251.150
8 23 ms 79 ms 36 ms dpr1-ge-6-0-0.dallasequinix.savvis.net [204.70.1
94.33]
9 91 ms 24 ms 21 ms dcr2-so-5-2-0.Dallas.savvis.net [204.70.194.66]

10 22 ms 69 ms 23 ms dcr2-so-6-0-0.dallas.savvis.net [204.70.192.50]

11 49 ms 104 ms 93 ms dcr1-so-5-0-0.chicago.savvis.net [204.70.192.45]

12 91 ms 51 ms 61 ms ber2-pos-1-0-0.Chicago.savvis.net [208.175.10.98
]
13 52 ms 46 ms 59 ms ber2-tenge-3-1.chicagoequinix.savvis.net [204.70
.196.26]
14 60 ms 50 ms 71 ms 208.174.225.202
15 * 47 ms * 208.174.225.202
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete.


There the issue seems that the return route had a major issue, though without a reverse trace I can't really say what the issue is. Everything looks perfectly fine now.


Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms dslrouter [192.168.1.1]
2 11 ms 10 ms 10 ms 10.1.38.1
3 11 ms 10 ms 10 ms so-0-1-1-0.CORE-RTR2.RES.verizon-gni.net [130.81
.9.21]
4 13 ms 11 ms 47 ms so-6-0-0-0.BB-RTR2.RES.verizon-gni.net [130.81.2
0.18]
5 11 ms 11 ms 11 ms so-7-0-0-0.ASH-PEER-RTR2.verizon-gni.net [130.81
.17.179]
6 11 ms 11 ms 11 ms 130.81.15.14
7 12 ms 12 ms 12 ms xe-4-0-0.cr1.iad1.us.nlayer.net [69.31.31.137]
8 19 ms 18 ms 18 ms so-3-2-0.cr1.nyc3.us.nlayer.net [69.22.142.101]

9 37 ms 36 ms 37 ms so-2-1-0.cr1.ord1.us.nlayer.net [69.22.142.106]

10 36 ms 37 ms 36 ms 60.po1.ar1.ord1.us.nlayer.net [69.31.111.130]
11 * * * Request timed out.
12 36 ms 37 ms 37 ms 201.216-86-145.nozonenet.com [216.86.145.201]


No loss at the end so I'm not seeing any issues here either. A ping would likely have been useful to seeif any of the packet loss translates through to the end, my guess is no.

Just a quick lesson in reading traceroutes.. If packet loss is shown somewhere in the route, other than the end, and that packet loss does not carry throughout the trace it is almost certainly just a router that has decided to drop such packets directed at it, etc. I know our routers have rate limits on these types of traffic, to prevent abuse, but these limits/dropped packets do not mean anything for any data actually passing through. To see if the packet loss passes through either ping the final destination, say 100+ pings, or do a lot more traceroutes. The more data you have the better.
Karl
03-21-2007, 11:33 AM
I also go through nlayer and pinging higher than usual.

I used to ping like 40 ms over a year ago, but I think my ISP did something to screw that up. But I was still getting like 70-80ms, now im at about 100ms (90ms on monday). Also Im pretty sure the number of hops used to be about 12.


That is all due to changes with merging peering relating to the combining of the SBC (owned by AT&T but still a separate network) and BellSouth networks, etc. We are working on a couple changes to improve our connectivity to both the SBC and BellSouth networks as those are both very important to our customer base.
silver_2000
03-23-2007, 10:57 PM
seems the nlayer connection blipped tonight ... I was sure it was server load but as it turns out load was fine about half way to steadfast nlayer was dropping packets
Kevin
03-25-2007, 12:58 AM
We have perceived that nLayer has been causing some problems for us lately, so we're definitely looking into it closely.
Justec
04-12-2007, 01:33 PM
That is all due to changes with merging peering relating to the combining of the SBC (owned by AT&T but still a separate network) and BellSouth networks, etc. We are working on a couple changes to improve our connectivity to both the SBC and BellSouth networks as those are both very important to our customer base.
Any word on this? Still pinging 100ms
Karl
04-13-2007, 11:23 AM
Any word on this? Still pinging 100ms

I'm hoping we'll have a more direct peering route with SBC set for May 1. These things take time. :-)
Justec
04-13-2007, 03:04 PM
Heh, I understand, glad to hear you've made some good progress!
Justec
05-08-2007, 04:18 PM
As promised!

Approximate round trip times in milli-seconds:
Minimum = 70ms, Maximum = 72ms, Average = 71ms

Thanks Karl
Henrik
05-08-2007, 07:41 PM
Nice to hear that your problems were sorted, Justin! :)