Latency/congestion Issues

[Erik]

As of late, I've been having issues with packet loss/drop on TCP/UDP traffic to my dedicated box as indicated below. These issues usually arise during NA's peek time between 6-11pm EST. Feedback is appreciated

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~

Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

1 7 ms 7 ms 7 ms 172.30.192.1
2 7 ms 84 ms 107 ms mart-gw.chi-mart.il.cable.rcn.net [207.229.191.1
29]
3 7 ms 7 ms 7 ms ge1-0-2.core2.chsl.il.rcn.net [207.172.19.61]
4 6 ms 7 ms 7 ms ge5-0.border1.eqnx.il.rcn.net [207.172.19.15]
5 131 ms 134 ms 133 ms eqix.ge-0-3-0.ord1.nlayer.net [206.223.119.61]
6 128 ms 123 ms 123 ms 60.po1.ar1.ord1.us.nlayer.net [69.31.111.130]
7 * * * Request timed out.
8 116 ms 118 ms 113 ms 201.216-86-145.nozonenet.com [216.86.145.201]

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~

Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.0.1
2 30 ms 7 ms 76 ms 10.65.112.1
3 12 ms 11 ms 38 ms d226-9-249.home.cgocable.net [24.226.9.249]
4 35 ms 24 ms 15 ms h64-187-46-221.gtcust.grouptelecom.net [64.187.4
6.221]
5 44 ms 71 ms 12 ms GE4-0.WANB-TOROON.IP.GROUPTELECOM.NET [216.18.63
.5]
6 52 ms 24 ms 113 ms 66.59.191.110
7 25 ms 52 ms 34 ms eqix.ge-0-3-0.ord1.nlayer.net [206.223.119.61]
8 25 ms 77 ms 82 ms 60.po1.ar1.ord1.us.nlayer.net [69.31.111.130]
9 * * * Request timed out.
10 30 ms 30 ms 40 ms 201.216-86-145.nozonenet.com [216.86.145.201]

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~

Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

1 <1 ms 4294967291 ms 4294967292 ms 192.168.1.1
2 20 ms 16 ms 16 ms 69.155.200.1
3 24 ms 19 ms 16 ms 10.254.254.5
4 21 ms 21 ms 21 ms 70.129.237.149
5 21 ms 49 ms 24 ms bb2-p7-3.rcsntx.sbcglobal.net [151.164.41.188]
6 31 ms 24 ms 59 ms ex1-p12-0.eqdltx.sbcglobal.net [151.164.40.29]
7 28 ms 46 ms 26 ms 151.164.251.150
8 23 ms 79 ms 36 ms dpr1-ge-6-0-0.dallasequinix.savvis.net [204.70.1
94.33]
9 91 ms 24 ms 21 ms dcr2-so-5-2-0.Dallas.savvis.net [204.70.194.66]

10 22 ms 69 ms 23 ms dcr2-so-6-0-0.dallas.savvis.net [204.70.192.50]

11 49 ms 104 ms 93 ms dcr1-so-5-0-0.chicago.savvis.net [204.70.192.45]

12 91 ms 51 ms 61 ms ber2-pos-1-0-0.Chicago.savvis.net [208.175.10.98
]
13 52 ms 46 ms 59 ms ber2-tenge-3-1.chicagoequinix.savvis.net [204.70
.196.26]
14 60 ms 50 ms 71 ms 208.174.225.202
15 * 47 ms * 208.174.225.202
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~

Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms dslrouter [192.168.1.1]
2 11 ms 10 ms 10 ms 10.1.38.1
3 11 ms 10 ms 10 ms so-0-1-1-0.CORE-RTR2.RES.verizon-gni.net [130.81
.9.21]
4 13 ms 11 ms 47 ms so-6-0-0-0.BB-RTR2.RES.verizon-gni.net [130.81.2
0.18]
5 11 ms 11 ms 11 ms so-7-0-0-0.ASH-PEER-RTR2.verizon-gni.net [130.81
.17.179]
6 11 ms 11 ms 11 ms 130.81.15.14
7 12 ms 12 ms 12 ms xe-4-0-0.cr1.iad1.us.nlayer.net [69.31.31.137]
8 19 ms 18 ms 18 ms so-3-2-0.cr1.nyc3.us.nlayer.net [69.22.142.101]

9 37 ms 36 ms 37 ms so-2-1-0.cr1.ord1.us.nlayer.net [69.22.142.106]

10 36 ms 37 ms 36 ms 60.po1.ar1.ord1.us.nlayer.net [69.31.111.130]
11 * * * Request timed out.
12 36 ms 37 ms 37 ms 201.216-86-145.nozonenet.com [216.86.145.201]

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[Justec]

I also go through nlayer and pinging higher than usual.

I used to ping like 40 ms over a year ago, but I think my ISP did something to screw that up. But I was still getting like 70-80ms, now im at about 100ms (90ms on monday). Also Im pretty sure the number of hops used to be about 12.

Miami -> SF

Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

  1    28 ms    27 ms    27 ms  65.14.252.30
  2    27 ms    27 ms    27 ms  65.14.252.30
  3    29 ms    27 ms    28 ms  65.14.254.29
  4    60 ms    59 ms    59 ms  ixc01mia-ge-0-3-0.bellsouth.net [205.152.145.181]
  5    57 ms    57 ms    57 ms  axr01mia-so-1-3-2.bellsouth.net [65.83.237.168]
  6    59 ms    59 ms    59 ms  axr00bct-so-0-0-0.bellsouth.net [65.83.236.10]
  7    59 ms     *       58 ms  AXR01BCT-1-0-0.bellsouth.net [65.83.236.55]
  8    46 ms    45 ms    45 ms  pxr00mia-2-0-0.bellsouth.net [65.83.236.18]
  9    45 ms    45 ms    45 ms  axr01asm-ge-5-1-0.bellsouth.net [65.83.236.187]
 10    47 ms    46 ms    45 ms  axr00aep-so-0-0-0.bellsouth.net [65.83.238.40]
 11    61 ms    59 ms    59 ms  pxr00ash-so-3-1-0.bellsouth.net [65.83.236.208]
 12    59 ms    57 ms    57 ms  g1-17.ar1.iad1.us.nlayer.net [69.31.30.21]
 13    59 ms    59 ms    59 ms  xe-4-0-0.cr1.iad1.us.nlayer.net [69.31.31.137]
 14    79 ms    79 ms    79 ms  so-3-2-0.cr1.nyc3.us.nlayer.net [69.22.142.101]
 15   100 ms    99 ms    99 ms  so-2-1-0.cr1.ord1.us.nlayer.net [69.22.142.106]
 16    99 ms    99 ms    99 ms  60.po1.ar1.ord1.us.nlayer.net [69.31.111.130]
 17     *        *        *     Request timed out.
 18    97 ms    97 ms    97 ms  201.216-86-145.nozonenet.com [216.86.145.201]

Trace complete.

SF -> Miami

traceroute to justec.us (67.35.87.15), 30 hops max, 46 byte packets
 1  241.216-86-149.nozonenet.com (216.86.149.241)  0.302 ms  0.268 ms  0.240 ms
 2  109.ae0.cr1.ord1.us.nlayer.net (69.31.111.57)  0.489 ms  0.402 ms *
 3  60.ae0.cr1.ord1.us.nlayer.net (69.31.111.129)  0.513 ms  0.411 ms  0.394 ms
 4  so-1-1-0.cr1.nyc3.us.nlayer.net (69.22.142.105)  20.398 ms  20.298 ms *
 5  69.31.94.46 (69.31.94.46)  20.435 ms  20.408 ms  20.347 ms
 6  xer01chi-pos-1-0.bellsouth.net (65.83.236.165)  66.626 ms  66.369 ms  66.805 ms
     MPLS Label=808 CoS=0 TTL=127 S=0
 7  pxr00chi-so-2-0-0.bellsouth.net (65.83.236.120)  40.987 ms  40.932 ms  41.029 ms
     MPLS Label=123216 CoS=0 TTL=127 S=0
 8  axr00asm-so-1-1-2.bellsouth.net (65.83.236.203)  70.483 ms  70.998 ms  70.557 ms
     MPLS Label=116064 CoS=0 TTL=127 S=0
 9  axr01bct-so-0-0-0.bellsouth.net (65.83.236.19)  70.571 ms  70.438 ms  71.422 ms
     MPLS Label=130336 CoS=0 TTL=127 S=0
10  AXR00BCT-1-0-0.bellsouth.net (65.83.236.54)  71.129 ms  72.372 ms  71.163 ms
     MPLS Label=176496 CoS=0 TTL=127 S=0
11  axr01mia-so-1-1-0.bellsouth.net (65.83.236.11)  70.546 ms  70.495 ms  70.446 ms
     MPLS Label=105136 CoS=0 TTL=127 S=0
12  ixc01mia-pos-5-0.bellsouth.net (65.83.237.15)  69.688 ms  69.755 ms  69.780 ms
13  her01mia-ge-1-1.bellsouth.net (205.152.145.182)  70.952 ms  70.882 ms  71.009 ms
14  65.14.254.30 (65.14.254.30)  71.758 ms  71.662 ms  71.897 ms
15  adsl-****.mia.bellsouth.net ()  98.516 ms  97.467 ms  97.593 ms

[Karl]

Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

1 7 ms 7 ms 7 ms 172.30.192.1
2 7 ms 84 ms 107 ms mart-gw.chi-mart.il.cable.rcn.net [207.229.191.1
29]
3 7 ms 7 ms 7 ms ge1-0-2.core2.chsl.il.rcn.net [207.172.19.61]
4 6 ms 7 ms 7 ms ge5-0.border1.eqnx.il.rcn.net [207.172.19.15]
5 131 ms 134 ms 133 ms eqix.ge-0-3-0.ord1.nlayer.net [206.223.119.61]
6 128 ms 123 ms 123 ms 60.po1.ar1.ord1.us.nlayer.net [69.31.111.130]
7 * * * Request timed out.
8 116 ms 118 ms 113 ms 201.216-86-145.nozonenet.com [216.86.145.201]

That seems to have been temporary congestion between nLayer and RCN. We do not see that commonly, but we are currently working on establishing a more direct peering relationship with RCN which should be set in the next month or two.

Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.0.1
2 30 ms 7 ms 76 ms 10.65.112.1
3 12 ms 11 ms 38 ms d226-9-249.home.cgocable.net [24.226.9.249]
4 35 ms 24 ms 15 ms h64-187-46-221.gtcust.grouptelecom.net [64.187.4
6.221]
5 44 ms 71 ms 12 ms GE4-0.WANB-TOROON.IP.GROUPTELECOM.NET [216.18.63
.5]
6 52 ms 24 ms 113 ms 66.59.191.110
7 25 ms 52 ms 34 ms eqix.ge-0-3-0.ord1.nlayer.net [206.223.119.61]
8 25 ms 77 ms 82 ms 60.po1.ar1.ord1.us.nlayer.net [69.31.111.130]
9 * * * Request timed out.
10 30 ms 30 ms 40 ms 201.216-86-145.nozonenet.com [216.86.145.201]

Here the route looks good, there is no packet loss on the final hop, which is the only place it really matters, unless it shows throughout the whole route, etc. It actually looks very good considering the 2nd and 5th hops fluctuate as much as they do.

Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

1 <1 ms 4294967291 ms 4294967292 ms 192.168.1.1
2 20 ms 16 ms 16 ms 69.155.200.1
3 24 ms 19 ms 16 ms 10.254.254.5
4 21 ms 21 ms 21 ms 70.129.237.149
5 21 ms 49 ms 24 ms bb2-p7-3.rcsntx.sbcglobal.net [151.164.41.188]
6 31 ms 24 ms 59 ms ex1-p12-0.eqdltx.sbcglobal.net [151.164.40.29]
7 28 ms 46 ms 26 ms 151.164.251.150
8 23 ms 79 ms 36 ms dpr1-ge-6-0-0.dallasequinix.savvis.net [204.70.1
94.33]
9 91 ms 24 ms 21 ms dcr2-so-5-2-0.Dallas.savvis.net [204.70.194.66]

10 22 ms 69 ms 23 ms dcr2-so-6-0-0.dallas.savvis.net [204.70.192.50]

11 49 ms 104 ms 93 ms dcr1-so-5-0-0.chicago.savvis.net [204.70.192.45]

12 91 ms 51 ms 61 ms ber2-pos-1-0-0.Chicago.savvis.net [208.175.10.98
]
13 52 ms 46 ms 59 ms ber2-tenge-3-1.chicagoequinix.savvis.net [204.70
.196.26]
14 60 ms 50 ms 71 ms 208.174.225.202
15 * 47 ms * 208.174.225.202
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete.

There the issue seems that the return route had a major issue, though without a reverse trace I can't really say what the issue is. Everything looks perfectly fine now.

Tracing route to 201.216-86-145.nozonenet.com [216.86.145.201]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms dslrouter [192.168.1.1]
2 11 ms 10 ms 10 ms 10.1.38.1
3 11 ms 10 ms 10 ms so-0-1-1-0.CORE-RTR2.RES.verizon-gni.net [130.81
.9.21]
4 13 ms 11 ms 47 ms so-6-0-0-0.BB-RTR2.RES.verizon-gni.net [130.81.2
0.18]
5 11 ms 11 ms 11 ms so-7-0-0-0.ASH-PEER-RTR2.verizon-gni.net [130.81
.17.179]
6 11 ms 11 ms 11 ms 130.81.15.14
7 12 ms 12 ms 12 ms xe-4-0-0.cr1.iad1.us.nlayer.net [69.31.31.137]
8 19 ms 18 ms 18 ms so-3-2-0.cr1.nyc3.us.nlayer.net [69.22.142.101]

9 37 ms 36 ms 37 ms so-2-1-0.cr1.ord1.us.nlayer.net [69.22.142.106]

10 36 ms 37 ms 36 ms 60.po1.ar1.ord1.us.nlayer.net [69.31.111.130]
11 * * * Request timed out.
12 36 ms 37 ms 37 ms 201.216-86-145.nozonenet.com [216.86.145.201]

No loss at the end so I'm not seeing any issues here either. A ping would likely have been useful to seeif any of the packet loss translates through to the end, my guess is no.

Just a quick lesson in reading traceroutes.. If packet loss is shown somewhere in the route, other than the end, and that packet loss does not carry throughout the trace it is almost certainly just a router that has decided to drop such packets directed at it, etc. I know our routers have rate limits on these types of traffic, to prevent abuse, but these limits/dropped packets do not mean anything for any data actually passing through. To see if the packet loss passes through either ping the final destination, say 100+ pings, or do a lot more traceroutes. The more data you have the better.

[Karl]

I also go through nlayer and pinging higher than usual.

I used to ping like 40 ms over a year ago, but I think my ISP did something to screw that up. But I was still getting like 70-80ms, now im at about 100ms (90ms on monday). Also Im pretty sure the number of hops used to be about 12.

That is all due to changes with merging peering relating to the combining of the SBC (owned by AT&T but still a separate network) and BellSouth networks, etc. We are working on a couple changes to improve our connectivity to both the SBC and BellSouth networks as those are both very important to our customer base.

[silver_2000]

seems the nlayer connection blipped tonight ... I was sure it was server load but as it turns out load was fine about half way to steadfast nlayer was dropping packets

[Kevin]

We have perceived that nLayer has been causing some problems for us lately, so we're definitely looking into it closely.

[Justec]

That is all due to changes with merging peering relating to the combining of the SBC (owned by AT&T but still a separate network) and BellSouth networks, etc. We are working on a couple changes to improve our connectivity to both the SBC and BellSouth networks as those are both very important to our customer base.

Any word on this? Still pinging 100ms

[Karl]

Any word on this? Still pinging 100ms

I'm hoping we'll have a more direct peering route with SBC set for May 1. These things take time. :-)

[Justec]

Heh, I understand, glad to hear you've made some good progress!

[Justec]

As promised!

Approximate round trip times in milli-seconds:
    Minimum = 70ms, Maximum = 72ms, Average = 71ms

Thanks Karl