Bad throughput on some ISP.

[misterd]

I don't want to create a trouble ticket on this issue since I don't believe this is SF's problem. But I'd rather share my story here just in case anyone is experiencing the same problem, or have suggestions.

I have been experiencing bad throughput on random ISPs. (According to complaints from customers) I can't seem to figure out what exactly is the cause of the problem. Traceroute from/to the server looks just fine.

Since yesterday, this problem shows up on my home connection, and yet again traceroute looks perfectly normal to me.

From my home connection, I tested the speed with servers hosted at SF, EV1 (ThePlanet), and FDC. Here's the result :

1 : SF
Inbound : McLeod -> nLayer -> SF
Outbound : SF -> xeex -> McLeod
Throughput : 10-20KB/s average (on a 100KB/s connection)

2 : EV1
Inbound : McLeod -> TimeWarner -> EV1
Outbound : EV1 -> TimeWarner -> McLeod
Throughput : maxed out my connection

3 : FDC
Inbound : McLeod -> PCCW -> FDC
Outbound : FDC -> WVfiber -> McLeod
Throughput : maxed out my connection /w random 1-2s drops (usual for FDC)

Throughput between these 3 servers are just fine. I can push at least 10mbps between either of them.

It seems like my ISP (McLeod) has either overloaded their nLayer or xeex link. That wouldn't have worried me much since the connection in my building is probably oversold anyways. But this kind of issue is showing up randomly on customers using RCN, Speakeasy & some random local ISPs. This worries me quite a bit since I don't know where the problem is.

I have double check with all of them that their connection is not overloaded by P2P apps, or other heavy user sharing the same connection. I was able to get traceroute from some of them, and they all looked fine.

Feedback & suggestion is much appreciated!

[ManagerJosh]

Could you post up results of a traceroute to steadfast.net and to your server on steadfast?

[Ray]

Additionally, is the throughput you're mentioning uploading to the server or downloading from the server?

[misterd]

To Steadfast

$ traceroute 208.100.15.*
traceroute to 208.100.15.* (208.100.15.*), 64 hops max, 40 byte packets
 1  unknown (192.168.1.1)  0.923 ms  0.559 ms  0.478 ms
 2  209.253.118.1 (209.253.118.1)  1.970 ms  2.135 ms  1.410 ms
 3  209.253.116.21 (209.253.116.21)  6.208 ms  10.984 ms  4.246 ms
 4  209-254-197-61.ip.mcleodusa.net (209.254.197.61)  8.324 ms  4.261 ms  4.035 ms
 5  chcgilwuh53jc01-ge1-3-0-0.mcleodusa.net (64.198.101.105)  9.073 ms  15.414 ms  18.404 ms
 6  chcgildthbujc01-so0-3-0-0.mcleodusa.net (64.198.100.46)  4.690 ms  53.777 ms  110.773 ms
 7  eqix.ge-0-3-0.ord1.nlayer.net (206.223.119.61)  5.090 ms  4.466 ms  4.585 ms
 8  60.po1.ar1.ord1.us.nlayer.net (69.31.111.130)  4.897 ms  4.955 ms  4.874 ms
 9  * * *
10  61.216-86-149.nozonenet.com (216.86.149.61)  21.721 ms  30.035 ms  8.804 ms
11  208.100.15.* (208.100.15.*)  5.913 ms  4.596 ms  6.101 ms

From Steadfast :

$ traceroute 209.253.118.*
traceroute to 209.253.118.* (209.253.118.*) from 208.100.15.*, 30 hops max, 46 byte packets
 1  208.100.15.* (208.100.15.*)  0.000 ms  1.000 ms  0.000 ms
 2  unknown.xeex.net (216.151.129.45)  0.000 ms  0.000 ms  0.000 ms
 3  CHCGILDTHBUJC01-GE0-0-0-0.mcleodusa.net (206.223.119.5)  0.000 ms  1.000 ms  0.000 ms
 4  CHCGILWUH53JC01-SO1-2-0-0.mcleodusa.net (64.198.100.45)  1.000 ms  0.000 ms  1.000 ms
 5  CHCGILWUH53JP01-GE0-0-0-0.mcleodusa.net (64.198.101.106)  1.000 ms  1.000 ms  0.000 ms
 6  * 209-254-197-62.ip.mcleodusa.net (209.254.197.62)  3.999 ms  1.999 ms
 7  209.253.116.22 (209.253.116.22)  4.000 ms  5.999 ms  16.996 ms
 8  * * *

The throughput I mentioned is downloading from the server (i.e. SF-->EV1, SF-->FDC, SF-->McLeod)

Unfortunately, I did not save traceroute when there was the problem. Traceroutes above was run at the time of writing this post. At this moment, downloading data from Steadfast seems to be normal now (maxed out my connection). But it's the same route as I tested earlier, and I would say response time is pretty much the same.

I know the internet is not perfect, but it just kind of annoy me when I can't find the source of the problem.

[Justec]

I've had a similar problem for months now. Im just glad im the only one complaining. I used to ping like 40-50ms to my server from Miami and then it went to 70ms and now 80ms (currently its 90ms but that happens some days hopefully it returns to 80ms soon).

I have bellsouth for an ISP and the problem seems to be more on the return route (download). I have some tickets open about this and will try to get more info to add to this thread. Maybe we can get to the bottom of this.

[techlici]

I have also experienced the same problem here in Ontario, Canada. Our ISP (Rogers telcom) have had major problem routing to steadfast. Some customers arn't even able to do a simple ping or load a basic page. WHen I first signed up it was fine, then about a month or so after it became terribleeee, I sent numerous emails to my ISP providing all kinds of technical data to aid in troubleshooting, however I never got a response for any of there network engineers. Instead I would get emails from there general support stayin they are trying there best. This made me veryyy frustrated as my site is targetted to people in Toronto, and well if my site is not available then somethign must be done. I then set off on a hunt to find another host that my ISP didn't have troubles routing to. Well several months later still looking for another host, but magically they fixed it, my site is now available to most people within toronto. Looking at the traceroute data it seems to be changing every now and then.

So ya, just sharing my experience, if it helps anyone, because i was VERY frustrated when I noticed it.

[Karl]

Rogers seems to have very limited outgoing routing and many congested peering points. It is generally quite difficult to get around the issues fully, but we had made some changes with our incoming network configuration a couple weeks ago, if I remember correctly, which should help.

[Justec]

My pings have been much improved since the beginning of May, but lately I get random packet loss for 30 mins here and 30 mins there then its normal for a while. Just wonder if anyone else on these forums has ATT (Bellsouth) as an ISP. I get about 10% packet loss and it is noticeable when browsing websites on my server.

Had a friend in Toronto do a ping and they are only getting <1% packet loss Thanks ATT :(