less then 1ms. I’m not a rocket scientist but it looks to me like latency does have an impact.

If you guys still dont get it, google “long fat networks”

…..around 3.5 to 4mbs download speeds on a 11mb DSL circuit compared to seeing exactly 11mb (+/- 300kb) when the latency is

Bill is talking about the impact caused to TCP by networks with high, fixed latency.

Here is another example besides Bills. I work for an ISP in Hawaii. We use an OC48 to get to the mainland. It has a fixed propagation delay of around 60ms RTT because the distance. On the other hand, I can set up a T1 line in my lab and get 5ms of latency off it.

It’s this fixed latency that can wreak havoc on a TCP session flow (like Bill explained) regardless of the size of the connection.

For example, a Windows PC that uses the default receiver buffers will not be able to take advantage of high speed connections in the 7mbs and up range if there is a lot of latency. The default RWIN value on a windows based PCs peaks out around 2.6mb/s on a path with 200ms RTT. It’s only through tweaking the registry that will allow a windows PC to take advantage of RFC1323 mechanisms. By doing some tweaking you can alleviate some of the performance issues caused by high latency connections to TCP.

On the other hand, if you have a MAC or Linux (supposedly Vista too) based PC you probably won’t have to do any tweaking as they generally have window scaling, SACK and a higher RWIN value enabled by default.

This subject is very near and dear to me as I am currently evaluating, in the lab, our 7 and 11mb DSL offering. One of the things I am testing is the results of website based speed tests to a local server setup with a speed test application. To this directly connected server I see

CDNs came about because content networks were interested in solving a problem on their own that ISPs and NSPs should be solving but are not for financial and political reasons. the manner in which they did this was to eliminate intermediate networks altogether and introduce faux localization. this is neither here nor there on topic.

a high latency network should experience no more issues than a low latency network. however, as more outstanding data will exist on a high latency network, the risk is bigger when something does become a problem.

simply saying “add more bandwidth’ is oversimplified as well. too many admins default to this when in reality you should first understand the cause of a problem instead of blaming the symptoms.

latency isn’t a problem unto itself. neither is bandwidth. more often than not, your assumptions on what your network is actually doing is the problem.

you didn’t provide any details of which ISP you’re using, but yes, a naive CALEA implementation could have everything routed the way you observe though not likely. most of that data is collected local to the node as gathering it on an aggregate interface downstream is harder to do due to the data rates involved.

it’s hard to speculate without seeing traceroutes to understand some of the topology involved. given what you said, it sounds like the majority of devices in their network have little more than static routes to the next router and have no real IGP awareness, or worse, have only one path through the network to other nodes.

it could be that their implementation has all remote nodes as circuits back to their ‘central’ office like a traditional backhauled dial network. in general seeing all traffic go through one node like that is indicative of a lean network with no other paths. this is fairly common in smaller isps as they can not afford the infrastructure as yet to allow for multiple exits from each pop to their core(s). in some cases this backhauling results in said traffic going via a ’scenic’ route. these cases can be financially and politically driven at times as well.

finding out why it is this way would require getting to know your ISP’s network engineers and noc. while they may not be able to share all the details, you could gain a better understanding of some of it. in some cases it may be simple oversight and misconfiguration, as they’re human and make mistakes too.

>”You are saying that latency causes network problems and that by improving latency you can improve your network. I assert that this is false. If you have latency problems, they are a symptom of network congestion. If your network is suffering from serious congestion, it probably needs more bandwidth.”

Wow. It is impressive how someone can miss the point so completely so many times. While network congestion will add to latency, latency is in and of itself a problem. In a network with zero congestion, latency will still be a problem. The problem is distance. More bandwidth cannot improve upon the speed of light. Sorry. This is the whole point of my article. Latency does cause issues unrelated to bandwidth or congestion. Those issues can be reduced with planning.

Thanks for commenting.

-Bill

Although you’re right in that most network protocols are bi-directional (eg. HTTP), congestion control only takes effect when the congestion window is reached. In a typical download over HTTP, the client making the request will not reach the congestion window size. The congestion control algorithms on the client are therefore irrelevant. It’s the server’s algorithms that matter, because it’s the one sending lots of data and hitting the congestion window ceiling.

> 2) I did not say that latency causes packet loss; I said there was a
> correlation between the two. I will drop more packets on my trans-atlantic
> MPLS circuit than on my point-to-point link between two locations in
> California, all other things being equal.

Correlation does not equal causation! As I explained, high latencies and lost packets are both symptoms of network congestion. What is the solution to network congestion? …. add more bandwidth!

> 3) Even if the window stays the same size, it still takes longer for a
> complete round trip transaction to occur over a highly latent connection.
> This is the whole reason RFC 1323 exists!

Actually, no. Read the introduction to RFC1323 that explains the reasons for its existence:

The introduction of fiber optics is resulting in ever-higher
transmission speeds, and the fastest paths are moving out of the
domain for which TCP was originally engineered.

> Obviously your contention is that high-latency networks should not have a
> problem because TCP will magically deal with the issue. The simple fact is
> that this is not the case. That is why companies design around this
> problem with CDNs and network accelerators.

No, this is not what I am saying. You are saying that latency causes network problems and that by improving latency you can improve your network. I assert that this is false. If you have latency problems, they are a symptom of network congestion. If your network is suffering from serious congestion, it probably needs more bandwidth.

I appreciate your comments and your interest. 3 things:

1) Most applications these days, but certainly not all, are bi-directional. Sometimes I’m the sender and sometimes I’m the receiver. That’s why I said the host on each side. Since the premise of this article is a WAN design, where sometimes clients are sending data and sometimes they are receiving it, I need to be aware of the limitations at both ends.

2) I did not say that latency causes packet loss; I said there was a correlation between the two. I will drop more packets on my trans-atlantic MPLS circuit than on my point-to-point link between two locations in California, all other things being equal.

3) Even if the window stays the same size, it still takes longer for a complete round trip transaction to occur over a highly latent connection. This is the whole reason RFC 1323 exists!

Obviously your contention is that high-latency networks should not have a problem because TCP will magically deal with the issue. The simple fact is that this is not the case. That is why companies design around this problem with CDNs and network accelerators. I hope you’ll check out and link to part 2 tomorrow.

Thanks,

-Bill