June 5, 2007

It’s Still the Latency, Stupid…pt.2

Buy this book!In part 1 of this series, I established the problem latency can cause in high speed networks. What one reader correctly referred to as “big long pipes.” To summarize, in large bandwidth networks that span long distances, network latency becomes the bottleneck that retards performance. The reason for this the impact of network delays on TCP windowing. In part, 2 I will discuss what to do about it.

Dealing with latency can be tricky business. The methods used to mitigate the impact of distance depends on many factors including the services being accessed, the protocols being used, and the amount of money you want to spend. What works for a home user does not work for a multi-national corporation. In general, there are 4 approaches one can take to deal with latency:

  1. Tweak the host TCP settings
  2. Change the protocol
  3. Move the service closer to the user
  4. Use a network accelerator

The first and least effective method is to tweak the TCP settings on your hosts. I say least effective for several reasons: It is hard to determine the correct TCP window size; not all operating systems support the RFC 1323 extensions; you may not have control of all the hosts; available bandwidth may change due to network congestion. Most importantly, some time sensitive applications such as VOIP will still exhibit problems in high latent networks, even if you tweak TCP. Still, if you are a home user on a big long pipe, this is the only option for you. Changing TCP is OS specific. Slaptijack.com has an excellent series on TCP tuning operating systems. Below are links to his specific guides as well as other sources:

If you are going to tweak your TCP settings you need to figure out what values to use. PJ turned me on to Iperf. Iperf is a bandwidth measuring tool that will help you tune your settings by measuring the bandwidth and RTT. The Iperf site also has several good links to other tools and resources for tuning TCP settings. Check it out.

One other host tweak that works strictly for browsing is to make sure your web browser has support for HTTP 1.1 enabled. HTTP 1.1 signals the web server to use gzip compression for file transfers. If you run a web server, you can enable gzip compression to make life a little easier on your users. The downside to gzip is it adds CPU overhead to your server.

For corporate networks, better options exist. Rather than tune individual hosts, networks and applications can be designed with latency in mind. One effective method is to change the protocol. Latency is a problem because TCP waits for an acknowledgement. UDP is considered a “less reliable” protocol because it does not wait for acks. For many applications UDP will work just as well as TCP, and will work better over long connections. UDP works best for temporal data, such as streaming audio or video, or multiplayer games, but can also be used for any application where the only data you care about is the current sample, such as stock prices or weather data. If you control the code, and can deal with lost or mis-ordered packets, UDP may be the way to go. For a good discussion of the differences in coding sockets in UDP instead of TCP, check out: http://michael.toren.net/mirrors/sock-faq/.

Another viable option is to move the service closer to the user. This is the approach major web sites take. Global companies such as Google and Yahoo create multiple points-of-presence (POPs) that are geographically dispersed. Users are then connected to the nearest POP through the use of global server load balancers (GSLBs). GSLBs generally work using DNS to direct the user to the closest server. The two big players in the GSLB space are Cisco GSS and F5 BigIP. I’ve used both in the past and prefer the Cisco GSS if you have a Cisco network. Some web sites use content delivery networks such as Akamai or RapidEdge to move the service closer to the user by creating a distributed content cache. Another approach is to move your network logically closer to the user. For instance, if you have a large number of users accessing your website from a specific ISP, consider getting a connection to their backbone. This will place you several hops closer to the user by keeping all your traffic “on net.”

On corporate networks, placing services closer to the user usually involved placing servers in their offices. Three services that most IT shops support that benefit from localization are authentication, DNS, and file services. Microsoft has three technologies that simplify the process of managing multiple distributed servers providing localized services. First, Microsoft Active Directory Services supports multi-master replication of all changes to the directory. This means that changes made on one domain controller, such as adding a new user, are replicated to all other domain controllers in the domain. Second, Microsoft Active Directory-integrated DNS also supports multi-master replication. This is much improved over the old master-slave DNS schemes. Third, Windows 2003R2 now supports Distributed File Services Replication (DFSR). DFSR provides for multi-master replication of file shares. Both Active Directory and DFSR are location aware, meaning users will always attempt to access the local resource first.

If you cannot move the services, adding network accelerators to your network design is a good alternative. Typically, network accelerators are placed in between your WAN routers and your LAN switches. Accelerators use four tricks to deal with latency:

  1. Local TCP acknowledgment. The accelerator sends an ack back to the sending host immediately. This ensures that the sender keeps putting packets on the wire, instead waiting for the ack from the actual recipient.
  2. UDP Conversion. The accelerators change the TCP stream to UDP to cross the WAN. When the packet reaches the accelerator on the far end, it is switched back to TCP. You can think of this a tunneling TCP inside of UDP, although unlike a VPN the UDP tunnel does not add any overhead to the stream.
  3. Caching. The accelerators notice data patterns and cache repeating information. When a sender transmits data that is already in the cache, the accelerators only push the cache ID across the WAN. An example of this would be several users accessing the same file from a CIFS share across your WAN. The accelerators would cache the file after the first user retrieves it, and use a token to transfer the subsequent requests.
  4. Compression. In addition to caching, network accelerators are able to compress some of the data being transmitted. The accelerator on the other end of the WAN decompresses the data before sending it to its destination. Compressed data can be sent in fewer packets, thus reducing the apparent time to send.

There are several good vendors of network accelerators. For those of you who care what Gartner thinks, their magic quadrant for this sector can be found here. My personal favorite in this group is the Juniper product line. They work well and scale nicely. Other vendors include:

When designing a global WAN (which was the original premise of this article), using a combination of localized services and network accelerators will drastically improve the performance of your network. As stated in part 1, you need to start your design by profiling the services using your network. Some services can be easily localized, such as authentication or file services. Other services, such as an ERP system or other large centralized database cannot be practically localized. For these applications, network accelerators are the way to go. As a general rule of thumb, if your WAN is expected to average more than 150ms latency end-to-end, you should consider latency mitigation strategies. If you have time sensitive applications such as VOIP or interactive terminal sessions (green screens), your threshold for mitigation is probably 100ms. If you are a home user on a fat pipe with a lot of latency, the best you can do is tweak your OS and wait for your ISP and the web sites or game servers you access to get better.

Other useful links when attempting TCP performance tuning:

Digg!

Thanks for stopping by.
If you found this article useful, please leave a tip.

18 Comments »

  1. fragglet said,

    June 5, 2007 @ 7:57 pm

    I’ve written a response to this as well:

    http://fragglet.livejournal.com/12153.html

    EDITOR’S NOTE: Fragglet’s comments have been copied from his blog to here to keep the discussion threaded on this site.

    Bill Dougherty has posted Part 2 of this “It’s the latency, stupid” article. Sadly, this is filled with as many factual errors as the previous one.

    Where do I start? First of all, HTTP: “HTTP 1.1 signals the web server to use gzip compression for file transfers”. This is pure and simply wrong. Go and read the HTTP/1.1 specification. Although gzip is mentioned, there’s no requirement that a HTTP/1.1 server should use gzip compression. I’d say that no browser has shipped for at least five years that uses HTTP/1.0, so this is a totally irrelevant suggestion to make. Even then, switching to HTTP/1.1 will not magically add gzip compression: it’s up to the web server to optionally send you compressed data instead of the normal uncompressed data. 99+% will not do this.

    Using HTTP/1.1 CAN provide an advantage, but for different reasons that are entirely unrelated to compression. The major difference between HTTP/1.0 and 1.1 is that HTTP/1.1 can reuse an existing connection to retrieve more files. HTTP/1.0 immediately closes the connection when a download has completed. This has advantages because of the way the congestion control algorithms work: they start off with a small TCP window size that is increased in order to determine the available bandwidth of the channel. With HTTP/1.0, this process is restarted when downloading each file. HTTP/1.1 allows you to reuse your existing connection that has already settled to a reasonable TCP window size. This is important for modern websites that have lots of images and other embedded content. As I mentioned before though, this is utterly irrelevant because all modern browsers already use HTTP/1.1 by default.

    Then Bill comes up with this gem: “One effective method is to change the protocol. Latency is a problem because TCP waits for an acknowledgement”. This is also wrong. He seems to be under the mistaken impression that TCP is a stop and wait protocol: that each packet is sent, an acknowledgement waited for, and then the next packet sent. What actually happens is that TCP sends a bunch of packets across the channel, and as the acknowledgement is received for each of packet, the next packet is sent. To use the trucks analogy again, imagine twenty trucks, equally spaced, driving in a circle between two depots, carrying goods from one depot to the other. Latency is not a problem, just like distance between the depots is not a problem: provided that you have enough trucks, the transfer rate is maintained. The TCP congestion control algorithms automatically determine “how many trucks to use”.

    TCP will restrict the rate at which you can send data. Suppose, for example, you’re writing a sockets program and sending a file across a TCP connection: you cannot send the entire file at once. After you have written a certain amount of data into the pipe, you cannot write any more until the receiving end has read the data. This is a good thing! What is happening here is called flow control. You physically can’t send data faster than the bandwidth of the channel you’re using can support. Suppose that you’re using 10KB/sec channel: you can’t send 50KB/sec of data across that channel. All that TCP is doing is limiting you to sending data at the physical limit of the channel.

    “If you control the code, and can deal with lost or mis-ordered packets, UDP may be the way to go”. While this is true, it’s misleading and potentially really bad advice, certainly to any programmers writing networked applications. If your application mainly involves transfer of files, the best thing to do is stick with TCP. The reason is that TCP already takes care of these problems: they’ve been thoroughly researched and there are many tweaks and optimisations that have been applied to the protocol over the years. One important feature is the congestion control algorithms, that automatically determine the available bandwidth. If you don’t use these kind of algorithms, you can end up the kind of collapse that Jacobson describes in his original paper on network congestion. If you use UDP, you’re forced to reinvent this and every other feature of TCP from scratch. As a general rule of thumb, it’s best to stick with TCP unless there is some specific need to use UDP.

    Finally, I’d like to examine his list of “tricks that network accelerators use”:

    “1. Local TCP acknowledgment. The accelerator sends an ack back to the sending host immediately. This ensures that the sender keeps putting packets on the wire, instead waiting for the ack from the actual recipient”. This is nonsense. TCP keeps putting packets onto the wire in normal operation. It doesn’t stop and wait for an acknowledgement. TCP acknowledgements should already be being transmitted correctly If you’re interfering with the normal transmission of acknowledgements, all you’re doing is breaking the fundamental nature of how the protocol and the sliding window algorithm work.

    “2. UDP Conversion. The accelerators change the TCP stream to UDP to cross the WAN. When the packet reaches the accelerator on the far end, it is switched back to TCP. You can think of this a tunneling TCP inside of UDP, although unlike a VPN the UDP tunnel does not add any overhead to the stream.” I fail to see what possible advantage this could bring.

    “3. Caching. The accelerators notice data patterns and cache repeating information. When a sender transmits data that is already in the cache, the accelerators only push the cache ID across the WAN. An example of this would be several users accessing the same file from a CIFS share across your WAN. The accelerators would cache the file after the first user retrieves it, and use a token to transfer the subsequent requests.” This is useful in the very specific case of CIFS, because SMB has known performance issues when running over high latency connections – it was designed for use on LANs, and the protocol suffers because of some assumptions that were made in its design. This doesn’t apply, however, to the majority of other network protocols.

    “4. Compression. In addition to caching, network accelerators are able to compress some of the data being transmitted. The accelerator on the other end of the WAN decompresses the data before sending it to its destination. Compressed data can be sent in fewer packets, thus reducing the apparent time to send.” Amusingly, what this actually does is decrease the bandwidth used, and has nothing to do with latency.

  2. bill said,

    June 5, 2007 @ 11:08 pm

    Simon (Fragglet),

    Thanks for your response. I took the liberty of copying your comments here to keep the discussion in one place. A couple of things:

    1) I did not say the only thing HTTP 1.1 does is trigger gzip. I have read the HTTP 1.1 spec. I think you will find here: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3 that section 14.3 of the specification discusses “accept-encoding: compress, gzip.” This allows the client to request gzip compression if it is enabled on the server. The server must also be configured to send gzip compressed in order for this to work. If you are a home user, you can enable or disable this in your browser. If you enable it, servers that are configured to compress with gzip will do so. Compression is a legitimate topic when discussing network performance, and it will help when dealing with a variety of network situations including high latency. It does however have a cost in terms of CPU overhead. http://www.seoconsultants.com/articles/1000/http-compression.asp

    2) Most modern browsers support HTTP 1.1. In Internet Explorer (the most popular browser), HTTP 1.1 is enabled by default but it is still configurable under Tools>Options>Advanced. http://schroepl.net/projekte/mod_gzip/browser.htm

    3) As discussed previously, on large bandwidth highly-latent connections where the tcp window is set too small, TCP will in fact wait for an ack before sending more data. This is true anytime the TCP window is smaller than the bandwidth delay product (BDP). This is the reason all of the operating systems linked above give you the option to change the TCP settings. I provided 8 links at the end of my article that disagree with your theory, and provide tips for performance tuning TCP.

    4) I did not suggest using UDP for file transfer applications. “For many applications UDP will work just as well as TCP, and will work better over long connections. UDP works best for temporal data, such as streaming audio or video, or multiplayer games, but can also be used for any application where the only data you care about is the current sample, such as stock prices or weather data.” Clearly in a file transfer we would care about more than the current sample. UDP works great for streaming applications like video and voice. An example of where the user could choose between TCP and UDP is if they were setting up a VOIP application using SIP. SIP will support TCP or UDP, but may perform better over highly latent connections using UDP.

    5) My description of how network accelerators work is accurate. You may think it is dumb, but Juniper and RiverBed have built profitable business models in spite of your opinion:

    General impact of network accelerators on long fat pipes: http://www.riverbed.com/technology/
    “For high bandwidth WAN links, RiOS 4.0 also provides new mechanisms like MX-TCP to “fill the pipe” high latency links, even in the face of high packet loss (e.g. lossy satellite links).”

    Local TCP Ack & TCP to UDP conversion: https://www.juniper.net. “Active Flow Pipelining – Extends the TCP performance improvements by terminating the TCP connections locally and using a more efficient transport protocol between WX and WXC platforms. This feature significantly benefits application performance on high-bandwidth or high-latency connections.”

    Caching: http://www.riverbed.com/technology/data_streamlining/index.php “Data Streamlining works by reducing the transmission of redundant bytes and by optionally prioritizing data based on bandwidth and latency requirements. Data Streamlining works across the key applications that enterprises care about the most, like Windows and UNIX file sharing (including MS Office), Email (including MS Exchange and Lotus Notes), CAD/PDM applications, ERP, databases, and performing data reduction on all applications that rely on TCP.”

    Compression: https://www.juniper.net
    “Molecular Sequence Reduction (MSR) is the flagship compression algorithm of the WX and WXC platforms. The patented technology, which has enabled enterprises to realize as great as a 10-fold increase in WAN capacity, has its roots in DNA pattern matching. MSR compression recognizes repeated data patterns and replaces them with labels, dramatically reducing WAN transmissions. MSR technology operates in memory, and its dictionary can store hundreds of megabytes of patterns. The MSR reduction capabilities benefit a broad cross-section of application types. It effectively reduces both short, chatty applications such as Citrix and HTTP as well as larger data patterns, such as Word files. Because MSR is dictionary based, it is able to eliminate patterns even when they are separated by large amounts of other data.”

    I really do appreciate you taking the time to both read and comment on my article. I would hope in the future you would take the time to think about what I am actually saying and follow the links I provide to review the source material. You might actually learn something, and you just might improve the performance of those Doom games you love so much. ;-)

    Cheers!

    -Bill

  3. Mark said,

    June 6, 2007 @ 4:26 am

    Nice follow up Bill.

    I had assumed you would just talk on the PC to PC windowing issues only. I was glad to see reference to network based enhancements instead of the usual “run DR TCP” answers.

    Now could you right up an article on how to explain to a VP of marketing why end users probably what actually see 11mbs on their 11mbs DSL sevice while running speed test to the mainland?

    Fragglet… Wow I really dont know what to say. You have some serious holes in your understanding of networking and TCP.

    You sound like an application guy that doenst really understand networking. don’t feel bad though, Im a networking guy that really doesnt understand applications :p

    Mahalo,
    Mark

  4. bill said,

    June 6, 2007 @ 9:13 am

    Mark,

    If you buy the plane ticket, I’ll be happy to come explain it to your VP…right after I get off the beach. ;)

    Your’s is an interesting problem, because as an ISP there is little you can do other than efficiently route packets. In theory, you could use network accelerators to create an accelerated backbone between your sites and major peering points around the globe. It would be cool but probably cost prohibitive.

    Thanks for the comments.

    -Bill

  5. Mark said,

    June 8, 2007 @ 6:21 am

    Haha, for some reason it’s always easy to get vendors/consultants interested in visiting us…. I wonder why.

    Sadly, I haven’t seen/heard of any wan accelerators that would be ideal for a service provider market. I really thought there would be something out there that the satellite providers use but I couldn’t really find anything worth while.

    I think we must be content with akamai for some local off loading and then work on educating our customers so they understand the issues being

  6. Mark said,

    June 8, 2007 @ 6:23 am

    …. +60ms away from anywhere.

    I’m sure you saw the PCMAG all state internet speed review. When I heard about it I immediately said that Hawaii and Alaska would be the worst. Someone asked me how I would know that… duh, it’s the latency stupid!

    ((your webpage really doesnt like carrot signs))

  7. Josh Betz » It’s still Latency pt.2 said,

    September 22, 2007 @ 6:36 pm

    […] http://www.edgeblog.net/2007/its-still-the-latency-stupid-pt2/. Back in June I put up the link for the first part of this article… here is part 2. I know it’s a little late, but I guess better late than never. Anyway, it’s pretty good stuff. Bookmark to: Trackback This Post […]

  8. bill said,

    October 29, 2007 @ 9:32 pm

    I just found this great write-up of the topic, including an analysis of Fraggle’s and my disagreement. If you liked this post, please take the time to read this:

    http://www.webperformancematters.com

    Thanks,

    -Bill

  9. Murda said,

    December 2, 2007 @ 3:55 pm

    Great articles Bill-we’re trying to explain to our HQ on the other side of the world why we are consistently having problems printing and downloading/uploading documents via Citrix. Why do we have problems? Because we have a 36000km round trip and we often get 250ms plus times which causes havoc with uploads and printing. Viewing is fine because the apps seem to be optimized for that kind of thing. I like your analogy.

    I was intrigued by Fragglet’s ideas too, the trucks analogy kind of falls over if you have the same number of trucks travelling a bigger distance-so the transfer rate is affected by the distance. And if you have more trucks it still takes longer to get from A to B and the time taken is what affects the transfer rate adversely. the trucks cannot travel any faster. If they could, it would be great.

  10. Stewart at Expand said,

    June 20, 2008 @ 12:08 pm

    Murda and others,

    Not many of the WAN Opt appliances have much help for Citrix/ICA or VDI/RDP traffic. While Expand cannot change the speed of light, we can optimize the Citrix and RDP traffic including the ability to aggregate packets, to mitigate to a certain extent the effect of the WAN on that traffic. We really offer significant benefits when it comes to dealing with Citrix congestion/remote printing issues/graphical applications as well as adding more users or less bandwidth. See the link above to start and call Expand if you want to discuss your use case. http://www.expand.com.

    Regards,

    Stewart

  11. bill said,

    October 26, 2008 @ 9:16 pm

    I found information on the Tsunami UDP project here. http://www.csm.ornl.gov/~dunigan/netperf/udp/UDP_Tsunami.html This is a project trying to overcome the problem of latency in high-speed long-distance networks for file transfers by using UDP instead of TCP for the transfer.

  12. Stuart Warren said,

    February 26, 2009 @ 9:59 am

    Hello Bill,

    I see on your article that you prefer Juniper. If you have a situation where you have horrid latency (aka Satellite) where RTT is up to a second, or 550ms on the best case, you should look at the Expand Networks optimizer. We did a comparison with 5 vendors and the obvious choices were not the winners. Expand was the only one that passed all the tests with flying colors.

    One thing your readers need to know is that some optimizers require you to turn off SMB Signing or their performance suffers. Expand supports SMB Signing and becomes part of the network and excels when it hits high latency links.

    Stu

  13. Latency Vs. Bandwidth - Part II - Super-Networking Blog said,

    March 31, 2009 @ 7:34 pm

    […] differences between the problem of Latency and Bandwidth on the Internet. The same blogger had a follow-up post that explained a few ways for you to help improve high […]

  14. It’s Still the Latency, Stupid…pt.1 | edgeblog said,

    May 20, 2011 @ 9:27 am

    […] the applications have been profiled, steps can be taken to mitigate the impact of network delay. In part 2 of this article, we will discuss methods of designing for latency mitigation.  Permalink Thanks for […]

  15. Mike said,

    July 3, 2012 @ 12:52 pm

    Any suggestions on Accelerators that can solve the latency problem with live video?
    We use EXPAND which solves the problem mainly by Local TCP acknowledgment described above. We did not get good results with Riverbed.
    EXPAND no longer exists and we would like to find other vendors that use a similar protocol.
    Any suggestions would be appreciated.

  16. bill said,

    July 6, 2012 @ 3:28 pm

    Mike, I would have suggested Riverbed since they bought Expand. Have you looked at Blue Coat?

  17. Mike said,

    July 9, 2012 @ 2:52 am

    Haven’t look at Blue coat yet.
    AM looking at CISCO , EXINDA, Silver Peak. not sure which ones have a similar protocol to EXPAND or have basic protocols that solve the latency problem with tcp latency.

  18. Mike said,

    July 9, 2012 @ 2:57 am

    Not yet.
    I am looking into CISCO, Exinda and Silver Peak. not sure which ones have basic protocol similar to EXPAND. I always thought all of them would solve out basic latency problem

RSS feed for comments on this post · TrackBack URI

Leave a Comment