One concept that continues to elude many IT managers is the impact of latency on network design. 11 years ago, Stuart Cheshire wrote a detailed analysis on the difference between bandwidth and latency ISP links. Over a decade later, his writings are still relevant. Latency, not bandwidth, is often the key to network speed (or lack thereof).

I was reminded of Cheshire’s article and the underlying principles recently when working on an international WAN design. What Cheshire noted was that light signals pass through fibre optics at roughly 66% of the speed of light, or 200*10^6 m/s. Regardless of the equipment or protocols you use, your data cannot exceed that theoretical limit. This limit equals the delay between when a packet is sent, and when it is received, aka latency.

In the US, we tend to focus on bandwidth and carrier technology when ordering circuits, completely ignoring latency. For instance, when choosing between cable and DSL for your house do you ever ask the carrier for its latency SLA? Maybe you should. Using a cable connection a ping to www.google.com in Mountain View, CA from my house (137 KM) yields an average ping time (aka round-trip time or RTT) of 73ms. The theoretical latency for this distance (round trip) is 1.37ms meaning my cable connection is roughly 50 times worse than the theoretical limit. No surprise that Comcast focuses on bandwidth and not latency in its marketing.

Cable and DSL circuits in the US are generally not business class and do not carry any service level agreement (SLA) on latency or availability. Businesses who use these circuits for business critical services do so at their peril. Business circuits such as Frame Relay and MPLS do generally include latency SLAs, but understanding the difference between the SLA and your actual experience can have a massive impact the performance of your network. For instance, let’s say a carrier advertises a 55ms round trip SLA in the US. This SLA equals the maximum latency between any two points of presence (POP) on their network.

The coast-to-coast distance in the US is roughly 5,000KM for a theoretical latency of 50ms, so a 55ms RTT SLA is pretty good. But that doesn’t mean packets on your network will only take 55ms to cross the country. When designing your WAN you must also account for the latency added by your network equipment and your servers, and the distance between the carrier’s POP and your offices. As a result, a well designed US WAN will still experience 75-80ms ping times. A poorly designed WAN can experience much worse times.

Now consider creating an international WAN. In this case, you typically will receive multiple SLAs from the carrier for different parts of the network. For instance, when designing an MPLS connection between California and the UK, the SLAs would be approximately 55ms within the US plus 95ms to cross the Atlantic Ocean plus 21ms to connect within the UK. Add the latency of your network and you get ping times of 175ms to 200ms.

At this point you are probably asking yourself “so what?” Two tenths of a second is no big deal. The answer is the impact of latency on TCP windowing. Transmission Control Protocol (TCP) has a flow control mechanism that senses latency and bandwidth between two hosts and determines the rates which data will be transferred. The TCP window is the amount a unacknowledged data a sender can transmit before waiting for a TCP ACK. As the latency increases, the TCP window shrinks, meaning the sender sends less data before waiting for an ACK. This helps reduce the amount of data that will need to be retransmitted in case a packet gets lost. Smaller windows equals more packets, and more packets equals more data because each packet carries the overhead of a 40 byte TCP/IP header regardless of if the payload is 1 byte or 1500 bytes.

The result is what I call the “Sandbag Problem.” Let’s say the two of us are trying to fill sandbags. My job is to scoop sand into a container and hand the full container to you (data). Your job is to empty the container into a sandbag and hand the empty container (ACK) back to me. Occasionally you drop the container so I have to fill it again (Retransmit). If we were standing next to each other, the time it takes for me to hand the container to you, have you empty it, and hand it back to me (latency) would be very small. Now imagine there is a 6′ wall between us, and I need to hand the container over to you.

The wall changes several aspects of our filling operation. First, the size of the container must be smaller because I cannot lift the same weight over my head that I can lift at waist level. Second, the time to complete one cycle would increase because it takes longer to lift the container 6′ than it does 3′. Third, you would drop more containers so retransmissions would increase. As the wall gets taller, the problem gets worse. If the wall were 10′ tall, we would be throwing containers instead of lifting them, so they would need to be even smaller. The containers would be traveling 20′ round trip instead of 12′ so the delay would increase 75%. And we would need to send a lot more containers to move the same amount of sand.

TCP works just like the sandbag problem. As distance increases, the TCP window shrinks, the time between transmission and acknowledgement increases, and the number of packets required to move the data grows. One reason for this is the effect of TCP congestion avoidance algorithms on the window size. The result is that the effective “speed” of the link decreases exponentially as the distance increases, regardless of bandwidth. RFC 1323 TCP Extensions for High Performance provides for mechanisms to deal with part of this problem. One method is to tune the TCP window on your hosts based upon a calculation of Bandwidth Delay Product (BDP). BDP = bandwidth x delay. Example: A 2Mb/s E1 link between California and the UK would have a BDP of 2.048Mb/s x 200ms = 51,200 Bytes. This is the ideal TCP window to fill the pipe so that the sender is not sitting idle waiting for ACK packets. Most hosts have a TCP Window default size of 64KB so, in this scenario, no adjustments would be needed. But, if the connection were a 45Mb/s DS3, then the BDP would be almost 1,100KB. In this scenario, TCP windows would need to be adjusted to use the available bandwidth at peak efficiency.

For most network applications, anything over 100ms latency is noticeable to your end users. Time sensitive applications such as VOIP or video teleconferencing suffer the worst experience when delay is introduced. Added to this is the impact of jitter. Jitter is the delay caused when packets travel alternative paths to the destination, and either arrive out of order, or with varying intervals between them. Applications such as e-mail that are bursty and not time sensitive do not feel the impact of latency to the same degree. How much of a problem is this for you today? One way to measure latency on your network is to use your carrier’s looking glass tools. A list of major looking glasses may be found at: http://www.nanog.org/lookingglass.html.

When designing for latency in a WAN it is important to first understand the applications on the network. After the applications have been profiled, steps can be taken to mitigate the impact of network delay. In part 2 of this article, we will discuss methods of designing for latency mitigation.

I’ve thrown together a little calculator for you to do some rudimentary math. Assuming an average of 30 days in a month, 24 hours in a day, 60 minutes in an hour, and 60 seconds in a minute. While the math is fairly straight-forward, it is by no means 100% accurate. Some months have less or more than 30 days, and on DST days, you get or lose an hour.

You should also be aware that while you are given some amount of transfer per month, this does not keep you from bursting above what the calculator shows. The calculator shows a “sustained” rate of data transfer. If you actually only use 1Mb/s of bandwidth, but you have a transfer allowance of 500GB/Month, you will be able to burst above the 1.5Mb/s sustained on occasion. 1-and-1, for instance, allows bursting up to 100Mb/s, but on their smallest VPS plan gives you 500GB/Month (roughly a T1/DS1 circuit sustained.)

Let me know if I’ve missed something.

-Jerry

Editors Note: Jerry Gilreath is our senior open systems consultant. He’s also the kind of guy who writes a JavaScript bandwidth calculator just for the heck of it. Jerry runs the excellent Gadget Workshop blog for everything expensive and geeky. Check it out.

To accommodate the DST change, most IT systems must be patched. Otherwise, timestamps will be off, and some applications my fail to work. For instance, if you synchronize your Windows Smartphone with Microsoft Exchange, and you want your calendar reminders to work, plan on applying patches or fixes to Windows XP, Windows 2003, Exchange 2003 & Windows Mobile. Otherwise, you may be late for that all-important TPS meeting.

Unfortunately, this change has not received the attention it deserves, so many IT shops have not yet started, and there are only 60 days to get patches tested and deployed. Also, it is not enough to assume that if your servers have the correct time, your applications and databases will work. Some applications are “system time aware,” yet others require their own unique patches.

To simplify matters, I have compiled the following list of major IT vendor links to their 2007 DST fixes:

• Microsoft Overview - microsoft.com
• Microsoft Windows Registry Hack - microsoft.com
• Microsoft Windows Mobile Hack - microsoft.com
• Microsoft Windows XP & 2003 - microsoft.com
• Microsoft Exchange - microsoft.com
• Microsoft Dynamics CRM - microsoft.com
• Sun Solaris - sun.com
• Sun Java - sun.com
• IBM - ibm.com
• IBM iSeriesV5R3 PTF SI24906 - ibm.com
• IBM Lotus Notes & Domino - ibm.com
• Novell Netware - novell.com
• Novell Groupwise - novell.com
• Apple OSX - apple.com
• Cisco IOS Workaround – “clock summer-time PDT recurring 2 Sun Mar 2:00 1 Sun Nov 2:00”
• Cisco IOS Release 12.4(11)XJ - cisco.com
• RedHat RHEL 4 - redhat.com
• Ubuntu tzdata - ubuntu.com
• Fortinet - forticare.com
• Juniper - juniper.net
• Oracle 11i - oracle.com
• MySQL - mysql.com
• PostgreSQL - postgresql.org
• WordPress Plugin - suominen.com

Finally, don’t forget firewalls, routers, switches, NTP appliances, time clocks, PBX systems, IVR/ACD systems, cell phones, PDAs, photo copiers, fax machines, and all the other devices on your network that recognize or require time/date to operate properly. All of these will likely need either patches or manual fixes to facilitate the change.

If you haven’t started yet, don’t wait. Compile a list of your time-dependant systems, applications, databases and devices. Prioritize the list based upon importance and impact if the device isn’t patched. Check the vendor’s website for fixing the problem. If you use outsourcers, such as hosting companies, application service providers (ASP), or payroll vendors, contact them and ask them to certify their readiness for the change. Create a plan and get it done. Hint: The fastest way to find vendor patches is with our old friend Google. Try this query: (site:vendorsite.com daylight saving time 2007).

If you find any other vendor patches, please feel free to post them here in the comments section.

Good luck,

-Bill

We determined that the likely scenario would be that the staff plugged it in to the network and obtained an “external” IP address from our DHCP servers. The likelihood that they would have statically assigned an IP seemed slim since they would have no way to determine which IPs would fall outside the DHCP range. Also, we counted on laziness to rule the day, since it would work fine with DHCP.

I came up with the following batch script to run against our DHCP servers. It dumps all current DHCP lease holders, and then checks them for known AP MAC address prefixes.

«File Download»

This is a simple but effective script. Put the main section of code in between the REM statements into a batch file. Create a text file called servers.tx2 with the IP addresses of your DHCP servers. Put the MAC addresses into a file called macs.tx2, and you are good to go. Note: you must be logged in as a domain admin, or at least as a user with rights to manage DHCP.

Sometimes the simplest answers are the best. When performing security audits, it is not practical or even possible to test every threat. A good security tester creates scenarios based upon the likely actions of the user, tests those scenarios, and then mitigates the threat. In this case, rogue APs were found and eliminated. Does this mean a more skilled person couldn’t figure out how to statically assign an IP and mask the AP from DHCP? Of course not. But the tests for that threat are harder, take longer, and cost more. Sometimes you go for the low hanging fruit. This test took less than 30 minutes to create, but yielded huge results. Hopefully you too will find it useful. If so, drop me a comment and let me know.

-Bill

UPDATE: The NETSH command used in this script requires Windows 2003 server. The WindowsXP version of NETSH does not have the DHCP option. Thanks to ALUNG for helpin me debug!

Digg This Story!

A firewall is the focal point in network and system security. This paper will look at proper firewall standards and best practices, modeled after Cisco SAFE and CERT, for using a firewall in an e-commerce network. Proper DMZ design and the physical placement of the firewall will be discussed. Also, firewall security policy rules, and how best to configure them. Besides normal firewall design, this paper will list other ways to secure the firewall itself, with proper logging and daily backups of the configuration, security audits, and disabling unneeded settings.

This paper will give network administrators a proper guide to securing a network and the firewall.

«download here»