One concept that continues to elude many IT managers is the impact of latency on network design. 11 years ago, Stuart Cheshire wrote a detailed analysis on the difference between bandwidth and latency ISP links. Over a decade later, his writings are still relevant. Latency, not bandwidth, is often the key to network speed (or lack thereof).

I was reminded of Cheshire’s article and the underlying principles recently when working on an international WAN design. What Cheshire noted was that light signals pass through fibre optics at roughly 66% of the speed of light, or 200*10^6 m/s. Regardless of the equipment or protocols you use, your data cannot exceed that theoretical limit. This limit equals the delay between when a packet is sent, and when it is received, aka latency.

In the US, we tend to focus on bandwidth and carrier technology when ordering circuits, completely ignoring latency. For instance, when choosing between cable and DSL for your house do you ever ask the carrier for its latency SLA? Maybe you should. Using a cable connection a ping to www.google.com in Mountain View, CA from my house (137 KM) yields an average ping time (aka round-trip time or RTT) of 73ms. The theoretical latency for this distance (round trip) is 1.37ms meaning my cable connection is roughly 50 times worse than the theoretical limit. No surprise that Comcast focuses on bandwidth and not latency in its marketing.

Cable and DSL circuits in the US are generally not business class and do not carry any service level agreement (SLA) on latency or availability. Businesses who use these circuits for business critical services do so at their peril. Business circuits such as Frame Relay and MPLS do generally include latency SLAs, but understanding the difference between the SLA and your actual experience can have a massive impact the performance of your network. For instance, let’s say a carrier advertises a 55ms round trip SLA in the US. This SLA equals the maximum latency between any two points of presence (POP) on their network.

The coast-to-coast distance in the US is roughly 5,000KM for a theoretical latency of 50ms, so a 55ms RTT SLA is pretty good. But that doesn’t mean packets on your network will only take 55ms to cross the country. When designing your WAN you must also account for the latency added by your network equipment and your servers, and the distance between the carrier’s POP and your offices. As a result, a well designed US WAN will still experience 75-80ms ping times. A poorly designed WAN can experience much worse times.

Now consider creating an international WAN. In this case, you typically will receive multiple SLAs from the carrier for different parts of the network. For instance, when designing an MPLS connection between California and the UK, the SLAs would be approximately 55ms within the US plus 95ms to cross the Atlantic Ocean plus 21ms to connect within the UK. Add the latency of your network and you get ping times of 175ms to 200ms.

At this point you are probably asking yourself “so what?” Two tenths of a second is no big deal. The answer is the impact of latency on TCP windowing. Transmission Control Protocol (TCP) has a flow control mechanism that senses latency and bandwidth between two hosts and determines the rates which data will be transferred. The TCP window is the amount a unacknowledged data a sender can transmit before waiting for a TCP ACK. As the latency increases, the TCP window shrinks, meaning the sender sends less data before waiting for an ACK. This helps reduce the amount of data that will need to be retransmitted in case a packet gets lost. Smaller windows equals more packets, and more packets equals more data because each packet carries the overhead of a 40 byte TCP/IP header regardless of if the payload is 1 byte or 1500 bytes.

The result is what I call the “Sandbag Problem.” Let’s say the two of us are trying to fill sandbags. My job is to scoop sand into a container and hand the full container to you (data). Your job is to empty the container into a sandbag and hand the empty container (ACK) back to me. Occasionally you drop the container so I have to fill it again (Retransmit). If we were standing next to each other, the time it takes for me to hand the container to you, have you empty it, and hand it back to me (latency) would be very small. Now imagine there is a 6′ wall between us, and I need to hand the container over to you.

The wall changes several aspects of our filling operation. First, the size of the container must be smaller because I cannot lift the same weight over my head that I can lift at waist level. Second, the time to complete one cycle would increase because it takes longer to lift the container 6′ than it does 3′. Third, you would drop more containers so retransmissions would increase. As the wall gets taller, the problem gets worse. If the wall were 10′ tall, we would be throwing containers instead of lifting them, so they would need to be even smaller. The containers would be traveling 20′ round trip instead of 12′ so the delay would increase 75%. And we would need to send a lot more containers to move the same amount of sand.

TCP works just like the sandbag problem. As distance increases, the TCP window shrinks, the time between transmission and acknowledgement increases, and the number of packets required to move the data grows. One reason for this is the effect of TCP congestion avoidance algorithms on the window size. The result is that the effective “speed” of the link decreases exponentially as the distance increases, regardless of bandwidth. RFC 1323 TCP Extensions for High Performance provides for mechanisms to deal with part of this problem. One method is to tune the TCP window on your hosts based upon a calculation of Bandwidth Delay Product (BDP). BDP = bandwidth x delay. Example: A 2Mb/s E1 link between California and the UK would have a BDP of 2.048Mb/s x 200ms = 51,200 Bytes. This is the ideal TCP window to fill the pipe so that the sender is not sitting idle waiting for ACK packets. Most hosts have a TCP Window default size of 64KB so, in this scenario, no adjustments would be needed. But, if the connection were a 45Mb/s DS3, then the BDP would be almost 1,100KB. In this scenario, TCP windows would need to be adjusted to use the available bandwidth at peak efficiency.

For most network applications, anything over 100ms latency is noticeable to your end users. Time sensitive applications such as VOIP or video teleconferencing suffer the worst experience when delay is introduced. Added to this is the impact of jitter. Jitter is the delay caused when packets travel alternative paths to the destination, and either arrive out of order, or with varying intervals between them. Applications such as e-mail that are bursty and not time sensitive do not feel the impact of latency to the same degree. How much of a problem is this for you today? One way to measure latency on your network is to use your carrier’s looking glass tools. A list of major looking glasses may be found at: http://www.nanog.org/lookingglass.html.

When designing for latency in a WAN it is important to first understand the applications on the network. After the applications have been profiled, steps can be taken to mitigate the impact of network delay. In part 2 of this article, we will discuss methods of designing for latency mitigation.

With a 100% rebate, how can you lose? Order today.

With that said, I want to actively promote the concept of e-Tipping. e-Tipping is a way to pay the blogger back for the hard work they have put into their blog, similar to leaving a tip at a restaurant. There are several ways to leave an “e-Tip”:

1. Click the ads!!! - Most blogs these days have ads. If you like the article you just read, visit the site’s sponsors. The blogger will make, on average, about $.05 per click…not much of a tip, but it adds up when a lot of people are reading your blog. UPDATE 06/01/2007: It is against Google’s terms and conditions to directly ask people to click your ad links. I respect Google and their terms, and would not want to circumvent their business model. Most Google ads are contextually related to the blog article. So, if you find an article valuable, take the time to look at the ads. If you are interested in any of the products, by all means click the ad, but please do not click ads soley for the purpose of driving up click revenue for the blogger.
2. Donate - Many blogs offer paypal links. If you find the articles especially useful, make a small cash donation. This is often the best way to support a blogger if you want him to provide you with specific additional information.
3. Leave a Comment - Blogging can be a lonely business. Comments show you care. They also make articles seem more relevant to the next reader.
4. Digg/Slashdot/Link the article - Bloggers want traffic. The more the better. Also, Google ads pay for page views, as well as clicks. Help the blogger promote their site, and they will continue to create great content.
5. Read the rest of the blog - Chances are you found the blog from a link aggregator. If you find the article useful, click out to the parent site, and scan some of the other articles. You’ll probably find other articles of interest, worthy of your time and e-Tips.

In the past month, edgeblog has benefited greatly from other blogs and sites linking to us. As an e-Tip to them, let me encourage you to visit each of these. They are quality sites worthy of your attention. Like the comic said, “I’ll be appearing here every night this week, and don’t forget to e-Tip your blogger!”

-Bill

Please visit our friends and leave an e-Tip:

• http://www.gadgetworkshop.com
• http://www.msmobiles.com
• http://www.thedigeratilife.com/blog/
• http://www.reddit.com
• http://www.geekrant.org
• http://www.zoliblog.com/blog
• http://www.afarther.com/
• http://blog.ambersail.co.uk/wordpress/
• http://bloggerfodder.net/
• http://hangoversunday.com/
• http://arghwebworks.com/
• http://reddiggulo.us/
• http://www.taleslinger.com/
• http://gfxfor.us/

Digg This Story!

Which brings me to the title of this article. During my studies for the CEH exam, I was exposed to the seriously flawed CartIt.cgi shopping cart application. CartIt.cgi is a widely used shopping cart that stopped being developed last year. The reason this application is flawed is that it uses hidden fields within the HTML POST to submit the price and quantity when the user clicks on the add-to-cart button. Hidden fields are easy to manipulate. One of the easiest is to use a local proxy, such as Paros, to intercept the POST, effectively launching a man-in-the-middle attack. This allows you to change the price before it is submitted to the server.

Example:

Doing a simple Google search for cartit.cgi+plasma, I found a web site that sells plasma TVs (Which shall remain nameless to prevent being sued). The website thinks it is selling TVs for $7,599, but we can pay whatever we want by intercepting the POST and changing the price. If you think the company would catch this error, think again. Many companies outsource the fulfillment of orders, and never check the prices being charged. Note: I do not endorse e-shoplifting, so I did not complete the above transaction, but I know for a fact that the site will accept the order for $.99. Now, $.99 is extreme enough to *maybe* raise a flag. A simpler approach is to just move the decimal over 1 or 2 places. This way, if the company does notice, they will assume it was a processing error on their side. So maybe this article should be titled: “How to buy a 65″ plasma for $75.99.”

Another simple search for CartIt reveals that many hosting companies are still actively supporting CartIt.cgi. For example, IM1 Web Hosting calls CartIt “a powerful e-commerce solution for merchants and professional Webmasters…CartIt is an extensible, scalable shopping cart system that can handle just about any product or product combination you throw at it.” Disgraceful.

Note also that the shopping cart displayed above was deemed secure by VeriSign, Control Scan, BBB Online, Mastercard, & Visa. How much confidence do you have in those programs now??? Hopefully not much.

The exploit described above is not unique to CartIt. There are many shopping carts that use hidden POST fields. A shopping cart should allow the user to submit the SKU and the quantity, but never the price. The price should be queried from a database. The point here is that if you do not know how your applications work, you cannot rely upon their security. If you are using a shopping cart provided by your hosting company to run your site, we recommend you check it for these exploits. Failing to do so can be hazardous to your bottom line.

-Bill

Digg This Story!

Editor’s note: The techniques described in this article are for educational purposes only. We do not encourage or endorse the manipulation of 3rd party web applications to change the price. E-Shoplifting is a crime.

According to our friends over at MS Mobiles, the situation changes Thursday, the 16th of November. Cingular is set to ship 2 new smartphones:

• The Samsung Blackjack is a Q killer. It offers G3 HSDPA broadband speeds, which are more than double the speeds of Verizon’s “high-speed” broadband. It is also a quad-band phone, making it well suited for world travelers.
• The HTC Hermes, aka Cingular 8525 is the latest version of its popular pocket PC phone, which offers full touch screen PDA functionality. It is similar to the Verizon XV6700, but like the Blackjack, the 8525 offers HSDPA broadband. The 8525 also has a 2 megapixel camera and 802.11 support.

Both of these have great multimedia capabilities, such as streaming audio and video, but they really shine as business tools. If you have Microsoft Exchange 2003, these phones are a must-have. In Exchange 2003, Microsoft beefed up its wireless activesync. With a Windows-based phone and an accessible Exchange 2003 web access server, you can synchronize e-mail, contacts and calendars over the cellular network. With Exchange SP2, Exchange can push data to your phone, but this is a huge battery drain. The better option is to set your phone to synchronize every 5-15 minutes. Exchange combines with a Windows Smartphone blows away the capabilities of blackberrys and you don’t need any extra software to roll this out to your entire company.

I’m going to order the Blackjack as soon as it is released on Thursday, and hope to have a real-world review up soon.

We determined that the likely scenario would be that the staff plugged it in to the network and obtained an “external” IP address from our DHCP servers. The likelihood that they would have statically assigned an IP seemed slim since they would have no way to determine which IPs would fall outside the DHCP range. Also, we counted on laziness to rule the day, since it would work fine with DHCP.

I came up with the following batch script to run against our DHCP servers. It dumps all current DHCP lease holders, and then checks them for known AP MAC address prefixes.

«File Download»

This is a simple but effective script. Put the main section of code in between the REM statements into a batch file. Create a text file called servers.tx2 with the IP addresses of your DHCP servers. Put the MAC addresses into a file called macs.tx2, and you are good to go. Note: you must be logged in as a domain admin, or at least as a user with rights to manage DHCP.

Sometimes the simplest answers are the best. When performing security audits, it is not practical or even possible to test every threat. A good security tester creates scenarios based upon the likely actions of the user, tests those scenarios, and then mitigates the threat. In this case, rogue APs were found and eliminated. Does this mean a more skilled person couldn’t figure out how to statically assign an IP and mask the AP from DHCP? Of course not. But the tests for that threat are harder, take longer, and cost more. Sometimes you go for the low hanging fruit. This test took less than 30 minutes to create, but yielded huge results. Hopefully you too will find it useful. If so, drop me a comment and let me know.

-Bill

UPDATE: The NETSH command used in this script requires Windows 2003 server. The WindowsXP version of NETSH does not have the DHCP option. Thanks to ALUNG for helpin me debug!

Digg This Story!

If you are looking for a compliance solution for e-mail, I highly recommend Hewlett-Packard’s RISS/RIM product suite. Symantec’s Enterprise Vault is also a good product, although in my experience it does not scale well beyond 1 data center. If you find this paper useful, drop me a comment and let me know how you are dealing with e-mail & SOX.
«download here»

 Digg This Story!