I’ve spent the past few days trying to develop a simple mathematical model to predict the expected availability of complex systems. In IT, we are often asked to develop and commit to service level agreements (SLAs). If the points of failure of the system are not analyzed, and then the system availability calculated, the SLA is flawed from the beginning. To complicate matters further, different people have different definitions of availability. For instance, does scheduled downtime for maintenance count against your system availability calculation?

Common Availability Definitions:

1. Availability = MTBF/(MTTR+MTBF) (Mean Time Between Failure, Mean Time To Recover). This is a classic definition of availability and is often used by hardware manufacturers when they publish an availability metric for a given server.
2. Availability = (Uptime + Scheduled Maintenance)/(Unscheduled Downtime + Uptime + Scheduled Maintenance). This is an IT centric availability metric where the business can support scheduled downtime after hours. This model works for some types of systems, such as a file server that isn’t needed at night, but it doesn’t work as well for websites, even though many web companies still use this for their SLAs.
3. Availability = Uptime/(Uptime + Downtime). This metric best applies to systems that are needed 24×7 such as e-commerce sites.

Availability is most often expressed as a percentage. Sometimes, people will refer to “four nines” (99.99%) or “five nines” (99.999%). To simplify things, the following table shows the minutes of downtime allowed per year for a given availability level:

Based on the above table, you can see there is a big difference between an SLA specifying 99% availability (88 hours of downtime per year) and 99.9% availability (8.8 hours of downtime per year). But how can we be sure what the expected downtime of the system is? In the most simplistic form, the expected availability of a system equals the expected availability of each of the components of the system multiplied together. So if the system was composed of two servers, and each server had an expected availability of 99%, then the expected availability of the system would be 99% * 99% = 98.01%. Note: I have used the term “expected availaibilty.” We are calculating a future expectation of the system over an extended period of time, not a historical availability. For the rest of this article, I will drop the term expected for brevity, but it is always implied.

The simplistic model above is useful for illustrating the point that downtime is cumulative. In other words, if I expect each component to be down 88 hours/year, and a failure of either component is a failure of the system, then the system has an expected downtime of 174 hours. Why not 176 hours? Well, occasionally both components will be down at the same time.

Real world systems are never this simple. Typically, the system will be comprised of multiple components, some with redundancy, and each with different levels of component availability. Modeling these requires slightly more complicated formulas, but once you have the concepts down, then the actual calculations can be quickly performed in a spreadsheet. Before we move forward, we need some basic notations to simplify our formulas:

Basic Nomenclature
Availability Component 1 = Ac₁
Availability Component 2 = Ac₂
Availability Component 3 = Ac₃
Availability Component n = Ac_n
Availability System = As

Now that that is out of the way, we are ready for our first formula. When a system is made up of n number of components that are each single points of failure then the system availability can be calculated as:

EQUATION #1: As = Ac₁ * Ac₂ * Ac₃ * …Ac_n

Consider a 24×7 e-commerce site with lots of single points of failure. We might model the site as having the following eight components:

If any of these components fails, the website will crash. The expected availability of the site would be 85%*90%*99.9%*98%*85%*99%*99.99%*95% = 59.87%. Note that we are modeling each component as a whole, rather than looking at its parts. We could break down the web service into software (Apache), code (our web site), and hardware (motherboard, hard drives, etc.). For our purposes, complexity does not necessarily improve the model, so we will deal with the service as a whole. Also, for this discussion we will use the third definition of availability above. To our users, it does not matter if the site is down because of maintenance or a hard drive failed.

Assuming we want to keep our jobs, we need to figure out a way to increase this availability. Two obvious targets for improving the site stability would be the web service and the firewall. The question is, what effect would adding another web server have on the availability of this service. This leads us to our second equation. When a system is comprised of two redundant components, then the availability of the system can be calculated as:

EQUATION #2: As = Ac₁ + ((1 - Ac₁) * Ac₂)

Using our example of a web server with an availability of 85%, then adding a second server would increase the availability to: 85% + (1-85%)*85% = 97.75%. The logic behind this is that when the 1st server is down (15% of the time), the second server is still up 85% of the time. This may or may not translate into actual real world availability. For instance, if the web server was down so often because we constantly need to take it offline to deploy new code, then adding a second server should translate to higher availability, because we could deploy code to one offline server, while the other server stays up. In this case, our real world availability increase might be greater than 12.75%. Conversely, if our service is down due to code bugs, then adding a second server could in some cases make the availability worse by exasperating the bug.

The point is that in general if you’ve accurately estimated the availability of the component, then the equation will work. Note also that the equation works even if the components have unequal availability estimates. Assume that the web server has an availability problem because the hardware is undersized. Now assume the second server we purchase has twice the capacity, and we determine that by itself the availability of the new server would be 90%, then our equation changes to: 85% + (1-85%)*90% = 98.5%.

So let’s add this back into our system calculation above. Assume that we add a second web server, and a second firewall, increasing the availability of each component system to 97.75%. Now, the availability of our system would be: 97.75%*90%*99.9%*98%*97.75%*99%*99.99%*95% =79.10%. Better, but still not great. It is hard to achieve any level of high availability when you have single points of failure. So let’s assume that we add a redundant component for all our servers and network equipment. Let’s also assume we add a second ISP for carrier diversity, but we are still within one physical data center. Our availability equation would now be: 97.75%*99%*99.9999%*99.96%*97.75%*99.99%*99.99%*99.75% = 94.3%. Getting better. Removing single points of failure improved our availability of the system from 60% (3506 hours of downtime/year) to 94.3% (500 hours of downtime/year).

Equation #2 above modeled adding a single redundant component. In some cases, we may add more than one redundant component. For example, we may have more than two web servers. In this case, we need to iterate through equation #2 multiple times to find the effect of the additional components, which brings us to our third equation. When trying to calculate the availability of a service with n number of redundant components, we calculate it as:

EQUATION #3: As = Ac_(n-1) + ((1 - Ac_(n-1)) * Ac_n)

In the case of our web service, adding a 3rd server would change the availability to: 97.75% + (1-97.75%)*85% = 99.6625%. Adding a fourth server would increase availability to: 99.6625% + (1-99.6625%)*85% = 99.949%. Note that there is a diminishing level of return. Adding a 2nd server increased availability by 12.75%. Adding a 3rd server only gained us 1.9125%. The fourth server bought us a paltry .2865%. And yet, even with 3 more servers than we need to service our load, we still have not yet achieved the elusive four nines of availability. Designing a highly available system requires us to make individual components highly available AND add redundancy of components. If the individual web server in our example had an availability of 90%, instead of 85%, then the availability of two servers would be 99% and three servers would be 99.99%.

Equations 2 & 3 have a simple flaw, in that they assume a single component can handle the load, and that the load is constant. What if under normal operations, one web server can handle the load, but under peak, we need three servers? Then our availability for three servers under normal load would be 99.775%, but under peak load, the availability would drop back to 85%. Under peak load, the failure of one server would cause a loss of the service, so we drop back to the availability of a single box. What if our peak load required 2 servers? In this case, the availability under peak would be 97.75%. If peak requires two servers, and we have three, then we can lose one server and still be operational, therefore our availability is the equivalent of having two servers. The important concept here is there is an inverse relationship between load and availability.

What should be obvious by now, is that achieving truly high levels of availability (99.9% - 99.999%) is very difficult and very expensive. One of the most expensive single points of failure to eliminate is the data center. In most cases, this effectively doubles the cost of your infrastructure, and the cost may be even greater than 2x, because you will often need to invest in technology to keep the servers in each data center in synch with each other.

Consider, however, the impact of adding a fully redundant data center. In our example above, the availability of our data center will redundant servers and ISPs was 94.3%. Adding a second data center with the necessary technology to make both centers work active-active (both data centers take traffic at the same time) would increase our availability to: 94.3% + (1-94.3%)*94.3% = 99.675%. Adding a second data center saved us 471 hours of downtime per year!

In this example, we assumed that each data center was an independent system, so a failure of a service in one data center would be a failure of the entire system in that data center. This is not always the case. For instance, if properly designed, a web server in one data center could connect to a database server in the other data center. In this case, the expected availability of the system would be higher that 99.675%. If you were able to design your site so that each service operated independent of the other services, then the availability in our example would increase from 99.675% to 99.888% (each service would have 3 redundant components, except the data center, which would have 1 redundant component).

These formulas are much easier to play with in Excel. Paste the following table into a spreadsheet, starting at cell A1:

Now that you have the basic concepts down, and the beginnings of a spreadsheet for calculating changes to our assumptions, you can focus on applying these theories to your unique situation. Start by breaking down your system, be it a web site, an accounting system, or a file server, into individual component services. For each service, determine the minimum number of units required for the system to work, and the expected availability of a unit.

Estimating availability can be a challenge. One method would be to look at historical data. If you don’t have access to good data, you can form an estimate based upon your standard operating parameters. For instance, if you release new code to your web server twice per month, and each release causes 2 hours of downtime, that would translate to 48 hours of downtime per year. If you expect to perform operating system maintenance once per quarter, at an estimated downtime of 2 hours per quarter, that would equal another 8 hours per year. If you also anticipate one hardware failure per year, and had a next business day warranty, that would translate to on average 41 hours of downtime per year (Friday outages get repaired on Monday. Saturday and Sunday outages get repaired on Tuesday). Adding these numbers up, we get: 48 + 8 + 41 = 98 hours of downtime per year, or an estimated 98.882% availability.

With a little work, you can estimate a realistic level of availability of your system. This is the cornerstone of creating realistic and attainable SLAs. These formulas can help IT negotiate SLAs with the business, and can help determine the comparative ROI of different solutions. For instance, let’s say you were trying to choose a web server solution, and you had two choices:

• Choice number 1 consists of 4 servers using cheap hardware with no internal redundancy. Each server costs $3,000. You estimate the availability of each server to be 75%.
• Choice number 2 consists of 2 servers using expensive hardware with redundant hard drives and power supplies. Each server costs $20,000. You estimate the availability of each server to be 99%.

You estimate the cost of downtime to be $500/hour, and you expect these servers to support your site load with a single server for the next 3 years, after which they will be replaced. Using the above numbers, Solution #1 has an expected availability of 99.6%, at a cost of $12,000. Solution #2 has an expected availability of 99.99% at a cost of $40,000. Solution #1 would experience 34 hours/year, or 102 hours over three years of downtime more than solution #2. Over three years, this extra downtime would cost $51,000. So by spending $28,000 upfront, you would get a three year ROI of 182%. Note that the model is only as good as your estimates. If the servers in solution #2 only had 95% availability, then their combined availability would be 99.75%, which would only provide 13 hours less downtime per year. In this case, you would only save $20,000 in downtime over three years for your $28,000 investment, so you would be better off with solution #1.

Designing and operating highly available systems is complicated work, but with a few simple formulas, it is possible to understand and predict their behavior at a macro level. This will allow you to make better decisions when choosing between multiple options, and give more realistic predictions when negotiating SLAs.

The first script in the series is a Windows Services lockdown script for Windows XP & 2003. Disabling services is generally a good idea to reduce the threat profile of your computer, and to improve its performance. Every security guide out there tells you to disable unnecessary services. A few of them also give some guidance as to which services are unnecessary. Few of them tell you how to disable them consistently.

There are three ways to disable services: 1) Use the Services MMC GUI. This is a time consuming process and is prone to mistakes. 2) Use Group Policy. This works well for environments that use Group Policy, but is harder to implement for stand-alone servers, such as web servers. 3) Use the sc.exe command line utility.

If you do not know the sc command, learn it! sc is a powerful utility for controlling services on local or remote hosts. sc will let you configure how services start, change the user account and password they run under, and start/stop/pause the services. The basic syntax of sc is:

sc <server> [command] [service name] <option1> <option2>

We are going to use 2 different sc commands in our service lockdown script: config & stop. These should be self explanatory, but config will allow us to disable the service, and stop will stop the service. To make this work, we need three files: 1) The script batch file; 2) a list of servers by name called hosts.txt; 3) a list of services we want to disable called services.txt. The two text files must be in the same directory as the batch file. The code is fairly simple:

REM***(c) 2007 William L. Dougherty
REM***Script created by Bill Dougherty to disable services on Windows 2003 & XP
for /f %%a in (hosts.txt) do call :serviceconfig %%a
goto :eom
:serviceconfig
for /f %%b in (services.txt) do (sc \\%1 config %%b start= disabled)
for /f %%c in (services.txt) do (sc \\%1 stop %%c)
:eom

Pretty simple, huh? We loop through a list of servers, and then for each server, we loop through a list of services. The service list must be the service name and not the display name of the services. Some service names are less than intuitive, such as the service name for IPSEC Services is PolicyAgent. You can get the service name from the services MMC by clicking the service and looking at its properties, or from the command line using sc query. Either way, once you build your service list the first time, you’ll rarely need to revist this.

This script requires you to have administrator access on the target host. This works great in a domain environment, but what if you are dealing with stand-alone servers? You can still use this script so long as the user credentials you are logged in as have admin access on the target. If not, the you’ll need to log into each server and run the script locally. If this is the case, you only need the script file and the list of services. You should modify the script like this:

for /f %%b in (services.txt) do sc config %%b start= disabled
for /f %%c in (services.txt) do sc stop %%c
:eom

This script works with both Windows XP and Windows 2003, although the two platforms have different services. There are several good sources for figuring out which services are unnecessary. I highly recommend the TechRepublic guides: XP Services Guide and 2003 Services Guide. Over the years, I have developed my own lists. I highly recommend you spend the time to research these services, and test the impact of disabling them BEFORE you run this script against your production network!

Windows 2003 services that can be disable:

Alerter
AppMgmt
ClipSrv
TrkWks
TrkSvr
MSDTC
ERSvc
helpsvc
HidServ
ImapiService
IsmServ
LicenseService
Messenger
mnmsrvc
NetDDE
NetDDEdsdm
nla
NtLmSsp
WmdmPmSM
appmgr
RemoteAccess
SCardSvr
SENS
LmHosts
TapiSrv
TlntSvr
Tssdis
Themes
uPS
uploadmgr
WebClient
AudioSrv
stisvc
WZCSVC

Windows XP services that can be disabled:

Alerter
AppMgmt
ClipSrv
TrkWks
MSDTC
ERSvc
FastUserSwitchingCompatibility
helpsvc
HidServ
CiSvc
Messenger
mnmsrvc
NetDDE
NetDDEdsdm
nla
NtLmSsp
SysmonLog
WmdmPmSM
RSVP
RemoteAccess
SCardSvr
SSDPSRV
LmHosts
TapiSrv
TlntSvr
Themes
uPS
upnphost
WZCSVC

I hope you find this script useful. Check back often for additional scripts in the series.

-Bill

Microsoft Windows Mobile Daylight Saving Time Patch

You can either download the CAB file directly to your cell phone, or download to your PC, copy it via activesync to your phone, and then run it.

Since I never recommend just trusting an executable you find online, you may want to check the contents of this file. If you save it with a .txt extension, you can open it with notepad. There will be some extra characters, but the bulk of the file should match the patch from Microsoft. If you want to create your own CAB file, you need the cabwiz.exe executable with is part of Visual Studio. Save the code below as a .INF file and then run cabwiz.exe against it.

[Version]
Signature=”$Windows NT$”
Provider=”Microsoft”
CESignature=”$Windows CE$”[DefaultInstall]
AddReg = RegSettings.All

[SourceDisksNames]
1 = ,”Common Files”, , C:\My_Cabs\DST2007WM5

[SourceDisksFiles]
readme.htm = 1

[DestinationDirs]
Files.Common = 0, %CE3%
DefaultDestDir = 0,%InstallDir%

[CEStrings]
AppName = “Update for 2007 Daylight Saving Time”
InstallDir = %CE3%

[Strings]
NEW = “SOFTWARE\Microsoft\Timezones\60″
ATL = “SOFTWARE\Microsoft\Timezones\50″
EST = “SOFTWARE\Microsoft\Timezones\35″
CTL = “SOFTWARE\Microsoft\Timezones\20″
MNT = “SOFTWARE\Microsoft\Timezones\10″
PAC = “SOFTWARE\Microsoft\Timezones\4″
ALS = “SOFTWARE\Microsoft\Timezones\3″
CHI = “SOFTWARE\Microsoft\Timezones\13″
MXC = “SOFTWARE\Microsoft\Timezones\30″
TJA = “SOFTWARE\Microsoft\Timezones\5″

[Files.Common]
readme.htm,,,0

[RegSettings.All]
;GMT-3:30 Newfoundland
HKLM,%NEW%, TZI, 0×00000001,d2,00,00,00,00,00,00,00,c4,ff,ff,ff,00,00,0b,
00,00,00,01,00,02,00,00,00,00,00,00,00,00,00,03,00,00,00,
02,00,02,00,00,00,00,00,00,00

;GMT-4 Atlantic
HKLM,%ATL%, TZI, 0×00000001,f0,00,00,00,00,00,00,00,c4,ff,ff,ff,00,00,
0b,00,00,00,01,00,02,00,00,00,00,00,00,00,00,00,03,00,00,00,
02,00,02,00,00,00,00,00,00,00

;GMT-5 Eastern US
HKLM,%EST%, TZI, 0×00000001,2c,01,00,00,00,00,00,00,c4,ff,ff,ff,00,00,0b,
00,00,00,01,00,02,00,00,00,00,00,00,00,00,00,03,00,00,00,
02,00,02,00,00,00,00,00,00,00

;GMT-6 Central US
HKLM,%CTL%, TZI, 0×00000001,68,01,00,00,00,00,00,00,c4,ff,ff,ff,00,00,0b,
00,00,00,01,00,02,00,00,00,00,00,00,00,00,00,03,00,00,00,
02,00,02,00,00,00,00,00,00,00

;GMT-7 Mountain US
HKLM,%MNT%, TZI, 0×00000001,a4,01,00,00,00,00,00,00,c4,ff,ff,ff,00,00,0b,
00,00,00,01,00,02,00,00,00,00,00,00,00,00,00,03,00,00,00,
02,00,02,00,00,00,00,00,00,00

;GMT-8 Pacific US
HKLM,%PAC%, TZI, 0×00000001,e0,01,00,00,00,00,00,00,c4,ff,ff,ff,00,00,0b,
00,00,00,01,00,02,00,00,00,00,00,00,00,00,00,03,00,00,00,
02,00,02,00,00,00,00,00,00,00

;GMT-9 Alaska
HKLM,%ALS%, TZI, 0×00000001,1c,02,00,00,00,00,00,00,c4,ff,ff,ff,00,00,0b,
00,00,00,01,00,02,00,00,00,00,00,00,00,00,00,03,00,00,00,
02,00,02,00,00,00,00,00,00,00

;GMT-7 Chihuahua, La Paz
HKLM,%CHI%, TZI, 0×00000001,a4,01,00,00,00,00,00,00,c4,ff,ff,ff,00,00,0a,
00,00,00,05,00,02,00,00,00,00,00,00,00,00,00,04,00,00,00,
01,00,02,00,00,00,00,00,00,00

;GMT-6 Mexico City, Monterrey
HKLM,%MXC%, TZI, 0×00000001,68,01,00,00,00,00,00,00,c4,ff,ff,ff,00,00,0a,
00,00,00,05,00,02,00,00,00,00,00,00,00,00,00,04,00,00,00,
01,00,02,00,00,00,00,00,00,00

;GMT-8 Tijuana, Baja California
HKLM,%TJA%, TZI, 0×00000001,e0,01,00,00,00,00,00,00,c4,ff,ff,ff,00,00,0a,
00,00,00,05,00,02,00,00,00,00,00,00,00,00,00,04,00,00,00,
01,00,02,00,00,00,00,00,00,00
HKLM,%TJA%, Display, 0×00000000, “GMT-8 Tijuana, Baja California”

I confirmed using Mobile Registry Editor from http://www.breaksoft.com/ that the keys did not exist before the patch, and now exist after the patch, on my Samsung Blackjack. The only way I know to test that the patch works is to change the date/time on your phone to March 11, 2007 01:59:00AM and see what happens. Since I am lazy, I plan to wait until March 11 & just see what happens. If you test this patch before then, please leave a comment and let me know if it works.

-Bill

UPDATE - 01/29/2007

Thanks to Eric for figuring this out: Once you apply the patch, you must manually change your timezone to something else and click apply. After that, change it back to your correct time zone. This forces the phone to re-read the registry file. I tested this on my Blackjack and it worked perfect. The phone changed the time correctly at 03/11/2007 2:00AM to 3:00AM and gave me a dialog box telling me it had adjusted for DST.

UPDATE - 02/06/2007

David found the following link to a similar fix for Australian time zone changes: http://www.pocketpcthoughts.com/articles.php?action=expand,53751. I have not tested these CABs so use at your own risk.

UPDATE - 03/07/2007

Mike found an updated patch released by Microsoft. Details can be found here:  http://www.microsoft.com/windowsmobile/daylightsaving/default.msp. I recommend everyone install this patch instead of my unsupported CAB file.