Measuring Web Server Performance with Httperf

Problem

You want to improve the performance of your application. As you experiment with various caching options or server configurations, it's critical that you measure their effects on performance so that you know what's working. You need a tool for accurately measuring the performance of your application.

Solution

Httperf provides facilities that generate various HTTP loads and measure the server's performance under those loads.

Download httperf from and install with: 

$ tar xvzf httperf-0.8.tar.gz
$ cd httperf-0.8
$ ./configure
$ make
$ sudo make install

By default, httperf is installed in /usr/local/bin/httperf. You invoke httperf from the command line, or you can put the command and parameters in a shell script to simplify repeated invocation. Using a shell script is a good idea, since you usually want to repeat your performance tests, and httperf has lots of parameters. For example: 

$ cat httperf.sh
#!/bin/sh httperf --server www.tupleshop.com \
 --port 80 \
 --uri /article/show/1 \
 --rate 250 \
 --num-conn 10000 \
 --num-call 1 \
 --timeout 5

This command specifies the server and port, followed by the page to be retrieved by each connection attempt. You also specify the rate that connections will be attempted (e.g., 250 requests per second) and the total number of connections to attempt (e.g., 1,000). The num-calls option tells httperf to make one request per connection. The timeout is the amount of time (in seconds) you're willing to wait for a response before considering the request a failure.

This command runs for approximately four seconds. To estimate any benchmark's run time, divide the num-conn value by the request rate (i.e., 10,000 / 250 = 40 seconds). When the command finishes, it generates a report that contains measurements showing how well the server performed under the simulated load.

Discussion

There are two common measures of web server performance: the maximum number of requests per second the server can handle under sustained overload, and the average response time for each request. Httperf provides a number of options that allow you to simulate a request overload or some other common condition. The important thing is that you can collect actual data about how different server configurations actually perform.

When deciding how much to adjust a specific variable in your configuration, such as the number of Mongrel processes to run, you should always start from a baseline reference point (e.g., measure the performance of a single Mongrel server). Then make one adjustment at a time, measuring performance again after each change. This way you can be sure that the change, such as adding one more Mongrel server, actually helps performance. Performance is tricky, and it isn't uncommon for seemingly innocuous changes to backfire.

The output of httperf is organized into six sections. Here's the output from the solution's command: 

Total: connections 7859 requests 2532 replies 307 test-duration 47.955 s Connection rate: 163.9 conn/s (6.1 ms/conn, <=1022 concurrent connections)
Connection time [ms]: min 896.1 avg 3680.8 max 8560.2 median 3791.5 
 stddev 1563.1
Connection time [ms]: connect 1445.8
Connection length [replies/conn]: 1.000
Request rate: 52.8 req/s (18.9 ms/req)
Request size [B]: 85.0
Reply rate [replies/s]: min 1.8 avg 6.4 max 16.0 stddev 4.4 (9 samples)
Reply time [ms]: response 1619.1 transfer 35.0
Reply size [B]: header 85.0 content 467.0 footer 0.0 (total 552.0)
Reply status: 1xx=0 2xx=13 3xx=0 4xx=0 5xx=294
CPU time [s]: user 0.61 system 30.27 (user 1.3% system 63.1% total 64.4%)
Net I/O: 7.8 KB/s (0.1*10^6 bps)
Errors: total 9693 client-timo 7261 socket-timo 0 connrefused 0 
 connreset 291
Errors: fd-unavail 2141 addrunavail 0 ftab-full 0 other 0

The six groups of statistics are separated by blank lines. The groups consist of overall results, results pertaining to the TCP connections, results for the requests that were sent, results for the replies that were received, CPU and network utilization figures, as well as a summary of the errors that occurred (timeout errors are common when the server is overloaded).

Not to belabor the point, but don't just look at the avg result from performance-measuring runs and think that you have a handle on how well your server is performing. min and max times, stddev values, and failures and errors are all trying to tell you things you need to know, but that can be complicated to understand. If you have to deal with serious server performance analysis, it really will pay off to learn at least some rudimentary stats and analysis concepts.

See Also