Boosting web server performance

Last year (2006) we figured we had a unique idea — lets podcast the Superbowl Commercials for the expats who live in Hong Kong and other places that we cover through our main site GeoExpat.Com. We had no clue about what would happen next — 40 thousand+ visitors showed up on DevLib.Org and downloaded over 2TB worth of videos in a week. Well, that brought our server down to its knees and along with the hassles of making sure that the videos got uploaded, handling a database crash and dealing with the excess bandwidth, we were not sure if this should be repeated again — specially since the folks over at YouTube had become the destination of choice since last year.

But, common sense did not prevail and we said “what the heck, lets do it again and lets do it with some better technology”.

In this post, we list some of the technologies used and how we optimized our server to handle over 80K visitors and 4TB of downloads in two days.

First, we moved to a server that had large and fast SCSI disks. Expensive — but worth getting a server on a web hosting company that offers RAID 5 SCSI hardware.

Once the server was allocated to us, we went about making sure that we got the most we could, from the hardware we had, within the software limitations we had ( Apache 1.3.x was the big one ).

This is how we did it.

1) Install Lighttpd to serve static content. Lighttpd is an awesome server which frankly speaking beats the hell out of Apache when it comes to performance. While it does not have the mindshare that Apache has, it does a superb job in getting the files from the disk to the network — better than any other server we’ve tested.

Lighttpd was installed on port 443. We used 443 as this port is usually never blocked and changed most of our links for static content ( javascript, CSS, images both within templates and CSS ) to use this port. ( Eg: http://www.domain.com:443/static/images/image1.gif ). This would ensure that Apache would not have to spin around loading new processes to handle these files.

2) We optimized Apache / PHP 5.2.x to use XCache - a high performance PHP cache which converts the PHP text files into op-codes and caches them in memory. This ensures that the PHP layer in Apache gets pre-compiled code, from memory and improves performance significantly.

3) MySQL 4.2 was set to handle as many queries as possible from within its query cache.

The result ..

For the first time ever, we were able to saturate the network port at 100Mbps.

Ok, so we’re not YouTube or WikiPedia .. but hey .. we are pretty happy with the result.

A big THANK YOU goes to the folks at Lighttpd … once we recover the costs of the bandwidth, we’ll figure out a way to make sure you get some cash from us.