Migrating Django and CherryPy to lighttpd

After learning that my 256mb Debian Xen VPS died instantly after being reddited, I decided to take some action to prevent the same thing happening again.

My blog is currently powered by Wordpress, so I enabled the wp-cache plugin that I had installed some time ago. It is however, one of the few PHP-powered things on my server. Then I decided to migrate to lighttpd, because it uses less memory than Apache, and because I like changing things on my server for fun. There were some hurdles, but I got there in the end. Here's what I'd do now if I had to do the migration again. They are not necessarily the best settings for all sites. I chose these settings as each site is small, and I don't have much CPU or RAM to play with.

I use vim as my text editor. If you prefer something else (nano, emacs etc), then obviously substitute where necessary:

Install lighttpd - apt-get install lighttpd . Then edit /etc/lighttpd/lighttpd.conf and set the port to 81. This will let us test lighttpd whilst Apache is running. Run apt-get install lighttpd again and it will finish successfully this time.

Enable lighttpd modules - I need fastcgi for PHP, Django and CherryPy. Enable it like so:
cd /etc/lighttpd/conf-enabled ln -s ../conf-available/10-fastcgi.conf 10-fastcgi.conf aptitude install php5-cgi
The Debian release has support for PHP4 by default. Change this to PHP5:
sed s/php4/php5/g 10-fast-cgi.conf
I also wanted ssl support:
ln -s ../conf-available/10-ssl.conf ssl.conf vim ssl.conf
Specify your pem file location. If you have only a key and crt, cat them together into a .pem. Note that you cannot test lighttpd with Apache running if you have the SSL module enabled, as they'll compete for port 443

Specify vhosts - You probably host multiple websites, and want the equivalent of Apache virtualhosts:
I have one file for web framework-powered sites, and one for sites that serve PHP, but its up to you:
vim vhosts.conf
This specifies a vhost's URL with a regex. In this case its middlerasenchurch.org.uk or any subdomain of it.
```
$HTTP["host"] =~ "(^|\.)middlerasenchurch\.org\.uk" {
        server.document-root = "/var/www/middlerasenchurch.org.uk"
}
```

Rewrite .htaccess files - lighttpd doesn't understand Apache's .htaccess files, but you can specify rewrites if the rewrite module is enabled (uncommented) in /etc/lighttpd/lighttpd.conf. If you need it, go and uncomment it. These rules make Wordpress work on my site:

$HTTP["host"] =~ "(^|www\.)maniacmartin\.com" {
        server.document-root = "/var/www/maniacmartin.com",
        alias.url = (
                "/svn" => "/usr/share/websvn/"
        )
        url.rewrite = (
        "^/?$" => "/blog/index.php",
        "^/blog/(wp-.+)$" => "$0",
        "^/blog/xmlrpc.php" => "$0",
        "^/blog/sitemap.xml" => "$0",
        "^/blog/(.+)/?$" => "/blog/index.php/$1"
        )
}

Make lighttpd spawn CherryPy - I used fastcgi to link lighttpd and CherryPy. First, I added this code to my python project, making it serve on fastcgi if it got passed the "--fastcgi" switch:

if "--fastcgi" in sys.argv:
	app = cherrypy.tree.mount(WebPWMan())
	# CherryPy autoreload must be disabled for the flup server to work
	cherrypy.config.update({'engine.autoreload_on':False})
	cherrypy.config.update({
			"tools.sessions.on": True,
			"tools.sessions.timeout": 5,
			"log.screen": False,
			"log.access_file": "/tmp/cherry_access.log",
			"log.error_file": "/tmp/cherry_error.log",
	})
	from flup.server.fcgi import WSGIServer
	cherrypy.config.update({'engine.autoreload_on':False})
	WSGIServer(app).run()
	
else:
	cherrypy.config.update({'server.socket_port':8051})
	cherrypy.config.update({"tools.sessions.on": True, "tools.sessions.timeout": 5})
	cherrypy.quickstart(WebPWMan())

Its important to redirect the logs away from stdout, as stdout is whipped away from CherryPy when you quit SSH, and the process will die. Also, logs should go to a folder that www-data can write to. (/tmp is probably not a good idea, but I don't read these logs anyway)

In a file in /etc/lighttpd/conf-enabled, add an entry like this:

$HTTP["host"] =~ "webpwman\.maniacmartin\.com" {
  fastcgi.server = (
    "/" => (
      "script.fcgi" => (
        "bin-path" => "/home/martin/webpwman/webpwman.py --fastcgi",
        "socket"          => "/tmp/webpwman.sock",
        "check-local"     => "disable",
        "disable-time"    => 1,
        "min-procs"       => 1,
        "max-procs"       => 1, # adjust as needed
      ),
    ),
  )
}

Configure Django in lighttpd - I run Django as seperate processes, and only pass it dynamic URLs, as lighttpd is much faster at serving static files. You could also do this with CherryPy, but my CherryPy site (webpwman) only has one static file, so it wasn't worth it.
My confs look something like:

$HTTP["host"] =~ "(^|\.)littledoves\.co\.uk" {
        server.document-root = "/var/www/littledoves.co.uk"
        fastcgi.server = (
                "/littledoves.fcgi" => (
                        "main" => (
                                # Use host / port instead of socket for TCP fastcgi
                                # "host" => "127.0.0.1",
                                # "port" => 3033,
                                "socket" => "/tmp/littledoves.sock",
                                "check-local" => "disable",
                        )
                ),
        )
        alias.url = (
                "/media/" => "/usr/local/django/django_src/django/contrib/admin/media/",
                # Change to match where you put Django
        )

url.rewrite-once = (
                "^(/media.*)$" => "$1",
                "^(/static.*)$" => "$1",
                "^/favicon\.ico$" => "/static/favicon.ico",
                "^(/.*)$" => "/littledoves.fcgi$1",
        )
}

Make an init script for your Django projects
I started out with the script at http://code.djangoproject.com/wiki/InitdScriptForLinux, saved in /etc/init.d/django-fastcgi
I changed the central part to:
```
start-stop-daemon --start --chuid $RUN_AS --pidfile $RUNFILES_PATH/$SITE.pid --startas /usr/bin/python $SITES_PATH/$SITE/manage.py runfcgi method=threaded socket=$RUNFILES_PATH/$SITE.sock pidfile=$RUNFILES_PATH/$SITE.pid
                       pidfile=$RUNFILES_PATH/$SITE.pid
            chmod 400 $RUNFILES_PATH/$SITE.pid
```
and changed the parameters at the top to:
#### SERVER SPECIFIC CONFIGURATION DJANGO_SITES="lejogger littledoves" SITES_PATH=/usr/local/django/ RUNFILES_PATH=/tmp RUN_AS=www-data #### DO NOT CHANGE ANYTHING AFTER THIS LINE!

This makes it use sockets, not TCP/IP ports. I used /tmp, not /var/run, as it needs to be a folder that www-data (or whatever user you run your Django instances as), can write to.

To make the init script run on startup:
chmod +x /etc/init.d/django-fastcgi update-rc.d django-fastcgi

Test everything - I can't stress this enough. You know what to do
Dont forget to restart lighttpd though:
/etc/init.d/lighttpd restart
While you're at it, you might want to specify different log files per virtual host too.

Go live - First we need to disable Apache from starting on boot:
cd /etc/rc2.d mv S91apache2 K09apache2

Change lighttpd to port 80 and go:
/etc/init.d/lighttpd stop vim /etc/lighttpd/lighttpd.conf /etc/init.d/apache2 stop; /etc/init.d/lighttpd reload

Its all going fine here, apart from that I've lost mod_svn, which lighttpd doesn't have. I've switched to svn+ssh, and plan to add some features to the PHP-based websvn and deploy it soon.

Posted 8th August 2008 in Computer, with 6 comments