maniacmartin

Don't forget about the hardware

Don't forget to consider the difference hardware can make to a systemSomewhat recently I built a new backups server at work as a proof of concept. Since it was only a proof of concept I just used the best machine available - a new Celeron desktop with 4 onboard SATA ports which I could use in software RAID. Pretty bad for a so-called server, but sufficient, I assumed, to keep up with 3 bonded 7-Mbit ADSL lines when running rdiff-backup. The proof of concept worked - the backups completed overnight when our bandwidth was cheap. Gradually, more backups were migrated to this server and the old backup server was retired. Then I noticed that backups were not finishing by 9am and costing us a considerable amount of money. I initially put this down to rdiff-backup and kicked myself for not using rsnapshot, a clever wrapper to the proven rsync binary. Closer examination, however, revealed that 80% of CPU time was allocated to disk I/O. iftop showed that the server was only pulling 2Mbit during backups. As a test, I bought a £35 4-port PCI SATA card and put it in the old backup server, an aging Pentium 4 with hyperthreading. I pulled the disks from the Celeron server and put 2 on the motherboard's only 2 SATA ports, and 2 on the PCI card. Amazingly, we then achieved a reduction of two thirds in the amount of time backups took to complete. It seems that the Northbridge in the new Celeron workstation actually couldn't keep up with 3Mbit of connectivity! Whilst I admit that rdiff-backup and rsync access the local files very heavily due to the remote binary diff algorithm, this result really surprised me. I assumed that local disk access will not be a performance issue for a task like this, but it evidently seems that this is not the case. No matter how low priority a server is, be sure to benchmark and get the right hardware for the job, because it'll only come back to haunt you later!

Apticron equivalent for Redhat RHEL

I love the Debian apitcron package, which emails you when there are updates to be installed. But what is the RHEL equivalent?For RHEL5 and above, this is really easy, just add a call to yum check-update -q in your crontab. This command will print a list of packages to be installed, and will return silently otherwise. For older versions of RHEL, which use up2date, you can use the following Python script (will need adapting not to use subprocess if you still use Python 2.3): http://wiki.maniacmartin.com/up2date This gets around the lack of a quiet parameter on up2date Simply change the top like so for Python 2.3: import os listing = os.popen("up2date -l", 'r').read()

I'm IPv6 ready

Why I've IPv6 enabled my server and how I did itWe hear the horror stories every now and then. We're running out of IPv4 space and soon the world as we know it is going to end. One day we'll run out of addresses and computers will have to live behind the evil jail of NAT. There is another way, and that is of course IPv6. ISPs are mostly burying their heads in the sand, and won't implement it until there are more IPv6 enabled websites. Webmasters are waiting on IPv6 addresses to be handed dished out by hosting companies who are too lazy to do anything until ISPs dish out addresses to surfers. A vicious cycle indeed. Luckily, we host with Bytemark, who dish out native IPv6 addresses so we don't have to rely on hackish Sixxs tunnels. I have a /56 on my VPS. The first thing to do was to to bring up 2 addresses in /etc/network/interfaces. I tried putting a separate ipv6 stanza (the way you're supposed to do it), so I used post-up commands by placing this under iface ethe0 inet static


  #Bring up additional ipv6 addresses on same if
  up ip -6 addr add 2001:41c8:10a:200::1/56 dev eth0
  up ip -6 ro add default via fe80::1 dev eth0
  up ip -6 addr add 2001:41c8:10a:200::2/56 dev eth0

I saved myself a reboot by also running the up commands on the command line. Just as I did for v4, I planned to use the first address for everything, except lighttpd, because Apache is already using port 80. I added an extra Listen line to /etc/apache2/ports.conf

Listen [2001:41c8:10a:200::1]:80

All of my vhosts listen on *:80, so that's good. Lighttpd's server.bind syntax only supports one bind address and port, but you can bind to multiple addresses like so:

server.use-ipv6 = "enable"
server.bind = "::ffff:212.110.165.233"
$SERVER["socket"] == "[2001:41c8:10a:200::2]:80" { }

Next I turned on ejabberd's IPv6 support by adding the inet6 keyword to thgis stanza in /etc/ejabberd/ejabberd.cfg:

{listen,
[
{5222, ejabberd_c2s, [inet6, {access, c2s}, {shaper, c2s_shaper}]},
...
]}

Next up wasmy IRC bouncer, ZNC. I'd told it to explicitly bind to a certain IP address so I could have a vanity address. That needed disabling so I can connect to IPv6-only IRC servers (which to be honest isn't going to happen anytime soon.) Postfix has IPv6 support since 2.2, and i have 2.5.5 so that should just work, as it currently binds to all addresses. For good measure, i added inet_protocols=all to /etc/postfix/main.cf For Dovecot, I added listen = [::] to /etc/dovecot/dovecot.conf. Note that listen = * refers to all IPv4 only. Bytemark's hosted TinyDNS servers support IPv6 records (prefix 6 for automatic rDNS, prefix 3 otherwise), but I stupidly totally forgot about this and used this generator to cook up some AAAA records to match my A records. A little testing with the SixXS IPv6-IPv4 and IPv4-IPv6 Website Gateway, which is basically an IPv6 -> IPv4 web proxy that refuses to retrieve anything hosted on IPv4, and I confirmed everything was good to. Stop Press! Aren't we forgetting something? In keeping with the tradition set by kame, and followed by Google and many others, I needed a bouncing logo thats only shown to surfers that connect via IPv4. Lucky I had an animated gif that I'd made earlier. In Django, you can do something like

":" in request.META["REMOTE_ADDR"]

to work out if your surfer is an IPv6 surfer. IPv4 users can sneak a peak at using the SIXXS gateway See it's that easy. If your host gives you IPv6 space, then you have no excuse not to be leading the way to the move to adopting IPv6.

Hotmail Spam

Or "why your mail from hotmail may not reach me"Lately I've been having a major problem with spam sent from hotmail addresses using hotmail's own SMTP servers. This has evaded sanesecurity's signatures and SpamAssassin, even though I've fed about 100 of these messages to sa-learn and also given some to steveb at sanesecurity. Some of these new spams I've been getting even have a negative SpamAssassin score! They are mostly random waffle (not 411, viagra or sex), with a hyperlink embedded. My email address is never in the To: or Cc: fields.. I have tried contacting Hotmail in the past about issues such as this, and its like talking to a brick wall. Previously, the most obvious spam was rejected by our SMTP servers at SMTP-time, and that which was likely to be spam, but not certain, was delivered to a Junk folder. Now, I am going to configure my server to accept then silently drop all email which is routed via a hotmail.com SMTP server where my name is not in the To: or Cc: fields. (Technical issues with the way the SMTP servers are setup prevent me from rejecting it at SMTP-time at present.) This means that if you use Windows Live mail to Bcc me an email then I will no longer receive it. You will not receive an automated reply, in the interests of reducing backscatter.

Adobe Air UPS Tracking

Track your parcels from the comfort of your linux desktop!Today I noticed that UPS have developed a widget that allows Windows and Mac users to track their parcels. This looked useful, as I would no longer have to sit refreshing a web page to satisfy my curiosity as to where my parcels I send are, but just wait for pop-up notifications from an application that sits in my system tray. First you'll need Adobe Air. Adobe's Linux installer doesn't perform all of the steps required for 64-bit systems, and their guide is a bit lengthy, so I've transcribed it into a shell script which you can download: install-adobe-air-ubuntu-64bit.sh The UPS site claims that Linux support is "coming soon", and they seem to do some user agent sniffing with Flash, so I downloaded it in a Windows and placed a copy here The first thing to do once you've made an account is to click Options at the bottom and untick "Show Widget Character", "Show News Ticker" and "Show News Flashes" to make these annoying resource-hungry bits go away. Now you're left with the bare window. Alas it has a huge black surround (which is transparent on Windows and Mac), it and consumes a fair amount of CPU power just idling, but it does appear to work. You can even minimize it to the system tray. Hopefully the Linux version is indeed coming soon, and will address these issues. At least UPS recognize that use Linux users exist, and chose a cross-platform technology. Now I can send my parcels from the comfort of my workplace, with a guarantee they'll be picked up between 10am and 5.30pm next day (when I'm at work; DHL/HDNL through Parcels2Go can't guarantee this timeframe) without having to take time to visit the post office, I can track and get progress alerts from this widget, and if the parcel is over 2kg, its cheaper than Royal Mail anyway, when booked through Interparcel. Despite the Economy service being advertised as non-guaranteed next-day, all of the parcels I have sent have arrived next day. What could be better than that?

Cycle lanes on pavements

Why I almost never use cycle lanes painted on pavementsYou know the cycle lanes I'm referring to. A white line painted down the middle of the pavement, so that cyclists and pedestrians can share it. Some of them are quite good, but the inner-city ones are generally a waste of time The main problem is that these lanes give way to every side road, so you have to slow down and look over your shoulder at every side road. It is also annoying if traffic is waiting to pull out the side road. Other annoyances are limited space to overtake slow cyclists or pedestrians spilling over from their side of the pavement, and a bumpier road surface than the main carriageway, tight corners and "Cyclists Dismount" signs. Luckily, in the UK "use of these facilities is not compulsory and will depend on your experience and skills" (The Highway Code for cyclists (61)), so you don't have to use them. They are sometimes useful as a legitimate way to bypass red traffic lights though. I recommend that you instead use bus lanes instead. "Most bus lanes may be used by cyclists as indicated on signs ” (65), so if there is a picture of a bicycle on the blue sign at the start of the bus lane, you can use it, and it'll be a smoother, faster ride.

Verbal Nagios Alerts

Have verbal nagios alerts with festivalHere's a fragile script I cooked up to do it Verbal Nagios

Coloured Shell Prompts

A script to automatically generate the shell prompt colour from the username and hostnameI like to have a different coloured shell prompt for every PC, so I don't end up executing commands on the wrong machine. Previously, I have edited .bashrc manually on every new machine, but it would be nice if this was done automatically when I check out my dot files from the repository onto the machine. If you make the following changes to your .bashrc, the colour and font-weight of the shell prompt will be chosen from a hash of your current username and the machines' hostname. My terminals have a black background. If yours have white, you might want to remove the white/yellow options and add black and brown instead, so the text shows up. Find if [ "$color_prompt" = yes ]; then change the next line to PS1='${debian_chroot:+($debian_chroot)}\[\033[`color_from_hostname`m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ ' Then, at the end of the file add: function color_from_hostname { hash=`(echo $USER; hostname) | md5sum | awk '{print $1}'` case ${hash#${hash%?}} in "0") c="1;34" ;; # Light blue "1") c="0;34" ;; # Dark blue "2") c="1;32" ;; # Light green "3") c="0;32" ;; # Dark green "4") c="1;36" ;; # Light cyan "5") c="0;36" ;; # Dark cyan "6") c="1;31" ;; # Light red "7") c="0;31" ;; # Dark red "8") c="1;35" ;; # Light purple "9") c="0;35" ;; # Dark purple "a") c="1;33" ;; # Light yellow "b") c="0;33" ;; # Dark yellow "c") c="1;37" ;; # Light white "d") c="0;37" ;; # Dark white # Run out of colours. Lets have some repeats "e") c="0;34" ;; "f") c="1;32" ;; esac echo $c }

Checking apache conf syntax

Finally, Debian now does this when you /etc/init.d/apache2 reload, rather than silently failing and killing your websites.For a long time, I've been meaning to make a patch to the force-reload part of /etc/init.d/apache2, so that it performs an apache2 -t, to test if the new configuration files have valid syntax, and aborts the reload if there is an error, rather than silently killing your websites dead in their tracks. However, it seems that Debian or upstream have beaten me to it, as my init file (2.2.9-10+lenny2) seems to have this feature. However, I don't seem to test the syntax of a conf file without reloading Apache. I get this error when running apache2 -t: apache2: bad user name ${APACHE_RUN_USER} The following command does the right thing though: /usr/sbin/apache2ctl configtest

Backupninja from Debian to Ubuntu

Some hints about using Backupninja to backup Debian servers to an Ubuntu boxAndrew and I currently use Backupninja for backing up our servers and laptops to our Ubuntu PVR, because this is the only machine at home thats guaranteed to be on 24-7. Backupninja was recommended to us by Michael and David. It runs on the servers that you want to back up, and uses a conf.d directory to store tasks, such including pgsql dumps which save to /var/backups, and then there is a final task which executes rdiff-backup to push over the files. As far as I can tell, rdiff-backup uses a combination of diffs, symlinks and rsync to have a snapshot history yet only use the bandwidth and disk space that a normal rsync would use. Backupninja insists that the version of rdiff-backup is the same at both ends, and although you can override it with the ignore-version conf file entry, I'd rather not as chaos and unrecoverable backups would probably follow. Since we had multiple Debian servers to be backed up, and only one one Ubuntu recipient, we chose to leave the Ubuntu boxes with the stock lenny version of rdiff-backup and change the Ubuntu version to match. (Also, rdiff-backup is not in lenny-backports) At the time of writing, Debian Lenny had v1.2.5 and Ubuntu Jaunty v1.2.7. We installed rdiff-backup_1.2.5-1build1_i386.deb on the Jaunty box but when the cron ran, got this lovely error: Fatal: rdiff-backup does not have the same version at the source and at the destination. After was not actually caused by the rdiff-backups having different versions, but by Ubuntu Bug 345086. Until v1.2.7, Ubuntu packaged rdiff-backup wrong. It installs to /usr/local/bin, not to /usr/bin as on Debian. As a result the Debian servers couldn't find the binary when connecting over ssh. I could have rebuilt the v1.2.5 deb from source so that it installs to /usr/bin, but in the end I settled on ln -s /usr/local/bin/rdiff-backup /usr/bin/rdiff-backup I've also filed a bug against Backupninja, with a patch so that it checks ssh's exit value and gives a more appropriate error if it can't find rdiff-backup in the right place on the remote server.