Backing Up

Since a close call with the near loss of my laptop, I have decided that after fifteen years of not backing anything up it was time to begin doing so. I am in the fortunate position of having access to a co-location box and a friends system to do a tertiary backup to, however you might want to adjust the following instructions to suit the resources that you have available to you.

The general idea for my backup strategy is:

use backupninja on the 'clients' to dump regularly (system wide daily whilst '/home' hourly for example) the data wanted to '/var/backups/backupninja/'
nightly duplicate '/var/backups/backupninja/' to my remote co-location box
nightly dupe the backup data on my co-location box to my friends box via FTP

Client Configuration

Backup of System

I recommend you LVM the backupninja directory ('/var/backups/backupninja') so that you can grow it later when need be:

# lvcreate -L 2G -n backupninja lvm-berk
# mkfs.xfs -L backupninja /dev/lvm-berk/backupninja
# mkdir /var/backups/backupninja
# # amend, but you get the idea
# echo "LABEL=backupninja /var/backups/backupninja xfs relatime,nodev,nosuid,noexec 0 5" >> /etc/fstab
# mount /var/backups/backupninja

When you are nearly out of space on there, you just need to type (no need to go into single user mode, reboot or umount '/var/backups/backupninja'):

# lvextend -L +1G /dev/lvm-berk/backupninja
# xfs_growfs /var/backups/backupninja

Install the following packages:

backupninja
rdiff-backup
[optional] python-pylibacl python-pyxattr python-xattr (if you use POSIX ACL's)

I make a few amendments to '/etc/backupninja.conf':

reportsuccess = no
admingroup = staff - as I make administrators on my laptop a member of the 'staff' group
usecolours = no

As root, copy and amend a template 'rdiff' like so:

# cp /usr/share/doc/backupninja/examples/example.rdiff /etc/backup.d/90-snapshot.rdiff
# [snipped some file editing]
# grep "^[^#].*" /etc/backup.d/90-snapshot.rdiff 
  when = daily
  options = --exclude-special-files --exclude-if-present=.no-kung-fu
  nicelevel = 15
  [source]
  label = berk.digriz.org.uk
  type = local
  keep = 60
  include = /var/spool/cron/crontabs
  include = /var/backups
  include = /etc
  include = /root
  include = /home
  include = /usr/local/bin
  include = /usr/local/sbin
  include = /var/lib/dpkg/status
  include = /var/lib/dpkg/status-old
  exclude = /home/*/.gnupg
  exclude = /var/backups/backupninja
  [dest]
  type = local
  directory = /var/backups/backupninja
# chmod 640 /etc/backup.d/*

The 'options' line makes sure you do not copy sockets/devices or fifo's unnecessarily and it gives us the ability to avoid wandering off down extra potentially large filesystems (when you use LVM you find yourself creating lots of mini-bitbuckets and project spaces where you need them) by creating a file at the top of the mountpoint that is not to be backed up. Just simply type:

$ touch /path/to/dir/.no-kung-fu

To test if everything is working fine, all you have to do is type:

# backupninja -d --run /etc/backup.d/10-action.rdiff

You should see in detail what your backup process will do daily.

Tweak for a Laptop Client

As I am running backupninja on a laptop, it would be crazy to let it use cron and more so have cron fire up once a minute just to see if something needs to happen. If you are using a laptop I would recommend you remove cron altogether and instead opt for fcron.

Once installed you need to add a hook so that fcron will run backupninja once a day. Run 'fcrontab -m systab' and amend the file so it looks like:

# fcrontab -l systab
22:36:34 listing systab's fcrontab
!bootrun(true),nice(15),serial(true)
& 03 03  * * *   run-parts --report /etc/cron.daily
& 17 03  * * 7   run-parts --report /etc/cron.weekly
& 02 04  1 * *   run-parts --report /etc/cron.monthly

# run hourly backupninja
@lavg15(0.5),mail(no) 1d /usr/sbin/backupninja -n
@lavg5(0.2),mail(no) 1h /usr/sbin/backupninja

Upload of Backup to Co-location Boxen

Now we need to look into exporting the backup to our secondary system, my co-location box. These instructions assume you trust your co-location box and do not need to encrypt the data.

# [snipped some file editing]
# cat /etc/backup.d/99-export.sh
  when(){ :; }
  when = hourly

  NICE=15
   SRC=/var/backups/backupninja

  DSTHOST=woodchuck.wormnet.eu
  DSTUSER=backup
      DST=/var/backups/wormnet/alex/berk.digriz.org.uk

  renice $NICE -g $$

  rsync --partial-dir=.rsync-partial --delay-updates -ar -e "/usr/bin/ssh -l $DSTUSER -i '$SRC/id_rsa'" "$SRC/dup/" "$DSTHOST:$DST"
# chmod 640 /etc/backup.d/*
# ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): /var/backups/backupninja/id_rsa
Enter passphrase (empty for no passphrase): [BLANK]
Enter same passphrase again: [BLANK]
Your identification has been saved in /var/backups/backupninja/id_rsa.
Your public key has been saved in /var/backups/backupninja/id_rsa.pub.
The key fingerprint is:
f6:a2:2f:5b:4e:41:e8:43:55:9e:57:d7:e4:61:ee:dc root@berk

You should now test the whole process to see if it is happy with:

# backupninja -d -n -t

Configuring the Co-location Boxen

It again might be worth LVM'ing your backup area here too but I'll leave that as an exercise for the reader.

# mkdir /var/backups/wormnet
# chgrp /var/backups/wormnet
# chmod 770 /var/backups/wormnet
# mkdir /var/backups/.ssh
# echo "no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding [contents of berk:/var/backups/backupninja/id_dsa_duplicity.pub]" >> /var/backups/.ssh/authorized_keys

# mkdir /var/backups/wormnet/alex
# chown backup:backup /var/backups/wormnet/alex
# chmod 700 /var/backups/wormnet/alex

Now before you can forget about the whole thing, you need to run the process manually from the client host once so that the SSH host key is primed:

# backupninja -d -n
...snipped...
Debug: ssh -o IdentityFile=/var/backups/backupninja/id_dsa -o PasswordAuthentication=no woodchuck.wormnet.eu -l backup 'echo -n 1'
The authenticity of host 'woodchuck.wormnet.eu (77.75.105.223)' can't be established.
RSA key fingerprint is 22:fd:5e:08:b3:c2:93:58:28:08:52:42:d7:ec:cd:ef.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'woodchuck.wormnet.eu,77.75.105.223' (RSA) to the list of known hosts.
Debug: Connected to woodchuck.wormnet.eu as backup successfully
...snipped...

Hopefully now you are cooking with a full backup system.

Why The Shell Scripts

The underlying reason is as I want to be able to do backups on a unreliable (aka disconnecting as my workstation is a laptop) which means really I am looking for something that can handy upload resumptions. It looks like duplicity does not do this so instead I have to point duplicity to do it's 'thang' but to another local directory on the harddisk. This directory I then use 'rsync' with some fancy partial file resumption parameters to shift the data to it's destination.

To make this process more painful the duplicity handler supplied with backupninja (one day I will enhance it and submit a patch upstream) does not support writing out to the local filesystem so we need to use a script for this. The second shell script is because the rsync support for backupninja (or at least the Debian one) is a bit naff...or I just simply do not understand it.

If you have a 24/7 connection to your backup box then you should dump the two shell scripts ('95-encrypt.sh' and '99-export.sh') and instead just do something like:

# grep "^[^#].*" /etc/backup.d/99-export.dup
  when = daily
  nicelevel = 15
  [gpg]
  sign = no
  password = passwordwithnospaces
  [source]
  include = /var/backups/backupninja
  exclude = /var/backups/backupninja/id_rsa*
  [dest]
  incremental = yes
  keep = 7
  sshoptions = -o IdentityFile=/var/backups/backupninja/id_rsa
  destdir = /var/backups/wormnet/alex/berk.digriz.org.uk
  desthost = woodchuck.wormnet.eu
  destuser = backup

Potential Improvements

Sanity Checking

If particular steps fail, then horrible things are going to happen...fortunately you would get an email and everything is incremental anyway.

Lock Down Accounts Shell

It probably is a good idea to look at using rssh to make sure people only can use rsync and get some chroot action too.

Account-per-Host

As I'm lazy, I am just using a single user account for all the backup hosts. As the backups are encrypted it should keep people generally happy (well we are all friendly on our co-location box, don't know about you and your friends) and the only problem we potentially have is overwriting one anothers files which hopefully we are clever enough to avoid.

For those who want to don both their belts and braces I would suggest seperate accounts per host.

GnuPG Encryption/Signing

If you really want to be hardcore, again I could not be bothered, you can create a public key pair suitable for use by GnuPG so asymmetrically encrypt and sign your backups for a highly level of confidence. I'll personally kept to asymmetric action for emails and Jabber.

Tertiary Backup from Linux Co-location to Windows Co-location

TODO

Microsoft Windows Server Configuration

Follow the instructions on configuring an rsync daemon for Microsoft Windows using the rsync.zip and srvany.zip packages.

Once working, I would recommend you amend the 'rsyncd.conf' file to look something like:

use chroot = false
strict modes = false
hosts allow = *
auth users = *
secrets file = C:/Program Files/rsync/passwd
log file = C:/Program Files/rsync/rsyncd.log

[wormnet]
path = C:/rsync/
read only = no

Then you should create the password file 'C:/Program Files/rsync/passwd' and put into it:

alex:testpassword

This means to connect you now need to use the username 'alex' with the password 'testpassword' to use the backup area 'wormnet'. Further amendments might be to place 'hosts allow' and 'auth users' line in the resource sections so that only particular hosts and user combinations can access it.