Reading and deleting archives painfully slow

[Chris Webb <chris@arachsys.com>]

I take daily archives of a number of machines, and had about three hundred
snapshots under one of my keys which I wondered whether might be responsible
for some tarsnap performance problems I've been seeing. I deleted all but
the five most recent of them, which took a startlingly long time:

  # tarsnap --list-archives | tee list | wc -l
  Please enter passphrase for keyfile /home/root/.tarsnap/key:
  323
  # time tarsnap -d `grep ^alpha- list | sort | head -n -5 | sed 's/^/-f /'`
  Please enter passphrase for keyfile /home/root/.tarsnap/key:

  real    3993m10.403s
  user    388m33.273s
  sys     19m46.026s
  #

However, I'm still seeing other operations as very slow, e.g. extracting a
single small file from the archive

  # time tarsnap -x -f alpha-2011-05-30 home/chris/.bashrc
  Please enter passphrase for keyfile /home/root/.tarsnap/key: 

  real    55m22.255s
  user    1m1.806s
  sys     0m2.137s

is taking nearly an hour! This is true even now there are only five
snapshots of around a 200GB total system image, which is pretty tiny as
backups go. It's becoming very hard to rely on tarsnap for normal archiving
and backups when it's this slow. Is there anything I can do to make tarsnap
snappier, or diagnose why it performs so badly?

When I do a new system backup each night, it starts at 2am and finishes at
around 4:30am, which isn't unreasonable at all given it's scanning a 200GB
filesystem, so it's mostly reading that's the problem here I think. I'm
accessing from a datacentre in the UK and can get just under 100Mb/s to a
test EC2 instance in the US, so I don't think it's constrained bandwidth,
although maybe there's a higher rtt from here than most US users.

Cheers,

Chris.