[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Delete archive time



On 04/18/12 12:13, Michael Stevens wrote:
> The machine I was particularly complaining about has taken almost
> exactly 48 hours to delete around 300 archives (one failed due to
> network problems my end).

Ok, 48 hours is definitely well into the annoying range...

> I had one delete I wanted to do left over, which I've timed:
> 
> root@osaka:~# time tarsnap -d -f osaka-home-2011-05-18
>                                        Total size  Compressed size
> All archives                         926836591828     596496483543
>   (unique data)                       96754663741      68973013309
> This archive                         148593871412      93015468246
> Deleted data                           8215346246       6582996263
> 
> real	14m37.343s
> user	0m42.731s
> sys	0m4.344s
>
> Network connection is roughly 10mbit down/1mbit up ADSL line.

... on the other hand, you do realize that you're deleting archives at a
rate of almost 170 MB per second, right? :-)

> In my case, the reason I'm deleting so much is that I have no automated
> way of cleaning up old incremental backups, and they're cheap, so I've
> let them build up, and am finding cleaning things up is slower than I
> expected.

Right.  And if the bits I'm working on right now turn out well this sort
of "periodic cleanup" should be a good way of doing things, since deleting
many archives at once will end up being faster than deleting them one by
one.

> I'm wondering if a cheap fix for this would be to provide some higher
> level operations (possibly as a wrapper), like "archive daily and keep
> last n days" (<insert attempt to get colin to do all the work here>).

That's not possible -- when you create a new archive, the Tarsnap server
has no way of knowing (nor *should* it be able to know, as a matter of
security) which old blocks your new archive is using.

-- 
Colin Percival
Security Officer, FreeBSD | freebsd.org | The power to serve
Founder / author, Tarsnap | tarsnap.com | Online backups for the truly paranoid