Re: Database backup and deduplication question

[Colin Percival <cperciva@tarsnap.com>]

Hi all,

Sorry about the long silence... between Christmas and New Years and
FreeBSD 9.0-RELEASE, I've fallen a bit behind on email lately. :-(

On 12/29/11 20:02, Greg Larkin wrote:
>> I ran those tests as shown, and each one worked as expected with a 2:1
>> ratio.  Then I ran a few more sequences, in case they help:
>>
>> sh-3.2# tarsnap --dry-run -cvf testarchive file1 file1 part1
>> New data                                105008116        105526431
>> sh-3.2# tarsnap --dry-run -cvf testarchive file1 file1 part1 part2
>> New data                                129278071        129911860
>> sh-3.2# tarsnap --dry-run -cvf testarchive file1 file1 part1 part2 part3
>> New data                                138220066        138902042
>>
>> The first sequence looks ok (file1 file1 part1), but after that, the
>> "New data" number increases more than expected.  I have another Mac, and
>> I'll try the same tests to see if there's any machine-specific issue here.
> 
> I just ran the same tests with the same files on a different Mac running
> 10.5.8 instead of 10.6.8.  This time, the results are as expected:
> 
> sh-3.2# tarsnap --dry-run -cvf testarchive file1 file1 part1
> New data                                105098024        105606395
> sh-3.2# tarsnap --dry-run -cvf testarchive file1 file1 part1 part2
> New data                                105244723        105751742
> sh-3.2# tarsnap --dry-run -cvf testarchive file1 file1 part1 part2 part3
> New data                                105331285        105837271
> 
> If you have any ideas about debug messages or other instrumentation that I
> can put in the source code, let me know.  I'll compare the shell
> environment between the two machines to see if that reveals anything
> interesting.

Very interesting!  Were these two machines using the same keys?  If not, can
you have them swap keys and try again to see if it's the machine or the keys
which determine the behaviour?

Once I know that I should be able to provide you with a debugging patch which
will help track down the issue.

-- 
Colin Percival
Security Officer, FreeBSD | freebsd.org | The power to serve
Founder / author, Tarsnap | tarsnap.com | Online backups for the truly paranoid