Re: Database backup and deduplication question

[Greg Larkin <glarkin@sourcehosting.net>]

On 1/8/12 6:05 PM, Colin Percival wrote:
> Hi all,
>
> Sorry about the long silence... between Christmas and New Years and
> FreeBSD 9.0-RELEASE, I've fallen a bit behind on email lately. :-(
>
> On 12/29/11 20:02, Greg Larkin wrote:
>>> I ran those tests as shown, and each one worked as expected with a 2:1
>>> ratio.  Then I ran a few more sequences, in case they help:
>>>
>>> [...]
>>>
>>> The first sequence looks ok (file1 file1 part1), but after that, the
>>> "New data" number increases more than expected.  I have another Mac, and
>>> I'll try the same tests to see if there's any machine-specific issue here.
>> I just ran the same tests with the same files on a different Mac running
>> 10.5.8 instead of 10.6.8.  This time, the results are as expected:
>>
>> [...]
>>
>> If you have any ideas about debug messages or other instrumentation that I
>> can put in the source code, let me know.  I'll compare the shell
>> environment between the two machines to see if that reveals anything
>> interesting.
> Very interesting!  Were these two machines using the same keys?  If not, can
> you have them swap keys and try again to see if it's the machine or the keys
> which determine the behaviour?
>
> Once I know that I should be able to provide you with a debugging patch which
> will help track down the issue.
>
Hi Colin,

All machines have been using unique keys up until now.  Following your
instructions, I swapped keys between the Leopard (Mac Book Pro) and Snow
Leopard (iMac) machines, and for good measure, I installed tarsnap on a
Lion machine (Mac mini) and tried both sets of keys there.  At this
point, it appears to be a machine-dependent problem:


# iMac using Mac Book Pro keys (tarsnap 1.0.31):

sh-3.2# tarsnap --keyfile mbp-keys/tarsnap.key --dry-run -cf testarchive
file1 file1 part1 2>&1 | grep "^New data"
New data                                105043884        105559008
sh-3.2# tarsnap --keyfile mbp-keys/tarsnap.key --dry-run -cf testarchive
file1 file1 part1 part2 2>&1 | grep "^New data"
New data                                157504658        158272905
sh-3.2# tarsnap --keyfile mbp-keys/tarsnap.key --dry-run -cf testarchive
file1 file1 part1 part2 part3 2>&1 | grep "^New data"
New data                                183736112        184636668


# Mac Book Pro using iMac keys (tarsnap 1.0.28):

sh-3.2# tarsnap --keyfile imac-keys/tarsnap.key --dry-run -cf
testarchive file1 file1 part1 2>&1 | grep "^New data"
New data                                105083384        105589862
sh-3.2# tarsnap --keyfile imac-keys/tarsnap.key --dry-run -cf
testarchive file1 file1 part1 part2 2>&1 | grep "^New data"
New data                                105191397        105696541
sh-3.2# tarsnap --keyfile imac-keys/tarsnap.key --dry-run -cf
testarchive file1 file1 part1 part2 part 3 2>&1 | grep "^New data"
New data                                105191404        105696548


# Mac mini using both sets of keys (tarsnap 1.0.28):

## Using MBP keys
sh-3.2# tarsnap --keyfile mbp-keys/tarsnap.key --dry-run -cf testarchive
file1 file1 part1 2>&1 | grep "^New data"
New data                                105098054        105606433
sh-3.2# tarsnap --keyfile mbp-keys/tarsnap.key --dry-run -cf testarchive
file1 file1 part1 part2 2>&1 | grep "^New data"
New data                                105244753        105751780
sh-3.2# tarsnap --keyfile mbp-keys/tarsnap.key --dry-run -cf testarchive
file1 file1 part1 part2 part3 2>&1 | grep "^New data"
New data                                105331315        105837309

## Using iMac keys
sh-3.2# tarsnap --keyfile imac-keys/tarsnap.key --dry-run -cf
testarchive file1 file1 part1 2>&1 | grep "^New data"
New data                                105083384        105589862
sh-3.2# tarsnap --keyfile imac-keys/tarsnap.key --dry-run -cf
testarchive file1 file1 part1 part2 2>&1 | grep "^New data"
New data                                105191397        105696541
sh-3.2# tarsnap --keyfile imac-keys/tarsnap.key --dry-run -cf
testarchive file1 file1 part1 part2 part3 2>&1 | grep "^New data"
New data                                105279204        105783302


You might think that based on the results above, tarsnap 1.0.31 is the
cause of the problem, since it's the obvious difference between the
three machines.  In order to remove that variable, I built 1.0.28
(current version from MacPorts) on the iMac and retested.  That forced a
rebuild of OpenSSL to bring it up to version 1.0.0f.  Previously, I had
1.0.0e installed on the iMac.  The results are interesting:

sh-3.2# tarsnap --keyfile mbp-keys/tarsnap.key --dry-run -cf testarchive
file1 file1 part1 2>&1 | grep "^New data"
New data                                105098054        105606442
sh-3.2# tarsnap --keyfile mbp-keys/tarsnap.key --dry-run -cf testarchive
file1 file1 part1 part2 2>&1 | grep "^New data"
New data                                105244753        105751786
sh-3.2# tarsnap --keyfile mbp-keys/tarsnap.key --dry-run -cf testarchive
file1 file1 part1 part2 part3 2>&1 | grep "^New data"
New data                                105331315        105837316

These numbers are what I expect from tarsnap, so is it possible that
some bug in OpenSSL triggered the original deduplication problem that I
noticed? 

Thank you,
Greg