[ZODB-Dev] Preliminary notes from fixing a bad data.fs

[Tim Peters]

[ethan fremen]
>> Right now, not being able to pack is a dangerous behaviour.

Dangerous how?  I'm assuming that the only reasons for a pack failing are
because the input file is damaged, or you don't have enough disk space to
write the packed file.  Seems to me you were in trouble either way before
you even thought about packing, and that it wouldn't serve a good end to try
to hide either condition.

>> I don't want to correct one problem by causing a worse one, but it
>> seems to me like some sort of logging of what had to be done to make
>> the pack succeed would be appropriate in this circumstance. The old
>> Data.fs is still around after a pack, so the user can dig through it
>> for the broken object.

I'm afraid it's more likely that most people wouldn't notice log msgs or
damage until a dozen packs later, and then they'd post here <wink> with a
semantically insane Data.fs which fstest.py and fsrefs.py nevertheless like
fine.  Then there's no clue at all remaining.

I have to note that most (all?) people who have posted here with problems
weren't following good practice to begin with:  they didn't have backups,
and didn't run the diagnostic tools until after their problems became
impossible to ignore in normal operation (I think I know where the damaged
.fs you've been looking at came from, and if so that's no exception).  I
have to expect they're the least likely to proactively look for warnings in
log messages too.

[Jeremy Hylton]
> I think the pack authors took the attitude that packing bogus data is
> more dangerous than not packing.  Not packing does not destroy any data.
> Packing a damaged data.fs can lead to arbitrary data loss.  In the limit,
> if the root object is unpicklable, pack deletes everything.

You say that like it's a bad thing <wink>.

> The old version of the data.fs is still around, but not for long if
> you've got a regularly scheduled pack.
>
> If pack fails, you're supposed to fix the data.fs some other way and then
> run pack.  I think this fails because there may not be any way to fix the
> data.fs so that it will pack.

No mechanical way.  fsrecover.py applies sometimes-Draconian measures to
suppress complaints from fstest.py, but there's nothing akin to fsrecover
for suppressing complaints from fsrefs.py.  Ethan has an interesting
approximation to the latter, but I'm not sure it should be conflated with
packing.

> Perhaps there should be a pack option to throw caution to the wind and
> pack no matter what, but it shouldn't be the default.

Definitely not the default.  I think I'd be happiest with a new, distinct
recovery tool, acting in relation to fsrefs as fsrecover stands to fstest:
shut up fsrefs complaints even if it has to throw away the entire database
to do so (which latter may in fact be necessary).