[ZODB-Dev] ZODB packing error

[Tim Peters]

[Ollie Eisman, Mon, Jan 05, 2004 at 11:38:55AM]

Note the date on that.  This was a gmane post I found in the zodb-dev
moderator queue this morning.

>> I was testing a cron script to automatically pack the ZODB and
>> noticed that after a few "packs" that some chunks of my Plone 2.0
>> RC2 web site had vanished. Luckily I had a backup copy of Data.fs
>> but this data loss event concerns me a lot. Has anyone else had a
>> similar problem? I am using Zope 2.6.2.

[Marius Gedminas]
> I've had a similar problem.  I believe I also used Zope 2.6.2 at the
> time.
>
> When you pack your database with 0 day history (which is very
> helpfully the default setting in the ZMI), some objects are likely to
> disappear. This happens they were they are only referenced from a
> transaction that was not yet committed during the pack.  It is not
> specific to 0-day packs, but it is more likely in this case.

If that's a new object, it will be added to the database when the
transaction commits.  If it's an object that existed before the pack began,
and was modified since, then it's quite possible that older revisions of the
object vanish due to the pack.  Then when you try to commit the transaction,
and if the revision current when the transaction began was removed by
packing, you'll get a POSKeyError in the server log, and the client will see
a ConflictError.  If it retries the transaction as a result, the new-new
revision will get added to the database.

Is there some other failure mode here?

> I got the impression that this bug was fixed in a latter version, but
> I'm not sure which one.  I hope one of the ZODB gurus will clarify the
> situation.

The ZEO ClientStorage.pack() call does default to current time (always did,
still does).  I don't know what ZMI does.

We did fix one major bug of this kind "recently":  if you packed to a time
earlier than a time previously packed to, it was possible to lose data.
That's fixed in the current releases of ZODB.