[ZODB-Dev] several critical bug fixes for ZODB 3.1 / ZEO 2.0

[Jeremy Hylton]

Those of you watching the checkins list probably noticed that Tim and I
have been checking in a lot of bug fixes over the last 10 days.  One of
our customers had been experiencing problems with corrupted BTrees.  We
fixed several different bugs as a result of our investigation of the
problem.  Each of these bugs can cause inconsistent data to be written
to the database.  At least one of them can cause data loss, too.  Most
of the bugs were in ZEO.

The bugs are fixed in CVS on the ZODB3-3_1-branch and we'll do a release
first thing tomorrow morning.  Anyone using ZEO ought to upgrade to ZODB
3.1.2 ASAP.  The release will be a beta release, but has seen more
extensive internal testing than the typical beta.  I hope the final
release will follow shortly, although the Zope 2.6.2 release schedule
may have an impact.

I've included the contents of NEWS.txt below.

Jeremy

What's new in ZODB3 3.1.2 beta 2?
=================================

Fixed a critical race condition in ZEO's cache consistency that could
cause invalidations to be lost.  A lost invalidation can lead to data
loss or data corruption.  The specific bug could occur when one client
loaded an object at the same time another client committed a change to
the object.  It was possible for the zrpc layer to re-order the
response to the load and the invalidation, effectively causing the
invalidation to be lost.  The bug was fixed by delaying the handling
of invalidations for an object while a load for the same object is in
progress. 

Fix a bug in conflict resolution that failed to ghostify an object if
it was involved in a conflict.  (This code may be redundant, but it
has been fixed regardless.)

The ZEO server was fixed so that it does not perform any I/O until all
of a transactions invalidations are queued.  If it performs I/O in the
middle of sending invalidations, it would be possible to overlap a
load from a client with the invalidation being sent to it.

The ZEO cache now handles invalidations atomically.  This is the same
sort of bug that is described in the 3.1.2b1 section below, but it
affects the ZEO cache.

Fixed several serious bugs in fsrecover that caused it to fail
catastrophically in certain cases because it thought it had found a
checkpoint (status "c") record when it was in the middle of the file.