[Zope-dev] 2.6.1b2?

Jeremy Hylton jeremy@zope.com (Jeremy Hylton)
Tue, 28 Jan 2003 12:53:37 -0500


>>>>> "BL" == Brian Lloyd <brian@zope.com> writes:

  >> > Here's the status - an engagement that we're doing has been
  >> > bringing up some issues regarding ZODB and ZEO in large-scale
  >> > environments. I think that the fixes are useful enough that
  >> > they should be in 2.6.1, but getting them "finalized" has taken
  >> > longer than I expected.
  >>
  >> I'd love to know what kind of thing 'large-scale' implies here,
  >> and what kind of problems the fixes fixed.

  BL> "Large-scale" meaning large numbers of ZEO clients, that mount
  BL> multiple ZEO-served databases that are each replicated using ZRS
  BL> (Zope Corp.'s replication / failover solution) :^)

  BL> The changes have to do with coordination of transaction commit
  BL> among multiple databases, manageability and
  BL> fault-tolerance. I'll ask Jeremy to be sure to update the
  BL> CHANGES.txt with the important changes.

I've included the current list of changes from ZODB3/NEWS.txt.  I
believe the list is complete, but would want Barry and Guido to
double-check.  The first change is possibily controversial.  I think
the others are fairly conservative.

Jeremy

The Transaction "hosed" feature is disabled in this release.  If a
transaction fails during the tpc_finish() it is not possible, in
general, to know whether the storage is in a consistent state.  For
example, a ZEO server may commit the data and then fail before sending
confirmation of the commit to the client.  If multiple storages are
involved in a transaction, the problem is exacerbated: One storage may
commit the data while another fails to commit.  In previous versions
of ZODB, the database would set a global variable "hosed" that
prevented any other transaction from committing until an administrator
could check the status of the various failed storages and ensure that
the database is in a consistent state.  This approach favors data
consistency over availability.  The new approach is to log a panic but
continue.  In practice, availability seems to be more important than
consistency.  The failure mode is exceedingly rare in either case.

The BTrees-based fsIndex for FileStorage is enabled.  This version of
the index is faster to load and store via pickle and uses less memory
to store keys.  We had intended to enable this feature in an earlier
release, but failed to actually do it; thus, it's getting enabled as a
bug fix now.

Two rare bugs were fixed in BTrees conflict resolution.  The most
probable symptom of the bug would have been a segfault.  The bugs
were found via synthetic stress tests rather than bug reports.

A value-based consistency checker for BTrees was added.  See the
module BTrees.check for the checker and other utilities for working
with BTrees.

ZEO
---

The ZEO version number was bumped to 2.0.2 on account of the below
minor feature additions.

The performance of full cache verification has improved dramatically.
XXX Get measurements from Jim -- somewhere in 2x-5x recall.  The
implementation was fixed to use the very-fast getSerial() method on
the storage instead of the comparatively slow load().

The ZEO server has an optional timeout feature that will abort a
connection that does not commit within a certain amount of time.  The
timeout works by closing the socket the client is using, causing both
client and server to abort the transaction and continue.  This is a
drastic step, but can be useful to prevent a hung client or other bug
from blocking a server indefinitely.

If a client was disconnected during a transaction, the tpc_abort()
call did not properly reset the internal state about the transaction.
The bug caused the next transaction to fail in its tpc_finish().