[Zope-dev] deadlock prevention for ZODB3 / Zope 2.6

Jeremy Hylton jeremy@zope.com
Tue, 12 Nov 2002 15:30:08 -0500


We recently discovered that ZODB3 applications, like Zope 2.6, can
deadlock when run in a system that uses multiple storages.  This was a
fundamental design flaw in ZODB that, happily, has a simple fix.

Brian and I are planning to commit these changes to the ZODB3 3.1 and
Zope 2.6 release branches and make them part of an upcoming bug fix
release.  This is a slightly dodgy plan, because there is a lot of
code that is being changed for this bug fix.  But the problem is
fairly serious and the solution is available now, we'd like to get it
in the hands of users more quickly than ZODB 3.2 and Zope 2.7.

In short, the solution is to sort storages before beginning the
two-phase commit process used by the transaction manager.  Since the
two-phase commit occurs during a transaction commit, we know all the
storages that will be used, and, thus, all the locks that will be
acquired.  By sorting the locks, we can guarantee that deadlock will
not occur.

The implementation of deadlock prevention requires changes to the
storage and transaction data manager APIs.  Each must now implement a
sortKey() method that returns a string that can be used to sort the
storages.  The sortKey must be globally unique; two storages can't
have the same key unless they are the same storage and one storage
must have the same key regardless of when it is used.  The sortKey
must be a string, because other objects may not sort in the same order
for all Python versions.

We have provided default implementions of sortKey() for BaseStorage
and Connection.  These implementations aren't necessarily correct, but
they will prevent outright breakage if you update to the maintenance
release.  BaseStorage uses the __name__ and Connection uses id().  All
storages and data managers should be updated to the new API.  The
default implementations may not prevent deadlock in a multi-storage
environment; they only prevent the update from breaking existing code.

Jeremy