[Checkins] [zopefoundation/ZODB] 227953: Simplify MVCC by determining transaction start tim...

[GitHub]

  Branch: refs/heads/no-more-load
  Home:   https://github.com/zopefoundation/ZODB
  Commit: 227953b977a9e195c4ce9bbb9acd9c5ee60c333a
      https://github.com/zopefoundation/ZODB/commit/227953b977a9e195c4ce9bbb9acd9c5ee60c333a
  Author: Jim Fulton <jim at zope.com>
  Date:   2016-05-04 (Wed, 04 May 2016)

  Changed paths:
    M src/ZODB/ActivityMonitor.py
    M src/ZODB/BaseStorage.py
    M src/ZODB/Connection.py
    M src/ZODB/DB.py
    M src/ZODB/DemoStorage.py
    M src/ZODB/FileStorage/FileStorage.py
    M src/ZODB/MappingStorage.py
    M src/ZODB/tests/BasicStorage.py
    M src/ZODB/tests/MTStorage.py
    M src/ZODB/tests/MVCCMappingStorage.py
    M src/ZODB/tests/PackableStorage.py
    M src/ZODB/tests/StorageTestBase.py
    M src/ZODB/tests/blob_packing.txt
    M src/ZODB/tests/blob_tempdir.txt
    M src/ZODB/tests/blobstorage_packing.txt
    M src/ZODB/tests/dbopen.txt
    M src/ZODB/tests/synchronizers.txt
    M src/ZODB/tests/testConnection.py
    M src/ZODB/tests/testDB.py
    M src/ZODB/tests/testDemoStorage.py
    M src/ZODB/tests/testFileStorage.py
    M src/ZODB/tests/testMVCCMappingStorage.py
    M src/ZODB/tests/test_doctest_files.py
    M src/ZODB/tests/test_fsdump.py
    M src/ZODB/tests/test_storage.py
    M src/ZODB/tests/testblob.py
    M src/ZODB/tests/testmvcc.py
    M src/ZODB/tests/util.py
    M src/ZODB/utils.py

  Log Message:
  -----------
  Simplify MVCC by determining transaction start time using lastTransaction.

This implements: https://github.com/zopefoundation/ZODB/issues/50

Rather than watching invalidations, simply use 1 + the storages
lastTransaction, which is equivalent to but much simpler than waiting
for the first invalidation after a transaction starts.

More importantly, it means we can always use loadBefore and get away
from load.  We no longer have to worry about ordering of invalidations
and load() results.

Much thanks to NEO for pointing the way toward this simplification!

Implementing this initially caused a deadlock, because DB.open()
called Connection.open() while holding a database lock and
Connection.open() now calls IStotage.lastTransaction(), which acquires a
storage lock. (It's not clear that lastTransaction() really needs a
storage lock.)  Meanwhile, IStotage.tpc_finish() calls a DB function
that requires the DB lock while holding the storage lock.  Fixing this
required moving the call to Connection.open() outside the region where
the DB lock was held.

To debug the problem above, I greatly improved lock-debugging
support. Now all of the ZODB code imports Lock, RLock and Condition
from ZODB.utils. If the DEBUG_LOCKING is set to a non-empty value,
then these are wrapped in such a way that debugging information is
printed as they are used. This made spotting the nature of the
deadlock easier.

Of course, a change this basic broke lots of tests. Most of the
breakage arises from the fact that connections now call
lastTransaction on storages at transaction boundaries.  Lots of tests
didn't clean up databases and connections properly.  I fixed many
tests, but ultimately gave up and added some extra cleanup code that
violated transaction-manager underware (and the underware's privates)
to clear transaction synchonizers in test setup and tear-down.  I plan
to add a transaction manager API for this and to use it in a
subsequent PR.

This tests makes database and connection hygiene a bit more important,
especially for tests, because a connection will continue to interact
with storages if it isn't properly closed, which can lead to errors if
the storage is closed.  I chose not to swallow these errors in
Connection, choosing rather to clean up tests.

The thread debugging and test changes make this PR larger than I would
have liked. Apologies in advance to the reviewers.