[ZODB-Dev] Reloading product in 2.8

[Tim Peters]

[Victor Safronovich, having problems with "auto refresh"]

Victor (et alia), I think a discussion of auto-refresh would be more useful
on a Zope list -- it's not really (AFAIK) a ZODB issue.

This is what changed in ZODB, from ZODB 3.3c1's NEWS file (and no, it won't
be changed back to the way it was):

"""
- Transactions have new, backward-incompatible behavior in one respect:
  if a ``Transaction.commit()``, ``Transaction.commit(False)``, or
  ``Transaction.commit(True)`` raised an exception, prior behavior was that
  the transaction effectively aborted, and a new transaction began.
  A primary bad consequence was that, if in a sequence of subtransaction
  commits, one of the commits failed but the exception was suppressed,
  all changes up to and including the failing commit were lost, but
  later subtransaction commits in the sequence got no indication that
  something had gone wrong, nor did the final (top level) commit.  This
  could easily lead to inconsistent data being committed, from the
  application's point of view.

  The new behavior is that a failing commit "sticks" until explicitly
  cleared.  Now if an exception is raised by a ``commit()`` call (whether
  subtransaction or top level) on a Transaction object ``T``:

    - Pending changes are aborted, exactly as they were for a failing
      commit before.

    - But ``T`` remains the current transaction object (if ``tm`` is ``T``'s
      transaction manger, ``tm.get()`` continues to return ``T``).

    - All subsequent attempts to do ``T.commit()``, ``T.join()``, or
      ``T.register()`` raise the new ``TransactionFailedError`` exception.
      Note that if you try to modify a persistent object, that object's
      resource manager (usually a ``Connection`` object) will attempt to
      ``join()`` the failed transaction, and ``TransactionFailedError``
      will be raised right away.

  So after a transaction or subtransaction commit fails, that must be
  explicitly cleared now, either by invoking ``abort()`` on the transaction
  object, or by invoking ``begin()`` on its transaction manager.
"""

That's mostly what Victor reported, instances of:

    TransactionFailedError: An operation previously failed, with traceback:

followed by the traceback of the _original_ exception that caused ZODB to
mark the transaction as being fatally hosed.  The original exception may
have happened long in the past (but after the current transaction began).

Each instance of such a thing points to one or two application bugs:

1. The original exception reported.

and

2. That the original exception was suppressed to begin with (e.g, perhaps
   by a bare "except:" clause in the app, catching more than it really
   intended to catch), and without aborting the current transaction at
   the time in response.

The originally suppressed exceptions in Victor's report:

> ValueError: Cache values may only be in one cache.

and

> InvalidObjectReference: Attempt to store an object from a foreign database
connection

suggest that app code is sharing persistent objects across threads, e.g.
loading an object O from one connection in thread A, and then in thread B
trying to store a mutated O via B's connection.  ZODB doesn't support that
(it's fine if A and B have their own _copies_ of O, what they can't do is
share an in-memory persistent object).

The other exception reported is a sign that a module has been loaded into
sys.modules more than once, under different dotted path names; while these
can be nightmarish to untangle, they're not ZODB issues either:

> PicklingError: Can't pickle <class
> 'Products.CMFNauTools.Explorer.FolderExplorer'>: it's not the same object
> as Products.CMFNauTools.Explorer.FolderExplorer