[ZODB-Dev] RE: [Zope-Annce] ZODB 3.2.4 release candidate1released

[Chris McDonough]

On Thu, 2004-09-09 at 22:34, Tim Peters wrote:
> ...
> 
> [Tim Peters]
> >> It not that it's the only thing you can do with it, it's that you *must*
> >> explicitly abort() or begin()
> 
> [Chris McDonough]
> > Er, why would begin() be allowed?
> 
> Because begin() implies abort()

Gotcha, thanks!

> > Tim has explained that the current behavior when a commit exception is
> > encountered is to abort all jars in the transaction and raise the
> > exception.  That's fine, but when the commit exception is suppressed, the
> > existing transaction remains in a usable state and there's no indication
> > that a failure ever occurred.  It's then possible for code that assumes
> > prior changes to persistent state in the same transaction were performed
> > successfully to do more changes and later commit its work without
> > incident.   I'd like that this wasn't the case, which is why I say +1 to
> > making it impossible to commit a transaction that has already failed a
> > commit once.
> 
> A practical question:  is it conceivable that this ever happens in Zope?

I know of one extremely likely place that it could happen now: in
ZCatalog, which commits subtransactions as a matter of course in order
to reduce memory usage during large batch inserts and updates of data. 
If an exception is raised by one of these subtransaction commits and
caught inappropriately, I assume that the pathological thing could
occur.

There are other minor places in Zope that do their own transaction
commit management:

- During Zope startup, which does explicit transaction management
  to install "Products".  Highly unlikely that any problems will
  occur here.

- When "auto-refresh" mode is on.  This is a highly volatile
  mode of development where Zope doesn't need to be restarted in
  order to detect changes to Python module files.  Basically,
  it registers a callback with "connection_open_hooks" that
  attempts to detect if changes have been made to on-disk
  Product files and if so, commits the transaction when it notices
  one has changed.  Evil!  I actually think this might not be
  a candidate for a problem area though because the commit
  is still called in the context of some Connection operation as
  opposed to in the context of arbitrary app code.  I don't think
  there is an opportunity for app code to catch this exception
  as a result.

- We commit a subtransaction when writing data into a
  Zope File or Image object, presumably to reduce memory usage
  during the transaction.  This happens when a Zope File/Image
  object is created, a common occurence typically managed by
  app code.

 - When a ZODBMountPoint is first traversed, we commit a
   subtransaction.  This code doesn't get invoked very often,
   not likely to cause a problem.

The only other explicit transaction commit control is present in the
Publisher.

  > Basically, if a commit failure occurs, the transaction needs to be put
> > into a doomed state.  And if all you can do with a failed transaction is
> > call abort() or begin() on it, it's doomed, but any other way that this
> > goal can be reached is fine by me too.
> 
> What is the universe of things "you can do with a transaction"?  Marking a
> Transaction object as doomed would not, for example, prevent you from
> loading objects from the database.  The Transaction object isn't involved in
> that.  The only effect I can think of would be to prevent you from doing
> another commit().  So let's get clear about *everything* you would want to
> see happen, if "prevent you from doing another commit()" isn't all you want.

I don't think it matters too much what is done with the data from the
connections involved in the doomed transaction after the first failed
commit, as long as the doomed transaction's connection modifications
don't make it back into persistent storage (and don't remain in the
connection cache for longer than the lifetime of the doomed
transaction).

It would be possible that the application would raise an error during
the "doomed" request/transaction because some data invariant was
violated because of the caught exception, but that's typically OK as
long as the inconsistent state doesn't make it back into persistent
storage and doesn't hang around in the connection cache.  At least I
think it's OK; fixing it is not really ZODB's responsibility.

It would also be possible to read inconsistent data from the connections
involved in the doomed transaction and write them out physically to a
nontransactional data storage that wasn't protected by the commit
prevention provided by the transaction manager, but that has always been
the case and there's not much we can do about that.

- C