[ZODB-Dev] RE: [Zope-Annce] ZODB 3.2.4 release candidate 1released

[Chris McDonough]

On Tue, 2004-09-07 at 20:53, Tim Peters wrote:
> I don't know of a scenario in current ZODB wherein catching something can
> make a database inconsistent.

I don't know of any scenario that can lead to storage-level
inconsistency but I believe that too-eager exception handlers can cause
app data inconsistency problems.  We spoke about it on this list in
May.  I would point you at the archived copy but I can't seem to find
it, so I paste part of the coversation below:

-- BEGIN STUFF --

Tim said in May:
> However, after the ConflictError raised by the *attempt* to commit has
been
> suppressed, that transaction is history, and a new transaction begins
(which
> is implicit, which makes it devilishly obscure).  Concretely:  if you
> replace the line above by these 5:
> 
>         self.assert_(not real_data2.has_key("a"))
>         self.assertRaises(ConflictError, cn2.getTransaction().commit)
>         self.assert_(real_data2.has_key("a"))
>         self.assert_(not real_data2._p_changed)
>         cn2.getTransaction().commit()
> 
> the test still passes.  Suppressing the error on the first attempt to
commit
> effectively wipes out (undoes) the changes we were trying to commit,
and
> silently refetches fresh data (and the final line there doesn't
actually
> commit any changes).  It's as if you had done an explicit transaction
> abort() immediately following the suppressed commit() exception.

Aha!  I always forget that people who use ZODB outside of Zope need to
implement their own strategies for dealing with conflicts and connection
management.

Zope's transaction/connection management model is defined mostly in
ZPublisher, where a transaction == a single request (where "request" is
defined as a single HTTP/FTP/DAV interaction) == the lifetime of a
single connection.  We typically never see the effect of this particular
phenomenon because for Zope it could only be seen "between" requests,
and we usually don't have any visibility there, except for when someone
does an explicit "get_transaction().commit()" in Zope code, which is
almost never done.  The key word in that last sentence being "almost".
;-)  I should probably do an audit to see where this happens and see if
it's possible to get rid of, because we just can't protect people from
overeager exception handlers there (we don't control the code that
people wrap around these places).

For Zope purposes, it might be better to create a transaction mode that
didn't do the implicit abort() and resync after commit fail and just
left the connection in an insane state that just plain refused to commit
itself (by repeatedly raising ConflictError during further commit
attempts involving that connection) until it was closed and reopened
explicitly.  Then again, Zope doesn't use the "local transaction"
connection mode, so I'm not sure if that has any effect on things.

-- /END STUFF --


> There's generally no *point* to continuing a transaction once it has
> experienced a ConflictError

At very least it shouldn't be allowed to commit itself.  The implicit
behavior of the above is problematic because the user never realizes he
began a new transaction if there is an overeager exception handler that
catches a conflict error at commit.  While this is not literally the
commit of a "half-finished" transaction, the behavior in practice seems
like it can result in the same symptom (you think you committed data
that you didn't, and later commits might have state that depend on that
uncommitted data in some way).  An example found in the wild of this
might be Zope's ZCatalog, which performs a subtransation every so often
during batch indexing operations.  Batch indexing operations are
typically invoked by arbitrary application code.  If one of the commits
implied by that indexing is lost due to an overeager exception handler,
the ZCatalog and index state could become insane.

I think it would be better to do away with the implicit abort and resync
on commit fail due to conflict error and instead just left the existing
transaction in an uncomittable state.  Doing an implicit abort and
resync just seems kinda silly, doesn't it?

>  -- current ZODB intends never to let a
> transaction commit if it experienced a ReadConflictError in its lifetime.

Or at least it never lets a transaction commit if that ReadConflictError
was raised by the Connection itself.  You can raise it from app code all
you like and it won't prevent the transaction from committing.

- C