[ZODB-Dev] RE: [Zope-Annce] ZODB 3.2.4 release candidate 1released

[Tim Peters]

[Florent Guillaume]
>> This was changed to:
>>     try:
>>         ...stuff...
>>     except ConflictError:
>>         raise
>>     except:
>>         LOG('an error happened: %s' % sys.exc_info())
>>         # continue anyway as we "shouldn't fail"
>>         # or we "want to return a default value if we fail"
>>     ...rest...
>>
>> To make sure we didn't inadvertently catch something that could make the
>> database inconsistent.

[Shane Hathaway]
> I'd like to point out another reason I think this approach to database
> consistency is futile:  restricted Python scripts can catch all
> exceptions, including conflict errors.  We could plug that "hole", but
> who knows when the next will show up?  This is a systemic problem.

Doesn't the problem go beyond just catching conflict errors?  Dieter (very)
recently fleshed out this scenario:

    try:
        a pile of code that modifies persistent state
    except:  # maybe a bare except, maybe not
        log something, but don't reraise
    commit()

The "pile of code" may or may not be in a lexically enclosing try block; it
could just as well be buried in a function body, with the try/except at some
higher level.  The commit() may be in a different function and occur 5
minutes later; etc.

The point is that *any* exception occurring in the pile of code will leave
some number of the pile's trailing statements unexecuted, so that the
persistent state is really unknown at commit() time.  I think the same
argument applies exactly to Florent's example:  what's so special about
ConflictError?  Perhaps just that ReadConflictErrors are inevitable? 

Even then, ReadConflictErrors are less harmful than other exceptions,
because suppressing one of those leaves the transaction in an uncommittable
state despite the suppression.  But a suppressed AttributeError (etc) has no
effect on whether commit() will succeed.

> Here is a solution I just stumbled upon this month:

This is a 3-pass solution.  That is, I had to read the next two paragraphs 3
times before I could understand what it was <wink>.

> transactions should not start implicitly in Zope.  After you commit or
> abort a transaction,

What if you try to commit() but the commit fails?  Same deal, right?  Do
commit and abort there include the cases of subtransaction commit and abort
too?  I'm assuming they do.

> the object system should be frozen until you explicitly begin the next
> transaction with get_transaction().begin().  Attempts to change objects
> between transactions should result in an immediate exception.

Here I read the last sentence as expanding on "frozen"'s intended meaning,
rather than as a requirement in addition to frozenness.  If that wasn't your
intent, holler.

Do you think "an attempt to transition an object to the changed state raises
an exception" would be an accurate implementation of frozenness?  Maybe not,
cuz that would still let changes occur in memory, it would just prevent
modified objects from getting *marked* as being changed!  That sounds less
like a cure than like a new disaster <wink>.  So I'm not 100% sure on what
you intend, but I'm not sure either that the current persistence model can
support it.  Do you have an implementation in mind?

> That way, conflict errors can simply abort the transaction without
> beginning a new one.

Which seems to be the *real* point.

> We will not have the partial transaction effect that we have today.

As above, a ReadConflictError prevents commit, so does not lead to partial
transactions (if I understand what that phrase means).  I don't think you
can get a write conflict error except *during* commit().  Are there flavors
of conflict error other than those two?

There's also what Dieter pointed out, that any suppressed exception can lead
to a "partial transaction".

Because of those, I confess I just don't get the continued fretting over
conflict errors specifically.  It is the case that a suppressed exception
*during* commit() (subtransaction or not) magically starts a new transaction
now, and that makes me shiver all over.  That's been actively discussed on
zodb-dev over the last two days.  Dieter doesn't seem to think there's
anything dubious about that behavior.  Do you think it would help if a
failing commit did not magically start a new transaction, but guaranteed
additional commits would continue to fail until an explicit begin() or
abort()?  I realize you're suggesting more than that, but I know how to
implement what I'm asking about there, and it doesn't seem to lead to
cascades of other difficulties.

> As a bonus, code that accidentally writes between HTTP requests will
> expose itself.

What does "write" mean to you?  Is it just that a persistent object gets
modified in memory, or does "write" also require that the modification get
committed?