[ZODB-Dev] RE: [Zope-Annce] ZODB 3.2.4 release candidate 1released

[Chris McDonough]

On Mon, 2004-09-13 at 14:53, Tim Peters wrote:
> A practical problem is that "the framework" has no choke point for errors.
> VersionLockError is raised by a variety of storage implementations that
> happen to support versions; likewise for various flavors of undo errors;
> POSKeyError is raised by one specific storage implementation;
> ReadConflictError is raised by a jar; ..., and storages in particular have
> no idea which Transaction object may be in current use (get_transaction()
> only reveals what the current thread transaction manager believes, but the
> user may not be using that transaction manager).  So there's a lot of code
> that would need to be fiddled to make these all "sticky", including external
> implementations of storages (and jars, if there are any externally defined
> jars).

Well I suppose there are a finite number of these places.

Looking at FileStorage (3.2 branch anyway): UndoErrors can probably
mostly be anticipated by app code because app code is what calls undo
routines directly and they are raised fairly predictably.  Same with
PackError.  So you could get away with not making these sticky and
pawning the responsibility for handling them properly off on the app
developer without feeling too badly.

There are a series of errors (CorrupedFileStorageError and friends and
errors that happen because IO is screwed) that are so serious that it
probably wouldn't matter if they were caught or not.  ReadOnlyError
doesn't matter much either due to its nature.

FileStorage raises some other errors in various places like ValueError,
TypeError, KeyError, IndexError but most of these appear to be used
during an internal signaling attempt or for uncommon "batch" operations
like iterating over the entire contents of the storage and thus don't
appear to be very dangerous for common operations.

Exceptions arent currently sticky but that might be candidates for
"stickiness" are FileStorageQuotaError (although I don't think anyone
uses that), VersionCommitError (particularly, Zope end-user app code
pretty much has no clue when it's in a version), POSKeyError,
VersionLockError, and of course ConflictError which is already sticky. 
In reality, versions and quotas aren't heavily used anymore at least
they've been informally deprecated by most Zope developers I know.

So of these, it looks like only POSKeyError is something that feels like
ZODB should manage stickiness for (because it's difficult to predict
when it will occur).  But it's raised in a lot of places that doesn't
have access to the current transaction. :-(  Maybe worse is that it
subclasses KeyError (having an except KeyError: in Zope is quite common
and has heretofore been deemed "safe"), so a developer could think his
code was perfect and still get hosed.

3.2's Connection.py raises a whole slew of errors but it seems to have
access to the transaction at all times as an instance attr.  Or at least
it pretends to.

I'm sure other fun exists in the other storages.  But if we could get
stickiness in place for "unpredictable" raises (I realize that's a
squishy term) from FileStorage, temporarystorage, Connection, DB and
utility modules, it might head off some potential data problems caused
by inappropriate exception handling.  Do you think it is worth it?  I
could help.  I think. ;-)

FWIW, I think Toby's storage is the only one with any real traction
outside of the ones that ship in the ZODB package currently.  Oh and
temporarystorage.

I don't know of any 3rd party jar implementations.  Well, that's not
really true, there are several I know of: a transactional "mailhost" and
a cache manager that saves files to the filesystem "transactionally"
(both for Zope).  But neither of these has enough balls to raise any
exceptions, IIRC. ;-)

(BTW, is it proper to refer to them as "jars" or "connections" or both?)

> > d)  A corollary: it would be nice if some exceptions couldn't be
> >     caught except explicitly.  It's nice in Zope's case because it
> >     removes a bit of data integrity maintenance responsibility from
> >     both the developer who uses Zope and from the developers who create
> >     ZODB and into a centralized top-level exception handler within
> >     the publisher, which can handle these sorts of things in a
> >     well-defined way.
> 
> That would be nice, yes.

This was not meant to be a nag, BTW.   Just trying to write it down to
get it clear in my own head even if it is patently restating the
obvious.  ;-)

> >> Armin Rigo has a speculative patch pending that tries to get "dangerous"
> >> exceptions raised *even if* PyErr_Clear() gets done at the C level.
> 
> > This seems like a sane thing as long as you can define what "dangerous"
> > is.
> 
> Presumably we'd split Python's exception hierarchy, and define a base class
> for dangerous exceptions.  For example, MemoryError definitely belongs in
> that class.  In a little-known disaster mode now, if you have code that ends
> up triggering MemoryError during dict lookup, the C-level dict lookup code
> can end up suppressing the MemoryError and turning it into a KeyError by
> accident.  So code can go crazy, failing to retrieve something it *knows* is
> in a dict.  That case was actually the trigger for Armin's patch.  Indeed,
> if a dict lookup triggers a ReadConflictError or POSKeyError (or anything
> else), the same thing can happen; PyDict_GetItem() suppresses all exceptions
> now.

Wow.  I wonder how likely a codepath that is.  It doesn't seem like an
RCE or PKE would be raised in the course of PyDict_GetItem() because the
dict it was operating against would already be unghosted and isn't its
own persistent object anyway, right?

- C