[ZODB-Dev] RE: [Zope-Annce] ZODB 3.2.4 release candidate 1released

[Tim Peters]

[Chris McDonough]
> Well I suppose there are a finite number of these places.
>
> Looking at FileStorage (3.2 branch anyway): UndoErrors can probably
> mostly be anticipated by app code because app code is what calls undo
> routines directly and they are raised fairly predictably.  Same with
> PackError.  So you could get away with not making these sticky and
> pawning the responsibility for handling them properly off on the app
> developer without feeling too badly.
>
> There are a series of errors (CorrupedFileStorageError and friends and
> errors that happen because IO is screwed) that are so serious that it
> probably wouldn't matter if they were caught or not.  ReadOnlyError
> doesn't matter much either due to its nature.
>
> FileStorage raises some other errors in various places like ValueError,
> TypeError, KeyError, IndexError but most of these appear to be used
> during an internal signaling attempt or for uncommon "batch" operations
> like iterating over the entire contents of the storage and thus don't
> appear to be very dangerous for common operations.
>
> Exceptions arent currently sticky but that might be candidates for
> "stickiness" are FileStorageQuotaError (although I don't think anyone
> uses that), VersionCommitError (particularly, Zope end-user app code
> pretty much has no clue when it's in a version), POSKeyError,
> VersionLockError, and of course ConflictError which is already sticky.

Only ReadConflictError is sticky.  The base ConflictError is used for write
conflicts, those occur during commit(), and currently start a new
transaction if one occurs.

> In reality, versions and quotas aren't heavily used anymore at least
> they've been informally deprecated by most Zope developers I know.

Jim formally deprecated versions a couple months ago:

    http://mail.zope.org/pipermail/zope3-dev/2004-July/011670.html

Well, actually, the formal part was announcing an *intent* to deprecate
versions.  They haven't actually been deprecated.  Sometimes you need a
photographic memory <wink>.

> So of these, it looks like only POSKeyError is something that feels like
> ZODB should manage stickiness for (because it's difficult to predict when
> it will occur).

POSKeyError should never occur.  

> But it's raised in a lot of places that doesn't have access to the
> current transaction. :-(  Maybe worse is that it subclasses
> KeyError (having an except KeyError: in Zope is quite common and has
> heretofore been deemed "safe"), so a developer could think his code was
> perfect and still get hosed.

Ewww -- hadn't thought of that.  So it's a doubly good thing that
POSKeyError is impossible <wink>.

> 3.2's Connection.py raises a whole slew of errors but it seems to have
> access to the transaction at all times as an instance attr.  Or at least
> it pretends to.

Yes, 3.2 Connection has to know the current transaction, because modified
objects are registered with the transaction in 3.2 (see
Connection.register).  In 3.3, Connections keep track of their own modified
objects, but the transaction manager the Connection should use is passed to
DB.open(); because resource managers (like Connections) "join" transactions
in 3.3, Connection still needs to know this.

> I'm sure other fun exists in the other storages.  But if we could get
> stickiness in place for "unpredictable" raises (I realize that's a
> squishy term) from FileStorage, temporarystorage, Connection, DB and
> utility modules, it might head off some potential data problems caused by
> inappropriate exception handling.  Do you think it is worth it?  I could
> help.  I think. ;-)

I think it may be worth it if it starts with the 2.8 line.  Trying to keep
multiple versions healthy is a horrendous drag on making improvements (like,
e.g., adding savepoints, or support for multiple database connections -- or
even this!).

> FWIW, I think Toby's storage is the only one with any real traction
> outside of the ones that ship in the ZODB package currently.  Oh and
> temporarystorage.
>
> I don't know of any 3rd party jar implementations.  Well, that's not
> really true, there are several I know of: a transactional "mailhost" and
> a cache manager that saves files to the filesystem "transactionally"
> (both for Zope).  But neither of these has enough balls to raise any
> exceptions, IIRC. ;-)
>
> (BTW, is it proper to refer to them as "jars" or "connections" or both?)

An instance of ZODB.Connection is a connection.  "jar" appears to be the
generic lingo in 3.2, and "resource manager" in 3.3, at which point "jar"
becomes an abbreviation for "Just Another ResourceManager" <wink>.

[... sorry for cutting things out, but I'm spending way too much time
     staring at email today ...]

>> Indeed, if a dict lookup triggers a ReadConflictError or POSKeyError
>> (or anything else), the same thing can happen; PyDict_GetItem()
>> suppresses all exceptions now.

> Wow.  I wonder how likely a codepath that is.  It doesn't seem like an
> RCE or PKE would be raised in the course of PyDict_GetItem() because the
> dict it was operating against would already be unghosted and isn't its
> own persistent object anyway, right?

The potential problem is that dict key comparison can invoke arbitrary
Python code.  You'd have to be certifiably insane to, e.g., use a persistent
object as a dict key, but that doesn't mean it can't happen.  Objects of any
user-defined type, persistent or not, could try to do something with
persistent objects in a __cmp__ or __eq__ method, and dict lookup may have
to invoke one of those.