[ZODB-Dev] RE: [Zope-Annce] ZODB 3.2.4 release candidate 1released

[Chris McDonough]

On Mon, 2004-09-13 at 22:51, Tim Peters wrote:
> [Chris McDonough]>
> > Ah ok.  So it probably wouldn't hurt to make it sticky then, I presume?
> 
> Sorry, I really don't know -- but I *guess* that would be OK.  I have
> more-or-less deep understanding of the pieces of ZODB that have broken on my
> watch and so needed bugfixes.  My understanding of the rest is fuzzy to
> non-existent.  Same way I learned about BTrees.  Luckily <wink>, they
> eventually turned out to be broken in many places (some gross, most
> microscopic), so I eventually learned almost everything about them
> 
> Digging thru the old UML models for ZODB:
> 
>     http://www.zope.org/Documentation/Developer/Models/ZODB
> 
> I see that the storage interface says of load():
> 
>     A KeyError exception is raised if there is not a record for the
>     given oid and version.
> 
> I can deduce that this is *why* POSKeyError derives from KeyError, but have
> no way to guess whether anything relies on that.  A few ZODB tests probably
> do.  But as you said, it's dreadful that app code can end up swallowing a
> POSKeyError in general code that doesn't care about, e.g., vanilla dict
> lookups failing.  POSKeyError is serious.
> 
> OTOH, what happens if you suppress one?  It means you tried to load a thing
> that couldn't be gotten.  If your app tries to make any use of that thing,
> it will get POSKeyError again, because the load didn't succeed, and never
> will.  So in that sense it's already "kind of" sticky.  It won't stop the
> current transaction from committing if it's suppressed, but if an app really
> needs a thing that isn't in the database, it will get POSKeyError over and
> over again.  Retrying the transaction can't cure that, either.

Despite the special behavior of POSKeyError (a particular object can't
be loaded), it seems that catching one inappropriately can produce the
same sort of app-level inconsistencies as any other kind of error.

Let's say an app-level invariant is that when the following setMyDict
method of a persistent object finishes, self.d should always contain
three key/value pairs represented by keys 'a', 'b', and 'c'...

def setMyDict(self):
   self._p_changed =  1 
   self.d = {}
   self.d['a'] = getattr(self, 'attr_that_is_available', None)
   self.d['b'] = getattr(self, 'attr_that_is_available2', None)
   self.d['c'] = getattr(self, 'generates_pos_key_error', None)

If POSKeyError is generated by the attr lookup of
"generates_pos_key_error" and its later caught in code above and the
transaction later commits, the invariant is broken.

Granted it's a screwed up situation because setMyDict's invariant can
just never be met while the object is missing.   But the problem with
allowing the transaction to commit is that you won't find out about the
broken invariant until much later, maybe even in unrelated code in
another thread or process.

That's no different than any other normal caught exception but FWIW, the
specific reason that I think this is a "special" exception that should
be "sticky-managed" is that it's just not very obvious that the
innocuous-looking attribute lookup for 'generates_pos_key_error' can
fail (no developer typically expects a KeyError of any type to emanate
from a getattr) so a developer would be unlikely to be aware that they
needed to do an explicit test for that there and prevent the transaction
from committing themselves in order to maintain the invariant (or at
least to advertise the fact that the invariant can't be met).

> If POSKeyErrors were common, I could conceive of code resorting to:
> 
>     try:
>         obj = thing1
>     except POSKeyError:
>         obj = thing2
> 
> in desperation, and then it would be bad if the intentionally suppressed
> POSKeyError prevented the transaction from succeeding.  But I strongly doubt
> that there is code like that, and POSKeyError really is supposed to be
> impossible (and despite that it's apparently common in 2.7.2 ...).

I think POSKeyError qualifies as a "fatal" error in the context of a
heavily ZODB-dependent application.  I've never explicitly or
intentionally caught a POSKeyError when writing a Zope app, and that's
evidence enough for me that no one else needs to do it either. ;-)

- C