[ZODB-Dev] Random POSKeyErrors

[Jim Fulton]

On Wed, Jul 21, 2010 at 10:52 AM, Pedro Ferreira
<jose.pedro.ferreira at cern.ch> wrote:
> Hello,
>
> We are receiving occasional error messages such as:
>
> """
> Traceback (most recent call last):
>   File
> "/usr/lib/python2.4/site-packages/cds_indico-0.97_rc1-py2.4.egg/MaKaC/webinterface/rh/base.py",
> line 459, in process
>     self._setSessionUser()
>   File
> "/usr/lib/python2.4/site-packages/cds_indico-0.97_rc1-py2.4.egg/MaKaC/webinterface/rh/base.py",
> line 224, in _setSessionUser
>     self._aw.setUser( self._getSession().getUser() )
>   File
> "/usr/lib/python2.4/site-packages/ZODB3-3.8.5-py2.4-linux-x86_64.egg/ZODB/Connection.py",
> line 815, in setstate
>     self._setstate(obj)
>   File
> "/usr/lib/python2.4/site-packages/ZODB3-3.8.5-py2.4-linux-x86_64.egg/ZODB/Connection.py",
> line 856, in _setstate
>     p, serial = self._storage.load(obj._p_oid, self._version)
>   File
> "/usr/lib/python2.4/site-packages/ZODB3-3.8.5-py2.4-linux-x86_64.egg/ZEO/ClientStorage.py",
> line 712, in load
>     return self.loadEx(oid, version)[:2]
>   File
> "/usr/lib/python2.4/site-packages/ZODB3-3.8.5-py2.4-linux-x86_64.egg/ZEO/ClientStorage.py",
> line 735, in loadEx
>     data, tid, ver = self._server.loadEx(oid, version)
>   File
> "/usr/lib/python2.4/site-packages/ZODB3-3.8.5-py2.4-linux-x86_64.egg/ZEO/ServerStub.py",
> line 196, in loadEx
>     return self.rpc.call("loadEx", oid, version)
>   File
> "/usr/lib/python2.4/site-packages/ZODB3-3.8.5-py2.4-linux-x86_64.egg/ZEO/zrpc/connection.py",
> line 699, in call
>     raise inst # error raised by server
> POSKeyError: 0x336b08e2
> """
>
> I read somewhere that this could mean our FileStorage is corrupted,

No, where 'corrupted' means that the structure of the file storage is broken.

> but
> I find it strange that it only happens occasionally.
> Any clue on what can be causing this?

You have a dangling reference. A database record has a reference to a
persistent object, via its object ID, and the referenced object has
been removed.

This can happen if you are using multiple databases, with cross
database references, and you garbage collect (typically through pack)
the databases individually.

It can happen if you do certain pathological things in your app. For example:

   x = some_collection.pop(key)
   transaction.commit()
   db.pack()
   some_collection[key] = x
   transaction.commit()

At this point, x has been deleted through packing, but some_collection
references it.  This is mostly theoretical. I don't think I've heard
of this happening in the wild. In theory, ZODB could check for bugs
like this, but it doesn't at this time.

POSKeyErrors have, in the past, resulted from bugs, especially pack
bugs, although I'm mot aware of any recent bugs that would cause
POSKeyErrors.  (Some would argue that cross-database references are a
bug, although I would call them an advanced feature that must be used
with care. :)

Jim

-- 
Jim Fulton