[ZODB-Dev] Loading objects directly by OID

[Greg Ward]

Hi all --

I'm having a problem loading an object directly by OID (ie., __getitem__
on the Connection).  Specifically, there is an object in our database
that cannot be loaded directly until it is loaded indirectly, by
traversing the object graph to get to it.  So far, I have only found one
such object, and I found it by accident.  (Well, actually, it was found
by my Grouch type-checking tool, because this mystery object doesn't
have any instance attributes, and the object schema says it should have
some.  Three cheers for Grouch!)

Anyways, the demonstration.  First, open the database (I'm skipping the
imports):

  >>> storage = ClientStorage(("localhost", 1972))
  >>> db = DB(storage)
  >>> connection = db.open()

Luckily, I happen to know the OID of the mystery object; let's try to
load it directly:

  >>> oid = '\0\0\0\0\0\x07\xf5\xc6'
  >>> connection[oid]
  Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    File "/www/python/lib/python2.1/site-packages/ZODB/Connection.py", line 160, in __getitem__
      p, serial = self._storage.load(oid, self._version)
    File "/www/python/lib/python2.1/site-packages/ZEO/ClientStorage.py", line 357, in load
      p, s, v, pv, sv = self._call('zeoLoad', oid)
    File "/www/python/lib/python2.1/site-packages/ZEO/zrpc.py", line 228, in __call__
      if type(r) is TupleType: raise r[0], r[1]
  KeyError: υΖ

Uh-oh!  The obvious reaction is, "Greg, you dolt, there is no such OID
-- that's what KeyError in this situation means".  Well, hold your
horses: now I'm going to walk the object graph to load this object
indirectly:

  >>> root["run_db"].runs[639].object_versions[5].sequence._key_map['S0002'].wafer_ids['W001'].description.resistivity.unit.dims
  <DimensionCollection at 83fbc88>

(Yes, our database really does go that deep... scary, eh?)

OK, so far I've shown nothing at all interesting.  Here's what it gets
confusing:

  >>> _._p_oid
  '\x00\x00\x00\x00\x00\x07\xf5\xc6'
  >>> _ == oid
  1

And now, I try to directly load this object a second time:

  >>> connection[oid]
  <DimensionCollection at 83fbc88>

Anyone know what the heck is going on here?  Do I totally misunderstand
what __getitem__ on a Connection is supposed to do?  I would have
thought that, given a static and unchanging database, connection[oid]
would always return the same thing for a given OID, no matter what other
database reads were done in the meantime.  Apparently not!

The only possibly useful clue I have right now is that the two objects
immediately preceding this DimensionCollection are not Persistent
objects.  That is:

  # 3 objects away is Persistent
  >>> wd = root["run_db"].runs[639].object_versions[5].sequence._key_map['S0002'].wafer_ids['W001'].description
  >>> wd._p_oid
  '\x00\x00\x00\x00\x00\x07\xf8\x9a'

  # but the last 2 objects before my problem object are not:
  >>> wd.resistivity
  <PhysicalValue at 83eed64: 1e+15 Ohm*cm>
  >>> hasattr(wd.resistivity, '_p_oid')
  0
  >>> wd.resistivity.unit
  <PhysicalUnit at 8334d7c: Ohm*cm>
  >>> hasattr(wd.resistivity.unit, '_p_oid')
  0

  # back to the mystery object
  >>> wd.resistivity.unit.dims  
  <DimensionCollection at 83fbc88>
  >>> wd.resistivity.unit.dims._p_oid
  '\x00\x00\x00\x00\x00\x07\xf5\xc6'

Oh yeah, this doesn't appear to be ZEO-related; if I shut down my ZEO
server and open the database with FileStorage instead, I get exactly the
same behaviour.

Oh yeah again: it's conceivable that a __setstate__() method somewhere
in that long chain of indirect loads is creating my mystery object on
the fly, which is why it's not in the database when this process starts
up.  Holes in that theory:
  * the OID of the mystery object is the same every time, and it is
    *not* max_oid+1.  Does ZODB ever allocate OIDs out-of-order?
  * I thought an OID wasn't allocated until the new object is
    committed to the database: correct?

Thanks --

        Greg
-- 
Greg Ward - software developer                gward@mems-exchange.org
MEMS Exchange                            http://www.mems-exchange.org