[ZODB-Dev] Type references

[Jim Fulton]

The instance record format changed in ZODB 3.3.

Prior to ZODB 3.3, the record format was:

   pickle1:
      ((class_.__module__, class_.__name__), args)
   pickle2
      state

where class_ is the instance's class, and state is the instance's
state (result of calling __getstate__).

In othwer words, the class of an instance was pickled by name.

In ZODB 3.3, the record format changed to:

   pickle1:
      (class_, args)
   pickle2
      state

There was a similar change to the object reference format, from:

      (oid, (class_.__module__, class_.__name__))

to:

      (oid, class)

Instance classes are pickled directly.  For traditional classes,
Python's standard class-pickling strategy ends up pickling these by
name, as "globals".  For instances of persistent classes, however, a
persistent reference to the class is stored in pickle1.

Using normal persistent references for references to persistent
classes has the advantage of simplicity.  If references to persistent
classes were pickled by name, then we'd need some sort of name
registry, which would be application specific.

Using normal persistent references for references to persistent
classes has some significant downsides, at least for us:

- If the export/import mechanism is used to copy objects, we'll end up
   copying the object classes.  This is almost certainly not what what
   we want. We can fix the export/import mechanism to handle this, by
   changing the record format used for export/import, if we have to.
   Note that the problem transcends instance references.  There is a
   similar issue with references from a persistent class to it's base
   classes. If we export a persistent class, we almost certainly don't
   want to export it's base classes.  Certainly, that's true for a copy
   operations.

- The implementation of an instance's class and the instances is very
   tightly bound.  In Zope 3, we'd like to be able to use persistent
   classes to prototype objects and later convert the persistent
   classes to normal global Python classes.  Direct object references
   make this difficult.

For ZODB 3.4, I'd like to add an option to use indirect *type*
references.  If this option is enabled, then *all* references to a
type would be stored by name.  We'd use the
persistent_id/persistent_load mechanism to implement this.  Any time
we encounter a type in persistent_id, we'll return:

    (None, (thetype.__module__, thetype.__name__))

Our persistent load mechanism would work pretty much as it does now,
except that, if the oid is None (first tuple item is None), as above,
it would simpley return the global object resolved from the module
name and global name, possibly using the existing class-factory hook.

We wouldn't bother doing this for classic classes. We'd only use this
mechanism for types.

This isn't, technically, a change in the record or reference formats.
It is a change to the way types get pickled.  Types will be pickled by
name, even for persistent classes.

This only applies to type refences. When a persistent type's own
record is written to the database, then it will use the normal record
format.

Note that, if we ever embrace pickle protocol 2, and assign codes to
key types like BTrees, then we'd adjust the mechanism to treat those
types differently.

Thoughts? Questions?

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org