[Checkins] SVN: keas.pbpersist/trunk/src/keas/pbpersist/ Added more documentation and increased test coverage.

Shane Hathaway shane at hathawaymix.org
Thu Jan 22 03:26:26 EST 2009


Log message for revision 94924:
  Added more documentation and increased test coverage.
  

Changed:
  U   keas.pbpersist/trunk/src/keas/pbpersist/README.txt
  U   keas.pbpersist/trunk/src/keas/pbpersist/pbformat.py

-=-
Modified: keas.pbpersist/trunk/src/keas/pbpersist/README.txt
===================================================================
--- keas.pbpersist/trunk/src/keas/pbpersist/README.txt	2009-01-22 05:24:28 UTC (rev 94923)
+++ keas.pbpersist/trunk/src/keas/pbpersist/README.txt	2009-01-22 08:26:25 UTC (rev 94924)
@@ -1,45 +1,87 @@
 
 
-Tests of keas.pbpersist
-=======================
+Tests and Documentation of keas.pbpersist
+=========================================
 
-Create a PContact instance.  PContact is a Persistent class that stores
-its state in a protobuf instance.
+Overview
+--------
 
-    >>> from keas.pbpersist.tests import PContact
-    >>> bob = PContact()
+The keas.pbpersist package provides a way to store ZODB objects using
+Google Protocol Buffer encoding rather than pickling.  The Protocol
+Buffer format is more limited than pickles, but it is also simpler,
+better documented, and not dependent on any particular programming
+language.
 
-Can't serialize it yet.
+Only Persistent objects that use the ProtobufState metaclass are eligible
+for this kind of serialization, so applications that want to take advantage
+of keas.pbpersist need to be refactored at the model level.  However,
+applications can be refactored gradually, since protobuf
+encoded objects and pickled objects can coexist and hold references
+to each other within a single database.
 
-    >>> bob.__getstate__()
-    Traceback (most recent call last):
-    ...
-    EncodeError: Required field ContactPB.name is not set.
+This package is expected to be fully compatible with most ZODB
+storages, including FileStorage, ZEO, RelStorage, and any other storage
+that treats object records as opaque binary streams.
 
-Fill out the required fields, then verify we can serialize it.
 
-    >>> bob.name = u'Bob'
-    >>> bob.__getstate__()
-    ('\x08\x01\x12\x03Bob', {})
+How to Use This Package
+-----------------------
 
-Register the protobuf serializer with ZODB.
+Register the protobuf serializer with ZODB.  This only needs to happen
+once per application run, but it does no harm if it happens more than once.
 
     >>> from keas.pbpersist.pbformat import register
     >>> register()
+    >>> register()
 
-Set up an in-memory database and put the PContact in it.
+Create a PContact instance.  PContact is a simple Persistent class that
+stores its state in a protobuf message.  Fill out the required field.
 
+    >>> from keas.pbpersist.tests import PContact
+    >>> bob = PContact()
+    >>> bob.name = u'Bob'
+
+Set up an in-memory object database and put the PContact in it.
+
     >>> from ZODB.DemoStorage import DemoStorage
     >>> from ZODB.DB import DB
     >>> import transaction
     >>> storage = DemoStorage()
-    >>> db = DB(storage)
+    >>> db = DB(storage, database_name='main')
     >>> conn1 = db.open()
     >>> conn1.root()['bob'] = bob
     >>> transaction.commit()
 
-Read the PContact from another connection.
+Let's take a peek at what got stored in the database.  Look Ma, no pickles!
 
+    >>> data, serial = storage.load(bob._p_oid, '')
+    >>> data
+    '{protobuf}\n \n\x14keas.pbpersist.tests\x12\x08PContact\x12\x07\x08\x01\x12\x03Bob'
+
+Here is a demonstration of how to crack open that record using nothing but
+language independent operations.  The ObjectRecord message holds the object's
+class, state, and references; the state field holds a class-specific
+protobuf message.
+
+    >>> from keas.pbpersist.persistent_pb2 import ObjectRecord
+    >>> rec = ObjectRecord()
+    >>> rec.MergeFromString(data[10:])  # skip the {protobuf} prefix
+    43
+    >>> rec.class_meta.module_name
+    u'keas.pbpersist.tests'
+    >>> rec.class_meta.class_name
+    u'PContact'
+    >>> len(rec.references)
+    0
+    >>> pbuf = PContact.protobuf_type()
+    >>> pbuf.MergeFromString(rec.state)
+    7
+    >>> pbuf.name
+    u'Bob'
+
+Ok, I'm happy that's possible, but now let's cut the noise and use nice
+ZODB operations to read the object.
+
     >>> conn2 = db.open()
     >>> bob2 = conn2.root()['bob']
     >>> bob2.name
@@ -47,9 +89,9 @@
     >>> conn2.close()
 
 Create another PContact and assign that contact to be a guardian for Bob.
-[Ed: I sure don't like the way we manage references right now.  I'd much
-rather see "bob.guardians.add().target = alice", but that is not yet
-possible.]
+The add() method below is part of Google's protobuf API.  The
+protobuf_refs attribute is provided by the ProtobufState metaclass
+in keas.pbstate.
 
     >>> alice = PContact()
     >>> alice.name = u'Alice'
@@ -57,15 +99,148 @@
     >>> bob.protobuf_refs.set(ref, alice)
     >>> transaction.commit()
 
-Find Alice through Bob in another connection.
+[Ed: I sure don't like the way we manage references right now.  I'd much
+rather see "bob.guardians.add().target = alice", but that is not yet
+possible.]
 
+Get Alice through Bob in another ZODB connection.
+
     >>> conn2 = db.open()
     >>> bob2 = conn2.root()['bob']
     >>> alice2 = bob2.protobuf_refs.get(bob2.guardians[0])
     >>> alice2.name
     u'Alice'
+
+Note that alice2 is a duplicate of alice, not the same object, because
+the two come from different ZODB connections.
+
+    >>> alice is alice2
+    False
+
+We also have the option of referring to objects by OID, but that
+is rarely necessary.
+
+    >>> conn3 = db.open()
+    >>> bob3 = conn3[bob._p_oid]
+
+ZODB loads all objects lazily, including objects serialized using
+this package.  When a persistent object's _p_changed
+attribute is None, the object is in the "ghost" state; it does not
+yet contain data.  Accessing an attribute transparently loads the object
+state.
+
+    >>> bob3._p_changed is None
+    True
+    >>> bob3.name
+    u'Bob'
+    >>> bob3._p_changed
+    False
+
+Clean up the extra connections.
+    
+    >>> conn3.close()
     >>> conn2.close()
 
+
+Packing
+-------
+
+Storages need to be able to follow chains of references in order to pack,
+so let's make sure the referencesf() function still does what it should.
+
+    >>> from ZODB.serialize import referencesf
+    >>> data, serial = storage.load(conn1.root()._p_oid, '')
+    >>> referencesf(data) == [bob._p_oid]
+    True
+    >>> data, serial = storage.load(bob._p_oid, '')
+    >>> referencesf(data) == [alice._p_oid]
+    True
+    >>> data, serial = storage.load(alice._p_oid, '')
+    >>> referencesf(data)
+    []
+
+
+Reference Capabilities And Limitations
+---------------------------------------
+
+Objects serialized using this package can refer to any other Persistent
+objects.  (Note that we already demonstrated that pickled objects
+can refer to protobuf serialized objects when we added "bob" to the
+root object in the database.)
+
+    >>> from persistent.mapping import PersistentMapping
+    >>> len(bob.guardians)
+    1
+    >>> ref = bob.guardians.add()
+    >>> bob.protobuf_refs.set(ref, PersistentMapping())
+    >>> transaction.commit()
+    >>> len(bob.guardians)
+    2
+
+However, objects serialized using this package can not hold a reference
+to non-Persistent objects.  (This is a limitation of keas.pbpersist, but
+not a limitation of keas.pbstate, which does not depend on ZODB.)  We don't
+detect bad references until the first phase of transaction commit.
+
+    >>> len(bob.guardians)
+    2
+    >>> ref = bob.guardians.add()
+    >>> bob.protobuf_refs.set(ref, {})
+    >>> len(bob.guardians)
+    3
+    >>> transaction.commit()
+    Traceback (most recent call last):
+    ...
+    POSError: Protobuf reference target is not a Persistent object: {}
+
+Aborting the transaction undoes the damage.
+
+    >>> transaction.abort()
+    >>> len(bob.guardians)
+    2
+
+We can make weak references, though!
+
+    >>> from persistent.wref import WeakRef
+    >>> ref = bob.guardians.add()
+    >>> bob.protobuf_refs.set(ref, WeakRef(PersistentMapping()))
+    >>> transaction.commit()
+    >>> len(bob.guardians)
+    3
+
+We can also make cross-database references.
+
+    >>> storage_beta = DemoStorage()
+    >>> db_beta = DB(storage_beta, database_name='beta', databases=db.databases)
+    >>> root_beta = conn1.get_connection('beta').root()
+    >>> ref = bob.guardians.add()
+    >>> bob.protobuf_refs.set(ref, root_beta)
+    >>> transaction.commit()
+
+Access all of those references in another connection.
+
+    >>> conn2 = db.open()
+    >>> bob2 = conn2.root()['bob']
+    >>> bob2.protobuf_refs.get(bob2.guardians[0])
+    <keas.pbpersist.tests.PContact object at ...>
+    >>> bob2.protobuf_refs.get(bob2.guardians[1])
+    {}
+    >>> bob2.protobuf_refs.get(bob2.guardians[1])._p_jar is conn2
+    True
+    >>> bob2.protobuf_refs.get(bob2.guardians[2])
+    <persistent.wref.WeakRef object at ...>
+    >>> bob2.protobuf_refs.get(bob2.guardians[2])()
+    {}
+    >>> bob2.protobuf_refs.get(bob2.guardians[3])
+    {}
+    >>> bob2.protobuf_refs.get(bob2.guardians[3])._p_jar is conn2
+    False
+    >>> conn2.close()
+
+
+Clean Up
+--------
+
 Close the object database.
 
     >>> transaction.abort()

Modified: keas.pbpersist/trunk/src/keas/pbpersist/pbformat.py
===================================================================
--- keas.pbpersist/trunk/src/keas/pbpersist/pbformat.py	2009-01-22 05:24:28 UTC (rev 94923)
+++ keas.pbpersist/trunk/src/keas/pbpersist/pbformat.py	2009-01-22 08:26:25 UTC (rev 94924)
@@ -176,6 +176,6 @@
         i.next()
         return i.next()
 
-
+protobuf_format = ProtobufFormat()
 def register():
-    register_format(StateTuple.serial_format, ProtobufFormat())
+    register_format(StateTuple.serial_format, protobuf_format)



More information about the Checkins mailing list