[ZODB-Dev] notes re trying to run ZODB on PyPy
davidglick at groundwire.org
Sat Sep 24 19:02:11 EST 2011
Alan Runyan asked me to post my notes from my attempt to get ZODB
running on PyPy. It's very much an experimental work in progress (that I
got distracted from), but hopefully this is at least useful to anyone
else who wants to attempt the same thing.
I first tried building ZODB trunk (with C extensions). Ran into the
* PyPy is missing an implementation of the _Py_ForgetReference macro
which is used by ZODB's persistent/cPickleCache.c
* The check_argument_cmp check in BTrees/objectkeymacros.h fails for the
root object of the database on PyPy
* I got a PyPy exception re wrap_objobjargspec which appeared to be some
issue with using the __setitem__ slot of an extension type
At this point I gave up on trying to build the C extensions, and instead
did the following:
1. Started with Jim's python-btrees branch
2. Replaced the 'persistent' package with a checkout of Tres'
pure-Python persistent implementation
3. Set exts =  in setup.py to disable all C extensions when installing.
Next I worked around some issues in the pure-Python branches:
* BTrees.___BTree.fsBucket class was missing toString and fromString
methods, which I implemented.
* persistent.picklecache.PickleCache class was missing
update_object_size_estimation method (and indeed the whole
limit-cache-size-by-bytes feature). I added it as a no-op stub.
* persistent.picklecache.PickleCache.__setitem__ was raising a KeyError
for a duplicate oid even when trying to set the same object already
stored under that key
* persistent.TimeStamp did not give the Python implementation, which I
fixed by an import of persistent.timestamp + module alias if
persistent.TimeStamp (the C module) is missing
* persistent.pyPersistence.Persistent's __new__ needs to accept *args
At this point I was able to start up the Pyramid ZODB scaffold and add
an object to the DB root.
Next I tried to get zodbshootout running as a benchmark. I didn't get
this working, as I got stuck on an issue with the pure-Python
PersistentMapping getting committed without its object state (appeared
to be something to do with failing to get unghosted before it gets
committed). Along the way, I noticed a couple incompatibilities with
PyPy because PyPy implements the cPickle module as an alias to the
pure-Python pickle module. This leads to some problems due to slight
semantic differences between cPickle and pickle. In particular,
cPickle.Pickler accepts a lone protocol argument; pickle.Pickler does
not (leading to an error in ZEO.zrpc.marshal.encode). And
pickle.Unpickler does not have a noload method, which
ZODB.serialize.referencesf uses to obtain the referenced oids from a
pickle without the overhead of unpickling.
That's as far as I got. Hope this is helpful to the next guy. :)
davidglick at groundwire.org
Online tools and strategies for the environmental movement.
Sign up for our newsletter: http://www.groundwire.org/email-capture
More information about the ZODB-Dev