[ZODB-Dev] notes re trying to run ZODB on PyPy

[David Glick]

Alan Runyan asked me to post my notes from my attempt to get ZODB 
running on PyPy. It's very much an experimental work in progress (that I 
got distracted from), but hopefully this is at least useful to anyone 
else who wants to attempt the same thing.

I first tried building ZODB trunk (with C extensions). Ran into the 
following issues:
* PyPy is missing an implementation of the _Py_ForgetReference macro 
which is used by ZODB's persistent/cPickleCache.c
* The check_argument_cmp check in BTrees/objectkeymacros.h fails for the 
root object of the database on PyPy
* I got a PyPy exception re wrap_objobjargspec which appeared to be some 
issue with using the __setitem__ slot of an extension type

At this point I gave up on trying to build the C extensions, and instead 
did the following:
1. Started with Jim's python-btrees branch 
(http://svn.zope.org/repos/main/ZODB/branches/jim-python-btrees)
2. Replaced the 'persistent' package with a checkout of Tres' 
pure-Python persistent implementation 
(http://svn.zope.org/repos/main/persistent/trunk/persistent)
3. Set exts = [] in setup.py to disable all C extensions when installing.

Next I worked around some issues in the pure-Python branches:
* BTrees.___BTree.fsBucket class was missing toString and fromString 
methods, which I implemented.
* persistent.picklecache.PickleCache class was missing 
update_object_size_estimation method (and indeed the whole 
limit-cache-size-by-bytes feature). I added it as a no-op stub.
* persistent.picklecache.PickleCache.__setitem__ was raising a KeyError 
for a duplicate oid even when trying to set the same object already 
stored under that key
* persistent.TimeStamp did not give the Python implementation, which I 
fixed by an import of persistent.timestamp + module alias if 
persistent.TimeStamp (the C module) is missing
* persistent.pyPersistence.Persistent's __new__ needs to accept *args 
and **kw

At this point I was able to start up the Pyramid ZODB scaffold and add 
an object to the DB root.

Next I tried to get zodbshootout running as a benchmark. I didn't get 
this working, as I got stuck on an issue with the pure-Python 
PersistentMapping getting committed without its object state (appeared 
to be something to do with failing to get unghosted before it gets 
committed). Along the way, I noticed a couple incompatibilities with 
PyPy because PyPy implements the cPickle module as an alias to the 
pure-Python pickle module. This leads to some problems due to slight 
semantic differences between cPickle and pickle. In particular, 
cPickle.Pickler accepts a lone protocol argument; pickle.Pickler does 
not (leading to an error in ZEO.zrpc.marshal.encode). And 
pickle.Unpickler does not have a noload method, which 
ZODB.serialize.referencesf uses to obtain the referenced oids from a 
pickle without the overhead of unpickling.

That's as far as I got. Hope this is helpful to the next guy. :)

cheers,
David


----------		
David Glick
 Web Developer
 davidglick at groundwire.org
 206.286.1235x32

Online tools and strategies for the environmental movement. 

Sign up for our newsletter: http://www.groundwire.org/email-capture