[ZODB-Dev] ZODB and Parallel-Python

[Flavio Coelho]

Hi,

I am trying to use PP with ZODB.

Parallel Python (www.parallelpython.com) provides works as a job
server which starts processes, thus providing a mechanism for the
parallel execution of python code on SMP machines or clusters. Each
job basically calls a function and then executes it (module
dependencies are passed as an import list for each job). For
concurrent writing to the DB I had to find a way to have each of these
processes create their own connections. However, PP, passes variables
to the job-processes as pickles, and the db object is not picklable so
it cannot be passed to PP jobs.

The solution I am trying is to offload the db handling code to a
separate module  that can be added to the import list of a PP job:

dbserve.py
Code:

import ZODB
import ZEO.ClientStorage

class Data:
    def __init__(self):
        #setting up ZEO server
        #os.system('runzeo -a localhost:8008 -f SN.fs &')
        # setup database
        storage = ZEO.ClientStorage.ClientStorage(('localhost',8008))
        self.db = ZODB.DB(storage)
    def getConn(self):
        conn = self.db.open()
        return conn.root()



That seems to be a solution, but fails because now each process tries
to create a ClientStorage object from which to derive a connection.
and the server won't accept it....

An error has occured during the function execution
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/ppworker.py", line 91, in run
    __result = __f(*__args)
  File "<string>", line 4, in cenario
  File "<string>", line 10, in Reps
  File "/home/flavio/Documents/Papers/VacBelief/src/dbserve.py", line
14, in __init__
    self.db = ZODB.DB(storage)
  File "/usr/lib/python2.5/site-packages/ZODB3-3.8.0-py2.5-linux-i686.egg/ZODB/DB.py",
line 270, in __init__
    storage.load(z64, '')
  File "/usr/lib/python2.5/site-packages/ZEO/ClientStorage.py", line
757, in load
    p, s, v, pv, sv = self._server.zeoLoad(oid)
  File "/usr/lib/python2.5/site-packages/ZEO/ServerStub.py", line 82, in zeoLoad
    return self.rpc.call('zeoLoad', oid)
  File "/usr/lib/python2.5/site-packages/ZEO/zrpc/connection.py", line
484, in call
    r_flags, r_args = self.wait(msgid)
  File "/usr/lib/python2.5/site-packages/ZEO/zrpc/connection.py", line
574, in wait
    raise DisconnectedError()


Anyone has any experience with ZODB and PP? any sugestions?
In summary: I need to expose a DB Object to multiple processes so that
they can derive their own connections But this object cannot passed at
job-creation time because they are not pickleable, and they apparently
cannot be created by each job because I cannot have multiple
ClientStorage objects targetting the same phisical zeo server (is this
a real limitation?)

thanks in advance,

Flávio C. Coelho