[ZODB-Dev] Opening your own parallel transactions/database connections in Zope

[Christian Theune]

Hi,

I hope you're fine with the post getting on zodb-dev.

I'm developing a link checking tool for Zope/Plone and have some
problems with stability in my threading code and hope you can give me
some advise.

The situation: There is a thread that accepts requests for checking a
link through a synchronized queue.  Currently i only allow to have one
of those threads.

The items in the queue contain a pysical path (read: tuple of strings)
for finding the object that represents information about the link in the
ZODB.

Now, in the loop of the thread I handle database connectivity in this
way (copy of the code, a little bit simplified, there are some sleep()
and application relevant lines missing):


-----
    def run2(self):
        # try to get a zodb connection
        work = None
        app=Zope.bobo_application()
        while 1:
            if work:
                work = 0
                try:
                    get_transaction().commit()
                except:
                    get_transaction().abort()
                    check_app._p_jar.sync()
                continue

            try:
                linkchecker, key = self.queue.get(block=0)
            except Queue.Empty:
                continue

            # We got work, so setup the database connection.
            check_app=app
            work=1
            get_transaction().begin()

            linkchecker = app.unrestrictedTraverse('/'.join(linkchecker))

            try:
                item = getattr(linkchecker, key)
            except AttributeError, x:
                continue

            ...here comes the code that actually runs the check, making
changes on the "item" ...
-----

This is a modified way of doing it as I found it in the BlueZBrother
module from blue dynamics. I simplified it a bit by reading various code
files from Zope and ZODB. My Zope version is 2.7 with standard ZODB.

I have the feeling that this code may disrupt some data in the ZODB as
we had a strange thing this weekend (maybe not related to linkchecker
but some other bug in Zope 2.7 or Plone or whatever unstable product has
been there too.) that we were suddenly missing 1 hour at the end of the
ZODB. We had some data modified within Plone (and I ran the linkchecker
every now and then) and suddenly some content was missing. The
transactions weren't visible within the Undo-Log and running fsrecover
on the Data.fs showed that exactly one hour was missing (clocks were set
correctly.)

Additionally sometimes the whole Zope server hangs when running the
linkchecker, but that may be a different problem as it could be through
the lockyness of some calls I do when requesting data from your owner
server...

Thanks for reading and thinking,
Christian 

-- 
Christian Theune, gocept gmbh & co. kg
http://www.gocept.com - ct at gocept.com
fon: 03496 3099112 fax: 03496 3099118 mobile: 0179 7808366