SV: SV: [ZODB-Dev] Flush

[Shane Hathaway]

Magnus Heino wrote:

> Ive got a persistent object being published by ZPublisher. This object has a
> start method that does start_new_thread(self.method, ()). self.method does
> not terminate before a stop method is called. While running, it performs
> come calculations. This way I can view what the thread is doing, by browsing
> with ZPublisher. However, it seems as if the object is flushed from memory,
> leaving a thread running... later on when I call some method on the object,
> it is loaded from disk into memory again. In setstate, I look at a attribute
> self.__isRunning and calls start_new_thread(self.method, ()) if it was
> running when before. This way the thread will start if it was running when
> the app was closed. But also if the object was flushed... :-P


Whoa, that won't work.  Under the current ZODB model, every thread has 
its own copy of objects.  When you start a new thread, you have to load 
new copies.  Otherwise you'll get rampant race conditions.

I guess there is some long-running operation you want to perform on the 
object every time it's loaded?  Note that ZODB reserves the right to 
unload objects as frequently as it needs; in fact, with the cache 
optimizations Toby has proposed, I think objects will be unloaded more 
frequently than today.

I'm going to assume there is some long-running operation you want to 
perform only once on the object, and that if the operation hasn't been 
performed, __setstate__ will launch a thread to do it.  Here's one way 
to write it for Zope (untested):


class MyClass (Persistent):

   _completed = 0
   _working = 0

   def beforeWork(self):
     if self._working:
       # Another thread is working on it.
       return 0
     self._working = 1
     return 1

   def work(self):
     # Do work and change self

     # Indicate the work is completed
     self._completed = 1
     self._working = 0

   def __setstate__(self, data):
     MyClass.inheritedAttribute('__setstate__')(self, data)
     if not self._completed and not self._working:
         start_new_thread(traverseAndCall,
           (self.getPhysicalPath(), 'beforeWork', 'work'))


def traverseAndCall(path, method_name_1, method_name_2):
     import Zope
     get_transaction().begin()
     get_transaction().note('/'.join(path) + '/' + method_name_1)
     app = Zope.app()
     ob = app.restrictedTraverse(path)
     m = getattr(ob, method_name_1)
     if not m():
       return  # Another thread is working on it.
     get_transaction().commit()  # Notify that work has started
     get_transaction().begin()
     get_transaction().note('/'.join(path) + '/' + method_name_2)
     m = getattr(ob, method_name_2)
     m()
     get_transaction().commit()  # Commit the work



You will most likely have to adjust that quite a bit, but effectively 
the second thread has its own copy of the database and it tries to 
prevent more than one thread from performing the work.  This works even 
if you mix in ZEO, though there is still a race condition.  It could be 
fixed, but this is just an example.

Shane