[ZODB-Dev] avoiding conflicts.

[Terry Kerr]

Hi,

Sorry for the large post.

I have a problem with conflicts with my zope app.  I have a method that is 
called from the web, and does quite a few things, updating attributes on global 
parent objects as well as attributes on the object the method is defined on, and 
  the method also posts off information to other systems via a HTTP.

The problem I have is that this method can take up to 10 seconds to complete, 
mainly due to the HTTP communication with the external systems, and during high 
load periods, this method may be called by several different users every second, 
thus, write conflicts emerge as the method updates the attributes on the global 
parent objects.  Zope's way of attempting to resolve the conflict is to re-run 
the method. This works fine, and the conlfict is evenutally resolved even if 
after several goes.  The problem however is that because the method is run 
several times, the information is posted to the external systems as many times 
which in my case is very very bad!

I have done some things to help avoid conflicts like moving the chunks of code 
that change attributes on parent objects at the beginning of the method and 
followed the code with a get_transaction().commit().  This seems to help reduce 
the number of conflict errors since the troublesome attrs are updated quickly 
and commited.  However, this is still not an ideal situation.

I have looked through ZODB developer docs, and have seen posts related to 
conflict resolution like defining _p_resolveConflict().  However, this idea is 
not really relivant to to my application because the attributes that I am 
updating are not simple counters, they are large sometimes complex lits and 
mappings.

One solution I guess would be to redesign the way I do this and have my troubles 
attrs moved across to their own objects which has a _p_resolveConflict() to 
resolve the conflicts.  Although, I see this as a difficult task.

Running Zope in single thread mode is not an option due to scalibility ;-(

The solution I have been experiementing with is a simple thread lock on the 
method to prevent more than one thread (application user) from running the 
method concurrently.  I have been told that this is a no-no, but I don't see why 
I shouldn't do it. The key to doing this is syncroinising the database 
connection immediately after the thread lock has been acquired, which I have not 
seen any mension of in postings related to this issue.  The code I am using is:


lock = thread.allocate_lock()

class myClass:

     def myMethod(self):
         lock.acquire()
         self._p_jar.sync()
         update some global attrs here()
         many slow bits of code here()
         get_transaction().commit()
         lock.release()
         return data

 From my initial experiements, the code works fine and I get no conflicts. The 
only problem I can see with this solution is that it won't be safe across ZEO 
clients if I ever run the application in a ZEO environment.  I guess that is a 
fairly major limitation!

If all ZEO clients were running on the same machine to take advantage of a 
multiple CPU machine, then I guess I could use a file lock in the file system 
rather than a thread lock, but this wouldn't help if the ZEO clients were 
running on different machines.

Any suggestions or comments would be much appreciated.


Please respond to terry@bizarsoftware.com.au since I am not on any lists.

terry


-- 
Terry Kerr (terry@bizarsoftware.com.au)
Chief Technical Officer
Bizar Software Pty Ltd (www.bizarsoftware.com.au)
+61 3 9530 9182