[ZODB-Dev] ConflictError Programming Recipe Required

Tue Nov 18 18:46:53 EST 2003

Thanks for the response. Kapil fairly nailed the issue. From rooting around 
the huge volume of Zope documentation, I conclude:

A:  The ConflictError is a result of design problems in my software. Just as I 
have to program to avoid deadlocks in SQL, so I also have to program to avoid 
this sort of problem in Zope.

B:   My functionality is essentially two processes, update database and send 
emails, with a serial dependency. In particular, where a database update 
suceeds I should have a relatively high certainty of sending the email. This 
is the basis of products such as IBMs MQ Series.

These are lessons I learnt many years ago, and forgot as soon as I had some 
new technology to play with.

Surveying the solutions available in Zope, there is not much. Essentially, in 
these circumstances, people seem to take the second task out of zope to the 
OS. This is not an option for me with a hosted domain, and it lacks elegance.

I have therefore decided to build a solution within the Zope framework. I put 
together a simple transactional queue. The concept is that the interactive 
session writes messages to the queue, which are only visible when this 
transaction commits(). A second thread can take items from the queue. These 
will only be physically removed when the readers thread commit(). A single 
transaction can only enqueue items or dequeue them, not both.

If anyone is having the same problems as me, the Transactional Queue is in my 
directory on Zope.org.

I also ignored all the warnings from ZopeLabs 
(http://www.zopelabs.com/cookbook/1058719806), and put in a background thread 
to do my processing.

Thanks Everyone for the Help

Kevin

> incidentally the python twain module is very nice, thanks :-)

Excellent Kapil, Your's is the first feedback I had in over a year. Some day 
I'll get around to that Python 2.3 build....

On Thursday 13 November 2003 01:31, you wrote:
> the root of the problem here is the use of non transactional resources
> within a transactional framework coupled with zope's behavior to retry on
> conflict errors. my suggestion in this case would be to give the email
> sending a transactional interface so that retries don't cause problems. the
> algorithm for which is to drop emails to an fs queue in a manner which
> respects txn results, and have a cron job inject to the mta queue or send
> directly.
>
> which amounts basically to backporting the zope3 mail package which does
> txn integration (the z3 txn interfaces are much different than z2 though).
>
> http://cvs.zope.org/Zope3/src/zope/mail
>
> other things that might be worthwhile taking a look at
>
> http://zope.org/Members/k_vertigo/Products/TransactionalFileSystem
>
> and
>
> http://dataflake.org/software/maildrophost
>
> as for general patterns avodiing for write conflicts, is to make the unit
> of change more finite, ie distribute the data across more persistent
> objects, and try to avoid mass changes or hot spots.
>
> ie. taken for example a room reservation system which stores all the room
> reservations as a dictionary on a single persistent object and has frequent
> updates to individual reservations (new attendee, new equip, etc). this
> type of system would be ripe for conflict errors on load, changing the
> container to a btree from a dict and making the individual reservations
> persistent objects would greatly improve the load distribution. this is
> more guideline than pattern as the actual design would be greatly
> influenced by a particular application's usage and load patterns.
>
> incidentally the python twain module is very nice, thanks :-)
>
> cheers,
>
> kapil
>
> On Tuesday 11 November 2003 11:20, Kevin Gill wrote:
> > Today, during an important task, the page I was loading raised
> > ZODB.POSException.ConflictError.
> >
> > It appears that the Zope system, being quite sensible tried the page
> > three more times before returning the exception to me. I cannot reproduce
> > the exception and the next time I ran the same page it did not occur.
> >
> > This was extremely embarassing for me. The page in question was to send
> > an email and text message to 150 people. Instead they now have received 5
> > texts / emails each. [ 5 = original + 3 retries + second attempt ].
> >
> > Clearly, the fault is my own. I should not have built a transaction which
> > is reversable which has the side effect of sending emails / texts.
> >
> > I am looking for advice in how to program this type of problem, a recipe
> > if you like.
> >
> >
> > There seems to me to be a number of theoretical options....
> >
> > 1.	commit after each email / text message
> > 2.	run a second thread where the first thread updates the database and
> > the second database sends the emails / text messages
> > 3.	Trap the error and resolve it
> >
> >
> > However, these all seem to have problems (to me)....
> >
> > 1.	I cannot seem to end the transaction and start a new-one while serving
> > a single page (and if I could, I am not sure that it would be sensible).
> >
> > 2.	This simply transfers the same problem onto a second thread. The
> > second thread is less visible and will reduce the maintainability ./
> > stability of the solution.
> >
> > 3.	I believe that it cannot be resolved. It occurred during a commit().
> >
> >
> > QUESTION : WHAT RECIPIES DO YOU USE IN A SITUATION LIKE THIS?
> >
> > Thanks
> >
> > Kevin
> >
> >
> >
> >
> > For the enthuastic here is my Zope Info....
> >
> > Zope 2.6.1
> > Linux: Debian Woody
> > Database: Postgres
> >
> > And my traceback ....
> > Site Error
> > An error was encountered while publishing this resource.
> >
> > ZODB.POSException.ConflictError
> >
> > Sorry, a site error occurred.
> >
> > Traceback (innermost last):
> >
> > Module ZPublisher.Publish, line 150, in publish_module
> > Module ZPublisher.Publish, line 127, in publish
> > Module ZPublisher.Publish, line 127, in publish
> > Module ZPublisher.Publish, line 127, in publish
> > Module ZPublisher.Publish, line 122, in publish
> > Module Zope.App.startup, line 142, in zpublisher_exception_hook
> > Module ZPublisher.Publish, line 102, in publish
> > Module Zope.App.startup, line 200, in commit
> > Module ZODB.Transaction, line 235, in commit
> > Module ZODB.Transaction, line 349, in _commit_objects
> > Module ZODB.Connection, line 391, in commit
> > __traceback_info__: (('BTrees._IOBTree', 'IOBucket'),
> > '\x00\x00\x00\x00\x00\x00\x03\xaa', '')
> > Module Products.TemporaryFolder.TemporaryStorage, line 134, in store
> > ConflictError: database conflict error (oid 00000000000003aa, serial was
> > 0350f384964c1baa, now 0350f38180f99a44)
> >
> >
> >
> >
> >
> > _______________________________________________
> > For more information about ZODB, see the ZODB Wiki:
> > http://www.zope.org/Wikis/ZODB/
> >
> > ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> > http://mail.zope.org/mailman/listinfo/zodb-dev