[ZODB-Dev] ConflictError Programming Recipe Required

[Kapil Thangavelu]

the root of the problem here is the use of non transactional resources within 
a transactional framework coupled with zope's behavior to retry on conflict 
errors. my suggestion in this case would be to give the email sending a 
transactional interface so that retries don't cause problems. the algorithm 
for which is to drop emails to an fs queue in a manner which respects txn 
results, and have a cron job inject to the mta queue or send directly.

which amounts basically to backporting the zope3 mail package which does txn 
integration (the z3 txn interfaces are much different than z2 though).

http://cvs.zope.org/Zope3/src/zope/mail

other things that might be worthwhile taking a look at

http://zope.org/Members/k_vertigo/Products/TransactionalFileSystem

and

http://dataflake.org/software/maildrophost

as for general patterns avodiing for write conflicts, is to make the unit of 
change more finite, ie distribute the data across more persistent objects, 
and try to avoid mass changes or hot spots.

ie. taken for example a room reservation system which stores all the room 
reservations as a dictionary on a single persistent object and has frequent 
updates to individual reservations (new attendee, new equip, etc). this type 
of system would be ripe for conflict errors on load, changing the container 
to a btree from a dict and making the individual reservations persistent 
objects would greatly improve the load distribution. this is more guideline 
than pattern as the actual design would be greatly influenced by a particular 
application's usage and load patterns.

incidentally the python twain module is very nice, thanks :-)

cheers,

kapil

On Tuesday 11 November 2003 11:20, Kevin Gill wrote:
> Today, during an important task, the page I was loading raised
> ZODB.POSException.ConflictError.
>
> It appears that the Zope system, being quite sensible tried the page three
> more times before returning the exception to me. I cannot reproduce the
> exception and the next time I ran the same page it did not occur.
>
> This was extremely embarassing for me. The page in question was to send an
> email and text message to 150 people. Instead they now have received 5
> texts / emails each. [ 5 = original + 3 retries + second attempt ].
>
> Clearly, the fault is my own. I should not have built a transaction which
> is reversable which has the side effect of sending emails / texts.
>
> I am looking for advice in how to program this type of problem, a recipe if
> you like.
>
>
> There seems to me to be a number of theoretical options....
>
> 1.	commit after each email / text message
> 2.	run a second thread where the first thread updates the database and the
> second database sends the emails / text messages
> 3.	Trap the error and resolve it
>
>
> However, these all seem to have problems (to me)....
>
> 1.	I cannot seem to end the transaction and start a new-one while serving a
> single page (and if I could, I am not sure that it would be sensible).
>
> 2.	This simply transfers the same problem onto a second thread. The second
> thread is less visible and will reduce the maintainability ./ stability of
> the solution.
>
> 3.	I believe that it cannot be resolved. It occurred during a commit().
>
>
> QUESTION : WHAT RECIPIES DO YOU USE IN A SITUATION LIKE THIS?
>
> Thanks
>
> Kevin
>
>
>
>
> For the enthuastic here is my Zope Info....
>
> Zope 2.6.1
> Linux: Debian Woody
> Database: Postgres
>
> And my traceback ....
> Site Error
> An error was encountered while publishing this resource.
>
> ZODB.POSException.ConflictError
>
> Sorry, a site error occurred.
>
> Traceback (innermost last):
>
> Module ZPublisher.Publish, line 150, in publish_module
> Module ZPublisher.Publish, line 127, in publish
> Module ZPublisher.Publish, line 127, in publish
> Module ZPublisher.Publish, line 127, in publish
> Module ZPublisher.Publish, line 122, in publish
> Module Zope.App.startup, line 142, in zpublisher_exception_hook
> Module ZPublisher.Publish, line 102, in publish
> Module Zope.App.startup, line 200, in commit
> Module ZODB.Transaction, line 235, in commit
> Module ZODB.Transaction, line 349, in _commit_objects
> Module ZODB.Connection, line 391, in commit
> __traceback_info__: (('BTrees._IOBTree', 'IOBucket'),
> '\x00\x00\x00\x00\x00\x00\x03\xaa', '')
> Module Products.TemporaryFolder.TemporaryStorage, line 134, in store
> ConflictError: database conflict error (oid 00000000000003aa, serial was
> 0350f384964c1baa, now 0350f38180f99a44)
>
>
>
>
>
> _______________________________________________
> For more information about ZODB, see the ZODB Wiki:
> http://www.zope.org/Wikis/ZODB/
>
> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> http://mail.zope.org/mailman/listinfo/zodb-dev