[ZODB-Dev] Re: ReadConflictError while re-indexing big fat in dex

[sean.upton at uniontrib.com]

Casey Duncan wrote:
> > I presume you mean index objects in batches with a commit 
> in between?
> > But that creates an inconsistent state surely?
> 
> That means some objects are reindexed and others aren't. 
> Whether that is acceptable or not is up to you.

This also has a side-effect: storage bloat between packing. As you commit
more frequently, the number of stale copies of indexes goes up, sometimes in
ways that are quite surprising - or quite painful.  This also increases pack
time, which is salt in the wounds.

> Another option is to use versions, which will lock the 
> catalog indexes that are changed in the version and prevent 
> any concurrent updates to it. This will prevent read conflict 
> errors, but will cause version lock errors if other clients 
> attempt to write.

Wow - a practical use case for versions!

Using versions as a lock isn't something I've thought to do before.  So you
have two processes writing to the indexes - A and B - A is more important
and starts first, then locks the indexes by entering a version then touching
each of them; B runs, tries to update, and fails with a lock error.  Does
this sound right?  If so, at least this avoids both the RCE and the bloat
asssociated with the alternative of increasing number/frequency of commits?

Sean

+----------------------------------------------------------+
 Sean Upton                              SignOnSanDiego.com 
 Site Technology Supervisor     The San Diego Union-Tribune 
 619.718.5241                 sean.upton at signonsandiego.com 
 2375 Northside Dr #300                 San Diego, CA 92108

  Plone Powered!  plone.org  ++  python.org  ++  zope.org   
+----------------------------------------------------------+