[ZODB-Dev] Re: ReadConflictError while re-indexing big fat index

[Casey Duncan]

On Mon, 02 Aug 2004 15:55:57 +0100
Chris Withers <chris at simplistix.co.uk> wrote:

> Casey Duncan wrote:
> >>1. What is the above error message really trying to tell me?
> >>    (it's a pretty obscure message ;-)
> > 
> > It is telling you that an ISet object in the PathIndex was changed
> > and commited by another client since the transaction began. In order
> > to protect data integrity, ZODB does not allow such "dirty reads",
> > instead it raises a ReadConflict error.
> 
> Heh, now that I finally understand this, it seems like ZODB isn't as 
> perfect as I thought ;-) You kinda take this stuff for granted with
> any RDB, the I in ACID and all that...

RDBMS usually solve this problem with locks (i.e., select for update or
select for insert). This can cause similar concurrency problems, but
generally they just surface as poor performance to the application.
 
> > You probably will need to reindex incrementally instead of in one
> > big transaction. You might also need to "throttle back" the reindex
> > speed so that it does not cause excessive read conflicts in the
> > other clients.
> 
> But how do I do either of those things?

You write some python, of course ;^) Not much though.
 
> I presume you mean index objects in batches with a commit in between?
> But that creates an inconsistent state surely?

That means some objects are reindexed and others aren't. Whether that is
acceptable or not is up to you.

Another option is to use versions, which will lock the catalog indexes
that are changed in the version and prevent any concurrent updates to
it. This will prevent read conflict errors, but will cause version lock
errors if other clients attempt to write.

If no other clients are writing this could be a practical solution.
 
> >>3. (Because I figure it's kinda relevent here and I like asking
> >>     questions in 3's ;-)
> >>    When is MVCC due to land and will it help with this problem?
> > 
> > Yup, that's the plan. MVCC is already in ZODB 3.3. Zope 2.8 will use
> > the new ZODB.
> 
> Can ZODB 3.3 get transplanted into Zope 2.7?

No, the persistent base class is now a new-style Python class. It
requires a new-style extension class to go with it. As you can imagine
this requires some serious changes to the low-level guts of Zope.
 
> Also, what's the status of the 2.8 release plan? Suddenly I want it a 
> lot now ;-)

As Jim said, we've tossed around having an alpha release soon. Currently
I thing the only thing that is definitely broken is ZClasses, which I'm
sure won't be a problem for you ;^)

-Casey