[ZODB-Dev] Re: ReadConflictError while re-indexing big fat index

[Casey Duncan]

On Mon, 02 Aug 2004 11:41:49 +0100
Chris Withers <chris at simplistix.co.uk> wrote:

> Hi All,
> 
> I'm pretty sure this has been gone over before, but I thought I'd
> spark off another discussion 'cos this is a little bit disturbing.
> 
> I'm currently getting this:
> 
>      * Module Products.ZCatalog.ZCatalog, line 649, in searchResults
>      * Module Products.ZCatalog.Catalog, line 753, in searchResults
>      * Module Products.ZCatalog.Catalog, line 496, in search
>      * Module Products.PluginIndexes.PathIndex.PathIndex, line 234, in
>      
> _apply_index
>      * Module Products.PluginIndexes.PathIndex.PathIndex, line 185, in
>      
> search
>      * Module ZODB.Connection, line 561, in setstate
>      * Module ZODB.Connection, line 590, in _is_invalidated
> 
> ReadConflictError: database read conflict error (oid 000000000001ff36,
> 
> class BTrees.IIBTree.IISet)
> 
> ...while re-indexing a big text index (nb: not a PathIndex)
> 
> The indexing is happening on a dedicated ZEO client so as not to
> impact on the site's performance, however, getting the above error on
> every page impacts somewhat more on the Site's performance ;-)
> 
> This raises some questions for me:
> 
> 1. What is the above error message really trying to tell me?
>     (it's a pretty obscure message ;-)

It is telling you that an ISet object in the PathIndex was changed and
commited by another client since the transaction began. In order to
protect data integrity, ZODB does not allow such "dirty reads", instead
it raises a ReadConflict error.

> 2. How SHOULD I try and re-index a large text index without
> effectively
>     taking down the whole site for an hour or two.

You probably will need to reindex incrementally instead of in one big
transaction. You might also need to "throttle back" the reindex speed so
that it does not cause excessive read conflicts in the other clients.
You actually won't see errors like the above in Zope until after it has
retried 3 time unsuccessfully.

> 3. (Because I figure it's kinda relevent here and I like asking
>      questions in 3's ;-)
>     When is MVCC due to land and will it help with this problem?

Yup, that's the plan. MVCC is already in ZODB 3.3. Zope 2.8 will use the
new ZODB.

-Casey