[ZODB-Dev] ReadConflictError while re-indexing big fat index

[Roché Compaan]

Chris Withers wrote:
> Hi All,
> 
> I'm pretty sure this has been gone over before, but I thought I'd spark 
> off another discussion 'cos this is a little bit disturbing.
> 
> I'm currently getting this:
> 
>     * Module Products.ZCatalog.ZCatalog, line 649, in searchResults
>     * Module Products.ZCatalog.Catalog, line 753, in searchResults
>     * Module Products.ZCatalog.Catalog, line 496, in search
>     * Module Products.PluginIndexes.PathIndex.PathIndex, line 234, in 
> _apply_index
>     * Module Products.PluginIndexes.PathIndex.PathIndex, line 185, in 
> search
>     * Module ZODB.Connection, line 561, in setstate
>     * Module ZODB.Connection, line 590, in _is_invalidated
> 
> ReadConflictError: database read conflict error (oid 000000000001ff36, 
> class BTrees.IIBTree.IISet)
> 
> ...while re-indexing a big text index (nb: not a PathIndex)
> 
> The indexing is happening on a dedicated ZEO client so as not to impact 
> on the site's performance, however, getting the above error on every 
> page impacts somewhat more on the Site's performance ;-)
> 
> This raises some questions for me:
> 
> 1. What is the above error message really trying to tell me?
>    (it's a pretty obscure message ;-)

An object (most probably the index) was modified and thus invalidated in 
another transaction. At the time that your indexing operation wants to 
commit, the ZEO server tells you that your are trying to modify an 
object that has been invalidated.

  > 2. How SHOULD I try and re-index a large text index without effectively
>    taking down the whole site for an hour or two.

Sigh. I've come to love the ZODB, but this issue is a dagger in my side 
and always makes me regret not choosing an RDBMS. I haven't seen any 
solution to do this - as far as I know this is not currently possible.

> 3. (Because I figure it's kinda relevent here and I like asking
>     questions in 3's ;-)
>    When is MVCC due to land and will it help with this problem?

I'd also like to know.

-- 
Roché Compaan
Upfront Systems                 http://www.upfrontsystems.co.za