[ZODB-Dev] ReadConflictError while re-indexing big fat index

[Jim Fulton]

Chris Withers wrote:
> Hi All,
> 
> I'm pretty sure this has been gone over before, but I thought I'd spark 
> off another discussion 'cos this is a little bit disturbing.
> 
> I'm currently getting this:
> 
>     * Module Products.ZCatalog.ZCatalog, line 649, in searchResults
>     * Module Products.ZCatalog.Catalog, line 753, in searchResults
>     * Module Products.ZCatalog.Catalog, line 496, in search
>     * Module Products.PluginIndexes.PathIndex.PathIndex, line 234, in 
> _apply_index
>     * Module Products.PluginIndexes.PathIndex.PathIndex, line 185, in 
> search
>     * Module ZODB.Connection, line 561, in setstate
>     * Module ZODB.Connection, line 590, in _is_invalidated
> 
> ReadConflictError: database read conflict error (oid 000000000001ff36, 
> class BTrees.IIBTree.IISet)
> 
> ...while re-indexing a big text index (nb: not a PathIndex)
> 
> The indexing is happening on a dedicated ZEO client so as not to impact 
> on the site's performance, however, getting the above error on every 
> page impacts somewhat more on the Site's performance ;-)
> 
> This raises some questions for me:
> 
> 1. What is the above error message really trying to tell me?
>    (it's a pretty obscure message ;-)

The search operation tried to read a data value that was
written since the start of the search transaction. This data
may be inconsistent with data already read and so cannot be used.

> 2. How SHOULD I try and re-index a large text index without effectively
>    taking down the whole site for an hour or two.

You could try increasing the number of retries.

Obviously, MVCC is the long-term answer.

> 3. (Because I figure it's kinda relevent here and I like asking
>     questions in 3's ;-)
>    When is MVCC due to land and will it help with this problem?

It has landed in ZODB 3.3, which has had a beta release. But you
*really* meant to ask when it will land in Zope.

It will land in Zope in Zope 2.8.  I think we should make a Zope 2.8
alpha release as soon as possible.  We need to find someone with time
to shepard the Zpe 2.8 release along.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org