[ZODB-Dev] ReadConflictError while re-indexing big fat index

[Jim Fulton]

Roché Compaan wrote:
> Chris Withers wrote:
> 
>> Hi All,
>>
>> I'm pretty sure this has been gone over before, but I thought I'd 
>> spark off another discussion 'cos this is a little bit disturbing.
>>
>> I'm currently getting this:
>>
>>     * Module Products.ZCatalog.ZCatalog, line 649, in searchResults
>>     * Module Products.ZCatalog.Catalog, line 753, in searchResults
>>     * Module Products.ZCatalog.Catalog, line 496, in search
>>     * Module Products.PluginIndexes.PathIndex.PathIndex, line 234, in 
>> _apply_index
>>     * Module Products.PluginIndexes.PathIndex.PathIndex, line 185, in 
>> search
>>     * Module ZODB.Connection, line 561, in setstate
>>     * Module ZODB.Connection, line 590, in _is_invalidated
>>
>> ReadConflictError: database read conflict error (oid 000000000001ff36, 
>> class BTrees.IIBTree.IISet)
>>
>> ...while re-indexing a big text index (nb: not a PathIndex)
>>
>> The indexing is happening on a dedicated ZEO client so as not to 
>> impact on the site's performance, however, getting the above error on 
>> every page impacts somewhat more on the Site's performance ;-)
>>
>> This raises some questions for me:
>>
>> 1. What is the above error message really trying to tell me?
>>    (it's a pretty obscure message ;-)
> 
> 
> An object (most probably the index) was modified and thus invalidated in 
> another transaction. At the time that your indexing operation wants to 
> commit, the ZEO server tells you that your are trying to modify an 
> object that has been invalidated.

As Chris pointed out, he is getting read conflicts, not write conflicts.


>  > 2. How SHOULD I try and re-index a large text index without effectively
> 
>>    taking down the whole site for an hour or two.
> 
> 
> Sigh. I've come to love the ZODB, but this issue is a dagger in my side 
> and always makes me regret not choosing an RDBMS. I haven't seen any 
> solution to do this - as far as I know this is not currently possible.

MVCC should solve this problem.

Jim


-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org