[Zope-dev] Re: More resilient indexes

[Roché Compaan]

On Mon, 2007-09-17 at 19:57 -0400, Tres Seaver wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Roché Compaan wrote:
> > Hi there
> > 
> > Are the current indexing solutions that we have in Zope 2 and 3 the best
> > that we can have? Is there any way to improve the amount of concurrent
> > indexing a field or text index can handle. The Zope 3 implementation
> > doesn't look significantly different to the Zope 2 one in that it still
> > uses a BTree for forward and reverse index.
> > 
> > I have a basic stress test where 10 concurrent threads index 10 random
> > words in a FieldIndex and get too many conflict errors to even consider
> > this a usable scalable indexing solution. I don't really want to index
> > objects in another backend. I really would like to make the ZODB work
> > for me here.
> > 
> > Are there solutions here? Can one employ some of the QueueCatalog
> > conflict resolution strategies to make indexes more resilient? Or should
> > one use some locking strategy instead?
> 
> Check out QueueCatalog, which batches up indexing changes for processing
> within a single thread.
> 
>   svn://svn.zope.org/repos/main/Products.QueueCatalog

I use QueueCatalog often and I know how it works. But if an application
requires immediate indexing then QueueCatalog is not a solution. 

Sorry if I was unclear but what I'm really asking is if it is possible
to improve the conflict handling of the current indexes that we have in
Zope. I am also asking if concurrent indexing in the ZODB is a realistic
goal.

The reason I mentioned QueueCatalog is not because of its batch indexing
in one thread, but because it has a lot of conflict handling on the
queue itself. Sessions might be another good example of a product that
tries hard to handle conflicts. Do these products have strategies that
can be made to work for indexes?

-- 
Roché Compaan
Upfront Systems                   http://www.upfrontsystems.co.za