[ZODB-Dev] ZODB-level indexing

[Jeremy Hylton]

These sound like the sorts of concerns that lead to the QueuedCatalog
product.  http://cvs.zope.org/Products/QueueCatalog/
I can't find a description of the product anywhere or a release or
anything like that.  Perhaps it isn't widely known.

The basic idea is to batch updates to the catalog, but putting change
events in a queue instead of sending them directly to the catalog.
Coincidentally, Tim, Jim, and I were doing so load testing of ZEO and
QueueCatalog last week; I wrote a few notes that might be helpful:
http://www.python.org/~jeremy/weblog/031031c.html

The drawback is that the catalog is a little bit out of sync with the
content.  You get improved performance and the cost of reduced
consistency.

It sounds like you are sketching another alternative that is also
interesting.  You're trying to get the indexing work out of your
application and push it back to the server.  Instead of integrating it
at the ZODB level, you might create a separate index application; it
would be just another application as far as ZODB is concerned.  You
could run it on the same machine as the ZEO server and communicate with
it via some kind of RPC (XML_RPC? CORBA?).

One question is: Should the client send queries and get results or just
load the index objects directly?  It's not obvious to me which solution
is better.

Jeremy