[ZODB-Dev] ZODB-level indexing

[Roché Compaan]

* Jeremy Hylton <jeremy at zope.com> [2003-11-05 16:35]:
> These sound like the sorts of concerns that lead to the QueuedCatalog
> product.  http://cvs.zope.org/Products/QueueCatalog/
> I can't find a description of the product anywhere or a release or
> anything like that.  Perhaps it isn't widely known.

QueuedCatalog only addresses performance hits suffered while objects are
indexed. You still end up with a lot of unnecessary invalidations being
pushed to clients.

> The drawback is that the catalog is a little bit out of sync with the
> content.  You get improved performance and the cost of reduced
> consistency.

We need up-to-date indexes most of the time.

> It sounds like you are sketching another alternative that is also
> interesting.  You're trying to get the indexing work out of your
> application and push it back to the server.

Exactly.

> Instead of integrating it at the ZODB level, you might create a
> separate index application; it would be just another application as
> far as ZODB is concerned.  You could run it on the same machine as the
> ZEO server and communicate with it via some kind of RPC (XML_RPC?
> CORBA?).

This sounds like a very workable idea and without the complication of
trying to integrate this with a ZODB storage. I doubt if a typical ZEO
setup will benefit from this because there is still a round trip from
client to server.

But we don't have a typical ZEO setup so this might work for us. We have
something that is closer to a poor man's multi master setup. In the
ideal multi master setup I want to replicate *only* changes to objects
between hosts and each master should index an object when changed
objects arrive.

At the moment we are going for application level replication - we
replicate specific calls to a remote host that we know will lead to
objects being indexed on the remote host. This requires a lot more
discipline throughout the app and increases the risk of having two sites
that become unsynchronized but its seems easier to implement.

-- 
Roché Compaan
Upfront Systems                 http://www.upfrontsystems.co.za