[ZODB-Dev] Re: Incremental Search Engine (was: RE: [BTrees] considerable optimization potential in"intersection")

[Casey Duncan]

On Tue, 10 Aug 2004 19:49:23 +0200
Dieter Maurer <dieter at handshake.de> wrote:

[..]
> Now almost all our documents are sorted by "recency".
> Would the docid order mean "recency", there were no longer any
> need to build the complete result set and sort afterwards.
> We could directly use the incrementally generated hits --
> and simply generate more hits when the user expresses a wish for
> them.

FWIW, I recently developed a RecentItemsIndex which keeps one or more
lists of the most recent cataloged objects (measured from a configurable
attribute) up to a specified maximum number.

This allows you to find the "top n" items that match a particular
criteria with no intersection. It also allows filtering by roles and
permissions so that (rudimentary) security filtering can still occur
(e.g., objects viewable by anonymous).

I plan to release this fairly soon, I am in the process of putting it
into production now. For now, you can get it from CVS here:

http://cvs.zope.org/Products/RecentItemsIndex

Because the index does both result matching and sorting simultaneously,
it does not play in normal catalog queries (the mechanism does not
contemplate this). Instead you query the index directly using its
"query()" method which returns a lazy result set just like ZCatalog.

-Casey