[ZODB-Dev] Re: Incremental Search Engine (was: RE: [BTrees] considerable optimization potential in"intersection")

[Dieter Maurer]

Casey Duncan wrote at 2004-8-9 15:57 -0400:
> ...
>So I assume a higher candidate will always be one that has not
>been seen, and the comparison is a cheap way to test that rather than
>build a set of seen docids?

Indeed. As hits are generated in an increasing order,
there is a number "current". All hits "<= current" have been
examined and hits "> current" are not yet known.

All indexes collaborate to push "current" towards higher values
finding hits in the process and skipping unsuccessfull candidates.

>Why does that require that docids are handed
>out in increasing order?

It does not. Any complete order is sufficient.

However, when docids were handed out in an increasing order, then
the docid order would have a semantic meaning: a higher value would imply
a newer document.

Now almost all our documents are sorted by "recency".
Would the docid order mean "recency", there were no longer any
need to build the complete result set and sort afterwards.
We could directly use the incrementally generated hits --
and simply generate more hits when the user expresses a wish for
them.

> ...
>Ok, it would be useful to see an example of the outer and inner query
>mechanism together for me to truely understand it. I suspect it would
>answer my questions above.

I expect to have the complete code for "IAnd" (Incremental And) searches
within 2 weeks.

-- 
Dieter