[Zope3-dev] catalog 'all documents' abstraction

[Martijn Faassen]

Gary Poster wrote:
> On Aug 30, 2005, at 1:57 PM, Martijn Faassen wrote:
[snip]

>> It would be helpful if someone could explain the motivations behind
>>  the extent catalog, by the way -- this information seems to be 
>> missing in zc.catalog. Am I at all on the right track with my 
>> thinking on it?

> It should be pointed out initially that the son-of-queued-catalog
> code doesn't have anything to do with extents.  I think Jim wants
> that factored out when we have time so that can be a mix-and-match 
> capability.  I think you are asking about extents themselves, though.
> 
Okay, I didn't realize yet glancing at this that this is *also*
son-of-queued catalog. Interesting. I'll glance at it some more. :)

> We had three use cases that led us to extents.
> 
> First, we wanted several catalogs that only indexed certain different
>  things.  This could have been done by subscribers, so this wasn't 
> terribly compelling by itself.

Okay, this is clear. It's not that clear to me how to efficiently make a
subscriber only handle one object type (I've been using the "is this a
IFoo? If not, return" pattern at the start of subscribers), but that's
another discussion.

> Second, we wanted to transparently support queries that merged
> results across catalog-like data structures.  The catalog defined the
> items we wanted to search through, while some of the other data
> structures kept track of a larger set of objects (subsuming the set
> that the catalog cared about).  Sometimes, users could perform a
> query that didn't actually use any of the catalog's data structures,
> but that should be filtered by the set of the catalog's objects--its
> extent.

I'm not sure I comprehend the motivations behind this one. Could you
elaborate?

> Third, we wanted to let our indexes data be usable for NOT queries.
> In order to do that, we needed an IFBTree structure that describes
> the complete set for a given catalog, so that a contained index can
> simply (and reasonably efficiently) subtract the query result from
> the full set.  The indexes in zc.catalog also use extents for some
> other similar tricks.

This one's also clear.

> An extent that accepts all objects would effectively be the data 
> structure you want, as I understand it.

I'm not sure -- 'not' is indeed context dependent, so which extent is in
use to determine the results of a 'not' operation depends on the query.
I think it's okay to ask the users to explicitly specify the extent when
they're doing the query, as long as there's an easy way to construct it
for the simple cases.

> It is actually (at least typically for us) different than the intid
> mapping because there are several classes of things that have intids
> that are not cataloged.   

> If more than one catalog all index the same
> objects, I'd first wonder  why the indexes were not all in the same
> catalog; 

Good question. I think one example of such a scenario is if you wrote a 
codebase, and I extended this codebase with some adapters which carry 
around information I also want indexed. I may decide not to introduce 
new indexes into your catalog but instead produce my own catalog to have 
the concerns separate from each other. In this case I'd want to do 
queries over multiple catalogs which index the same objects.

> I'd second say that  yes, they probably could share a
> filter-less extent.

Why filter-less? I mean, wouldn't you want to filter on object type?

> If we want any of zc.catalog in the Zope core, each component will 
> certainly need a proposal, by the way: we're offering this as code
> that has helped us out and that we think might help others, either
> directly or as ideas, so we are not duplicating effort.  We're not
> proclaiming it to necessarily be our next core step.

Understood. I'm giving you feedback. :) We (Infrae) are going to put 
some code online eventually that we produced in the project we're 
working on, for the same reasons.

Regards,

Martijn