[Zope3-dev] catalog 'all documents' abstraction

[Martijn Faassen]

Jim Fulton wrote:
> Martijn Faassen wrote:
[snip]

>  > ). I also think however that it's the wrong
> 
>> place the ask for this information, as this doesn't work with the 
>> extentcatalog.
>  
> Well, it depends on what you meant by "indexed" above.  Different indexes
> index different objects.  The extent catalog tried to define an extent for
> which it makes sense to apply different (not) operations.

And is the idea that multiple extent catalogs can share an extent?

(By the way you say 'the extent catalog tried', does this mean something 
else is being considered now?)

>> The catalog itself seems like the wrong place to ask as well however, 
>> as things would get hairy in the case of a query over multiple 
>> catalogs -- which catalog would be asked for all ids that are indexed?
> 
> In general, I'd like the catalog to remain fairly small and free of logic.
> I wanted to say this in the other thread you started on cataloging, but
> didn't get to it.  Ideally, a catalog wouldn't have any query logic
> at all.  People should be able to invovate on query algorithms without
> affecting catalogs.

This is already clear; I've been trying to do so in the project I'm 
working on, though I'm more focusing on a sensible query language (well, 
of Python objects) than performance algorithms.

At the same time I believe Zope 3 *does* need query systems built in 
eventually. While it's fine to allow people to design their own query 
languages and algorithms, not everybody is able to do this, and those 
who are able to don't always want to. Even if they did, I don't want us 
to end up with 5 different query systems either.

So, while I agree that a query language in the core should not exclude 
someone from building something better, I do believe a catalog query 
language package is needed in the core.

(To be absolutely clear: I also think the RDF avenues being explored are 
very interesting, and I don't want to imply that this is not an 
interesting direction, but I do think we need something for the plain 
catalog too)

>> Hm, perhaps this isn't ideal either, as this would get hairy in case 
>> of a query that spans multiple catalogs -- which catalog will be asked 
>> in that case for a list of all documents? 
> 
> I think in the particular case of "not", you have to have an implicit or
> explicit set that you are subtracting something from.  The "right" set is
> application specific.  In any case, I think the query logic should be
> in separate query components.

I agree that the catalog should remain nice and simple.

That said, catalogs right now already have an implicit concept of 
'everything indexed', which for instance is already used for 
re-indexing, it's just not made available to someone who wants to build 
something on it.

The extent catalog makes this more explicit by defining an extent, so 
perhaps this is the way to go. The extent could be a query parameter to 
help the query engine figure out what to do in case of 'not'. For simple 
use cases, the extent can be constructed from the intid utility, perhaps.

It would be helpful if someone could explain the motivations behind the 
extent catalog, by the way -- this information seems to be missing in 
zc.catalog. Am I at all on the right track with my thinking on it?

Regards,

Martijn