[Zope3-dev] catalog 'all documents' abstraction

[Jim Fulton]

Martijn Faassen wrote:
> Hi there,
> 
> Now that there's a plain catalog and an extent catalog, and while I was 
> implementing a 'not' operator for a query language, I ran into some 
> missing abstraction that would be convienient; a way to get all the 
> object ids that are indexed,

For some definition of indexed. :)

> preferably in the form of a IFBTree so I 
> can do fast intersections and the like (which are needed for "not", 
> which is an intersection of all object ids with those object ids I don't 
> want).
> 
> Right now this information is not really available in a standard or a 
> very efficient manner. For the normal catalog you can ask the IIntId 
> utility for all int ids, which is reasonable and should be fairly 
> efficient, though it'd be nice to have something come out in IFBTree 
> form (or perhaps the intid's IOBTree can be intersected with an IFBTree 
> directly? that'd be nice..

Yup.  This is a wart in the current BTree implementation. You can't do
merging operations across BTree types. We've been able to get around
in the past, because we didn't support operations accross catalogs.
Now, however, this is getting to be more of a problem.

 > ). I also think however that it's the wrong
> place the ask for this information, as this doesn't work with the 
> extentcatalog.

Well, it depends on what you meant by "indexed" above.  Different indexes
index different objects.  The extent catalog tried to define an extent for
which it makes sense to apply different (not) operations.

> The catalog itself seems like the wrong place to ask as well however, as 
> things would get hairy in the case of a query over multiple catalogs -- 
> which catalog would be asked for all ids that are indexed?

In general, I'd like the catalog to remain fairly small and free of logic.
I wanted to say this in the other thread you started on cataloging, but
didn't get to it.  Ideally, a catalog wouldn't have any query logic
at all.  People should be able to invovate on query algorithms without
affecting catalogs.

> Hm, perhaps this isn't ideal either, as this would get hairy in case of 
> a query that spans multiple catalogs -- which catalog will be asked in 
> that case for a list of all documents?

I think in the particular case of "not", you have to have an implicit or
explicit set that you are subtracting something from.  The "right" set is
application specific.  In any case, I think the query logic should be
in separate query components.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org