[Zope3-dev] KeywordIndex

[Jeff Shell]

I'm working on a simple application which is the first time I get to
use the catalog in Zope 3. I'm writing against Zope 3.1b1. I was
dismayed not to see KeywordIndex in the main catalog set, but then I
found it in zope.index.keyword. But it seems to be a bit behind. I
have it somewhat working through subclassing, etc, but it's been
purely guess work on my part to get things this far. In my product
package, I have the following:

--
from zope.app.catalog.interfaces import IAttributeIndex, ICatalogIndex
from zope.app.catalog.attribute import AttributeIndex
from zope.app.container.contained import Contained
from zope.index.keyword.index import KeywordIndex as KeywordIndexBase
from zope.proxy import removeAllProxies
from BTrees.IFBTree import IFTreeSet, IFSet, multiunion

class IKeywordIndex(IAttributeIndex, ICatalogIndex):
    """ Interface-based catalog keyword index. """

class KeywordIndex(AttributeIndex, KeywordIndexBase, Contained):
    implements(IKeywordIndex)
    
    def index_doc(self, docid, value):
        # All security proxies need to be removed from the value.
        value = removeAllProxies(value)
        return super(KeywordIndex, self).index_doc(docid, value)
    
    def apply(self, query):
        # Keyword index doesn't implement apply(query) either.
        return self.search(removeAllProxies(query))

    def _insert_forward(self, docid, words):
        """insert a sequence of words into the forward index """
        # Replaces parent _insert_forward because apply() claims to want IFSets
        idx = self._fwd_index
        has_key = idx.has_key
        for word in words:
            if not has_key(word):
                idx[word] = IFSet()
            idx[word].insert(docid)
--
I first overrode index_doc because the base KeywordIndex does an
isinstance(value, (ListType, TupleType)), which failed on a security
proxy guarded value. Then I added 'apply()' when I noticed that the
base KeywordIndex didn't implement apply. Looking at the other
supported indexes and at the index interfaces in zope.index, I noticed
that IFSets were what was desired as the output of apply(), and that's
when I replaced _insert_forward with a near identical copy that uses
IFSet.

This works... so long as I only search for one keyword. If I search
for more than one through the catalog interface (and I imagine I would
get the same result manually), I get the following traceback:

--  
File "/Users/jshell/Documents/Programming/kbase/lib/python/br/kbase/browser/search.py",
line 22, in search
    results = catalog.searchResults(tags=query)
  File "/Library/ZopeX3/3.1/lib/python/zope/app/catalog/catalog.py",
line 105, in searchResults
    results = self.apply(searchterms)
  File "/Library/ZopeX3/3.1/lib/python/zope/app/catalog/catalog.py",
line 84, in apply
    r = index.apply(index_query)
  File "/Users/jshell/Documents/Programming/kbase/lib/python/br/kbase/catalog.py",
line 36, in apply
    return self.search(removeAllProxies(query))
  File "/Library/ZopeX3/3.1/lib/python/zope/index/keyword/index.py",
line 139, in search
    rs = f(rs, docids)
TypeError: invalid argument
--
'f' is IISet.intersection()

The implementation of search() in the base KeywordIndex uses IISets
for default values. I don't know if this is conflicting with the
IFSets I set up in my subclass. I tried quickly editing
zope.index.keyword.index to use IFSets instead, but I got the same
traceback and then quickly reverted back to leaving the code
untouched.

It's been *years* since I've even touched simple indexing code, so I
don't really know what's going on here or what's required.

I would really like to have Keyword Index. In fact, such an index is
the core of my application. I can throw together my own, I'm sure,
that's a bit more brute force for my own purposes if necessary. I
don't claim to have a solid understanding of how indexes and the
catalog work (although it's been much easier to figure out in Zope 3,
thanks!)

Is there any reason why KeywordIndex seems half-abandoned? I guess
it's not exposed to zope.app because of this. What would it take to
make it catch up to FieldIndex?

Thanks
--
Jeff Shell