[Zope-dev] Re: Caching ZCatalog results

[Philipp von Weitershausen]

Tres Seaver wrote:
> Tres Seaver wrote:
>> Roché Compaan wrote:
>>>> On Fri, 2007-02-23 at 06:55 -0500, Tres Seaver wrote:
>>>>> Roché Compaan wrote:
>>>>>> I'm curious, has anybody played around with the idea of caching ZCatalog
>>>>>> results and if I submitted a patch to do this would it be excepted?
>>>>>>
>>>>>> I quickly coded some basic caching of results on a volatile attribute
>>>>>> and I was really surprised with the amount of cache hits I got
>>>>>> (especially with a Plone site that is a heavy user of the catalog)
>>>>> +1.  I think using the 'ZCachable' stuff (e.g., adding a RAMCacheManager
>>>>> and associating a catalog to it) would be the sanest path here.
>>>> Cool idea. I haven't done any coding involving OFS.Cache though. Looking
>>>> at it briefly it looks like one can modify the catalog to subclass
>>>> OFS.Cacheable and then use the ZCacheable_get, ZCacheable_set and
>>>> ZCacheable_invalidate methods to interact with a cache manager. This
>>>> needs to be pretty explicit though. Are there any side effects that I
>>>> should guard against if the catalog subclasses OFS.Cache?
>> I don't think so.  Here are some random thoughts on the idea:
>>
>>  -  The 'searchResults' method must pass its keyword arguments as
>>     part of the cache key.
>>
>>  - I don't know if there is a reasonable way to do 'mtime' for
>>    the catalog:  we would like to be able to get an mtime cheaply
>>    for the BTrees (indexes, the 'data' container), but I don't know
>>    if that is possible.
>>
>>  - The "right" place to do this feels like the 'searchResults' of
>>    ZCatalog, just before it calls 'self._catalog.searchResults'.
>>
>>  - The CMF's catalog overrides 'searchResults', but calls it at
>>    the end, so everything there should work.
> 
> Hmm, on further thought:
> 
>  - It isn't safe to stash persistent objects in the RAM Cache manager,
>    because they can't be used safely from another database connection.
> 
>  - The result set you get back from a query is a "lazy", which will
>    be consumed by each client:  no two clients will see the same
>    thing.
> 
> Maybe this won't work, after all.

I have had little exposure to the actual innards of the Zope 2 catalog, 
so this might be too wild for good old Zope 2, but here it goes:

In Zope3, catalog indices actually only deal with integers (int ids) 
that only *represent* actual objects (this is for optimization reasons, 
and so they don't actually have to hang on to persistent objects 
themselves). This way a Zope 3 catalog search result is actually a list 
of integers, not a list of objects (or brains, or whatever). Of couse, 
you can use that list of integers to look up the corresponding objects 
with an integer ID utility (and there is a convenience API for that, but 
that's not important).

A wild guess is that the Zope 2 index does the same or at least 
somethign similar. Perhaps not in a nice componentized manner like Zope 
3 does (using a separate utility for the int id mapping), but I do 
recall the ZCatalog storing "uids". RAM caching those integers should be 
absolutely possible. Heck, they don't even need much space and can be 
kept efficiently in data structures...

It may require a bit of hacking the catalog, of course. Perhaps it's 
time to start thinking about componentizing the Zope 2 catalog to make 
such things easier in the future?

-- 
http://worldcookery.com -- Professional Zope documentation and training
Next Zope 3 training at Camp5: http://trizpug.org/boot-camp/camp5