[Zope3-dev] Florent's O-R blog entry

[Michel Pelletier]

This is a general reply, Martijn just summed up many of the points so
nicely (as usual) that I'm using his email as a starting point...

> From: Martijn Faassen <faassen at infrae.com>
> Subject: Re: [Zope3-dev] Florent's O-R blog entry

> I have had some opportunity to work with the Zope 3 catalog recently, 
> and I have a few comments. First of all, I agree with the main idea that 
> the Zope 3 catalog is not a hack, and is clean and flexible. I believe 
> the catalog should be invested in, as I think it's cool.

agreed.

> Now as to where I see areas where features are lacking in the Zope 3 
> catalog:
> 
> Underfeatured query API
> -----------------------
> 
> I do think that currently the API to query it is woefully underfeatured.
> 
> I've tried to work on this problem and am sitting on some code that just 
> needs a bit of time to polish and release that allows a simple query 
> language on top of the catalog. It's just building up a tree of python 
> objects for queries, nothing special, but it is a lot higher level than 
> what's already there.

This is an important feature that the catalog needs, query logic.  The
argument that Python (without at least a query API) is its query
language no longer holds water for me.  Good query languages should
allow you to define what you are looking for, now how you look for it,
and your code, inspired by something like Dieter's AdvancedQuery, is
absolutely necessary.

But... I think we are missing another layer here.  I pointed this out in
my first reply to Gary's post about Zemantic.  I think a three-tiered
approach needs to be taken to searching, just like it has been taken
with many other aspects of Zope 3.  I think this removes the whole
argument of catalog vs. rdbms for "enterprise" systems and make the
answer even better.  

1) The model tier contains the searchable sources.  These sources
provide a simple search source interface.  I do not think it's possible
and I do not propose to have a consistent query interface across various
source implementation, but it's certainly possible to have a consistent
source management interface and it's possible for sources to describe
themselves and their searchable content using a common schema language.
Sources can be local components (catalogs, rdbms) or remote (google,
wikipedia, etc)

2) The controller tier provides query logic (agents is a common term
here).  Agents are components that implement a particular query
interface, and know how to query registered searchable sources or other
agents based on that source or agent's description.  Agents take care of
the dirty work (like result merging and joins) and provide a clean
interface to submit a query and retrieve results.  What the "query"
looks like and which search source is used depends on the agent and its
configuration.  If they know how to, agents can query agents to delegate
their work.

3) The view tier is a simple, high-level interface that the "user" (in
this case the typical third-party developer) primarily interacts with.
It is used to discover, query and manage sources and agents.  It doesn't
hide so much the underlying complexity as it provides a way to manage
the complexity.

Consider your typical CMF/Plone based application today.  They have a
portal_catalog that is the sole searchable source for the application
framework.  To add another source, one must, in general, hack the
application or the framework (note Archetypes gets around this, to a
degree, but only for catalogs, and only on values that map to an AT
field).  

If a third-party product wanted to plug into the search framework, they
need to either hack the existing portal_catalog, or if that is not
possible (which it very often is not due to name and usage conflicts at
the index level) they must create their own catalog.  But now the two
searchable sources are completely disconnected and unrelated and various
interfaces and things have to be hacked to include both sources in the
application's logic.

-Michel