[ZODB-Dev] Re: Community opinion about search+filter
faassen at startifact.com
Thu Mar 15 13:04:03 EDT 2007
Adam Groszer wrote:
> I'd like to ask your opinion, your experiences about searching and
> filtering in quite large object DBs.
> We need to add search and filter functions to our current app, where
> the user might be able to create quite _sophisticated_ filter criterias.
> (The app is a pure Z3 app, subject is document management)
> Currently we're looking at something based on catalog/indexes.
> As I checked the most comfortable solution would be based on
> Some questions arose:
> - Is it necessary/worth adding indexes on all attributes?
> - How does the index perform on modification and retrieval?
> The biggest problem is that this will be our first try, so we're
> missing experiences and are a bit puzzled about the right solution.
> Certain is that moving to RDB is not an option.
I think one of the main limitations of the current catalog (and
hurry.query) is efficient support for sorting and batching the query
results. The Zope 3 catalog returns all matching results, which can then
be sorted and batched. This will stop being scalable for large
collections. A relational database is able to do this internally, and is
potentially able to use optimizations there.
It would be very nice if someone could look into expanding hurry.query
and/or the catalog to support these cases. It would be interesting to
look at what Dieter Maurer has done with AdvancedQuery in Zope 2 in this
regard as well.
More information about the ZODB-Dev