[ZODB-Dev] ZODB queries (was ZODB and ORBit-Python problems)

[Jim Fulton]

Christian Robottom Reis wrote:
> 
> On Tue, 22 May 2001, Greg Ward wrote:
> 
> > It sounds like you don't want a query language per se, but rather an
> > efficient query mechanism.  So would I!  I don't much mind writing
> > queries in Python -- simple queries are probably a bit wordier than they
> > would be with (say) OQL, but at least I don't have to learn another
> > language.  More importantly, I don't have to embed that new language in
> 
> Greg, 100% correct. I had ignored this point, but it's important -- the QL
> isn't the important part - a mechanism for querying, however, _is_.

Well, there is one other factor to consider. An advantage of SQL/OQL is that
there is an implied query optimizer behind the language that will automatically
use indexes if they exist. This is something that you don't get by using
Python as a query language. The Python programmer must either be aware
of and use indexes or it needs to use some searching/indexing object that
encapsulates the indexes, like the Catalog. 

> SQL/OQL is nice if you want to provide for user-defined queries, but who
> wants users, anyway? :-)

IMO SQL/OQL is not really user friendly except for toy problems. ;)
  
> > However, I *would* like to see some (documented and comprehensible)
> > infrastructure for indexing a ZODB.  (I've attempted to read the code
> > for Catalog and related classes, but came away understanding less than I
> > went in with.)  As it stands, if I want to query every Thing object in
> > our database, I have to loop over our collection of all Things, load
> > each into memory, and query it.  Our database isn't very big, so this
> 
> I believe both SearchIndex and Catalog (higher-level? who knows?) exist to
> solve this. As far as them being undocumented, I've offered to write-up as
> soon as somebody shows me some simple, short, working code that uses them.

I think that documentation is only part of the problem. I don't think that
Catalog/SearchIndex have been maintained in a way that facilitates reuse outside
of Zope. This is not intentional. In fact, Michel Pelletier tried hard to make
Catalog usable without ZCatalog.  I'd be happy to see these usable outside of Zope, 
but that takes time, and time is short. I'd be willing to work with someone on this.

> I don't know, however, if a full solution is orthogonal to pickling, and
> nobody has answered if this point is moot. AFAICS if you use unstructured
> storage, your indexes will replicate attribute data that's in the DB, and
> you'll have the problems of syncing that during updates and deletes. Apart
> from that, you need a whole lot of indexes -- one for each query method.
> Am I off-track here?

I'm not sure what you mean by a "structured storage". Are you thinking of a 
storage that includes notions like "tables" and "indexes"?  

Indexing "can" be orthogonal to pickling. I chose to use a "micro kernel"
approach for ZODB, keeping as little as possible in the database layer
as possible.. I think that this approach has been pretty successful.
It provides a lot of flexibility. I certainly think it's a valid approach.

I wouldn't worry about duplicating attribute values in indexes. This is really
just the price of admission for indexing. "Structured" storages pay this price too.

I wouldn't worry about syncing, as that's what transactions are for.
 
A number of people have argued for providing indexes at the database
level.  I'm open to this, but every time I think hard about it, I find
some aspect of it that makes it unattractive to me.

> > isn't a big deal (yet).  I can see that it would be a showstopper for
> > someone with (say) hundreds of thousands of Thing objects in their
> > database.
> 
> Which is where I fit in, unfortunately. :-)

Right, so you need some indexing infrastructure to help you with it.

Jim

--
Jim Fulton           mailto:jim@digicool.com   Python Powered!        
Technical Director   (888) 344-4332            http://www.python.org  
Digital Creations    http://www.digicool.com   http://www.zope.org