Object Relationships (was Re: [Zope3-dev] zwiki: performance of findChildren())

[Shane Hathaway]

(This is quite relevant to Zope 3, so I'm keeping this topic here, but 
I've changed the subject.)

Jeffrey P Shell wrote:
> Well, then we get into the design issue of 
> Bob.receipts.receipt1.purchaser pointing back to Bob.  Actually, I just 
> don't know how to design for this scenario in the ZODB/Zope.  I've heard 
> that (at least, up until recently) maintaining hard persistent links to 
> objects elsewhere in the tree was a bad thing to do.  Even now - for 
> some reason, I'm scared of doing it (maybe it's just the lack of an 
> event framework to catch deletes, etc).

The need to avoid "hard links" is not actually a ZODB issue.  It's a 
constraint that only arises because Zope makes an object system look 
like a filesystem.  Users rightfully expect things to work in a certain 
way when they look at a filesystem.  For instance, when you delete a 
folderish object you expect everything it contained to disappear as 
well.  You rightfully expect to reclaim the space those objects were 
taking up.  You also expect to be able to replace an object with another 
and have all references to the original object automatically see the new 
object.  In the filesystem model, it's cruel to the user to break these 
assumptions.

However, ZODB is very good at maintaining relationships other than 
containment.  You can make many strong references to a single object and 
expect each of those references to be maintained persistently.  There is 
no reason not to do this except to maintain the filesystem model.

>> Why don't collections -- persistent dict and lists and BTrees -- fulfill
>> this need?
> 
> 
> Because there's no mention about the impact of one to many references to 
> another persistent object in another object tree.  Sometimes I think 
> about making a reference and changing the acquisition, and then I get 
> yelled at about potential security risks.  I always hear why I 
> shouldn't, or why I should be careful when doing it, or how some 
> upcoming version of the software may address the issue (but no mention 
> of it ever shows up).  And it's one of the reasons we keep ending up 
> putting up with relational databases to store business objects in.
> 
> And also, because there has been no event tool up until now, so just 
> storing a path and building a connection to an object from that path 
> often feels like an exercise in hope - hope that the path will stay 
> faithful to the object one wants at all times.  We try to program around 
> this by making our applications that need to keep track of such 
> inter-object links as solid as possible, but something always leaks 
> through now and again.
> 
> I hope I'm explaining this well.  I'm just realizing that I really don't 
> know how to try to integrate an existing relational model and 
> application into Zope via Ape, and this is the main sticking point.  And 
> that's not even ZODB related - it's just the Zope development paradigm 
> I've found myself in over the years, which stems from the "yeah, you 
> don't really want to make links like that" statement that's come up time 
> and again.

I think I know what you're saying.  You have a tree of objects, and you 
want to take advantage of relational queries to find objects.  In this 
context, it feels like BTrees are a low-level alternative to relational 
queries.

> It's been almost a year and a half since I first posted a question about 
> how this will work in Zope 3.  I think that the ObjectHub quickly 
> stemmed out of that, but what I'm hearing is that there's still no way 
> for me to write the business applications that I would really like to 
> write, now that Gary's said that the ObjectHub is the basis to build 
> what I want, but it's not what I want.  And I wish I had the skills and 
> time to put in here, but these other apps that are barely paying the 
> bills sadly come first. :\
> 
> http://mail.zope.org/pipermail/zope3-dev/2002-January/000407.html
> 
> Note: in that message, I mentioned a Zope/CMF app that needed 
> relations.  I don't think mxmRelations existed at the time, so I ended 
> up writing my own relationship manager and event framework.  This ended 
> up putting me so far behind schedule (fortunately it was just an 
> internal-use system) that it was the nail in the coffin for this whole 
> project, which is still a bitter taste in my mouth.  So, I apologize if 
> I come off a bit offensive about this issue.  It's the one wall I crash 
> against more often than any other.

Most Zope applications don't hit this wall because they use a ZCatalog 
as one big relation.  Catalogs are great for text searching, but make a 
poor substitute for arbitrary relations and queries.

I'm looking for a transparent way to execute arbitrary queries in Ape. 
The difficult part is providing an interface that a "dumb" ZODB storage 
(like FileStorage) can implement.  I want very much to retain the 
ability to run an application on either FileStorage or Ape/SQL, even if 
the FileStorage version has to spelunk the entire database to answer 
queries.

Shane