[ZODB-Dev] Unofficial AdaptableStorage Wiki

[Shane Hathaway]

Phillip J. Eby wrote:
> At 12:37 PM 2/20/03 -0500, Shane Hathaway wrote:
> 
>> Thanks for doing this, Chris.  Writing the initial documentation is 
>> terribly difficult on a project like this: I became so familiar with 
>> the project that I could hardly see how other people could be 
>> *unfamiliar* with it... if you know what I mean.  The solution to that 
>> dilemma is to give myself time, taking advantage of memory decay to 
>> make myself less familiar with the project, or let someone else try to 
>> document it.
> 
> 
> Some kind of conceptual introduction to the concept of keychains and the 
> use of names would be helpful, too.  Looking at the interfaces, it 
> appears that there is some kind of concept of traversal paths involved, 
> but I'm not clear on why that's useful.

Here's my first attempt at explaining keychains.  Keychains are not 
related to traversal paths.

The idea is that different branches of your database can be stored in 
different ways or have different classification policies.  Let's say 
your whole database is in a relational database, except for the objects 
in ZCatalogs, which you want to be stored in ZODB.  You could create a 
mount point for every ZCatalog, but that could be cumbersome.  Instead, 
you make the object mapper for ZCatalogs into what I've been calling (in 
my head) a "domain mapper".  The main difference between an object 
mapper and a domain mapper is that a domain mapper has sub-mappers (some 
of which may be domain mappers themselves).  To access the sub-mappers, 
you use keychains with multiple keys.

Let's make this more concrete.  The current Zope 2 / Postgres mapper 
uses tuples with a single integer for nearly all keychains, except the 
root, which uses an empty tuple.  (Oops, I wrote in the wiki that keys 
are strings.  I think that's not necessarily true.)  The single integer 
in the tuple is the primary key used in the database tables.

Now let's say the keychain (45,) refers to a ZCatalog object.  Normally, 
objects inside the catalog would get a similar keychain: (46,), (47,), 
etc.  But we want objects contained by the catalog to be stored in ZODB, 
not the relational database.  To make this happen, the ZCatalog domain 
mapper generates two-item tuples for subobject keychains.  The second 
item would be an 8-byte OID string.  Subobjects of the catalog at (45,) 
use keychains that start with 45:

(45, '\0\0\0\0\0\0.b')
(45, '\0\0\0\0\0\0.c')
(45, '\0\0\0\0\0\0.d')

and so on.  The meaning of each item in the keychain is encapsulated in 
a domain mapper.

This means that a single database can cross domains and "mount" data 
sources transparently.  Unfortunately, nothing in AdaptableStorage takes 
advantage of this yet, except for the domain mapper at the root, which 
doesn't do much to clarify it.

Another reason you might use a domain mapper is to change classification 
policies for branches of the database.  You might decide that new .txt 
files on the filesystem usually map to OFS.Files, but when inside CMF 
sites, new .txt files map to CMF Documents.  You can do this without 
changing CMF (since it's probably not part of CMF's mission), and 
without creating a mount point for every CMF site.

The Zope 2 / filesystem mapper uses paths as the first item in the 
keychain.  It was tempting to use Zope's physical paths as keychains, 
but I decided that would dilute the concept of domain mappers.  It would 
be like calling every container a mount point.

> By way of contrast, EntityDM's in PEAK simply use an opaque value for a 
> _p_oid, and if a particular serialization implementation needs a path, 
> it's certainly welcome to use, say, a tuple of keys of some kind for 
> that purpose, but the thing that manages the objects' state and 
> transactions, doesn't really *care* about the key format.  So I'm 
> curious whether I should view the keychain idea in AdaptableStorage as a 
> cool enhancement or as meaningless extra overhead.  ;)  And I won't know 
> that until I understand what it is and what it's for.  :)

I think keychains are quite valuable, but since I haven't actually made 
much use of them yet, I don't have evidence to back it up.  I'm very 
open to better names for "keychain" and "domain mapper".  I'm also open 
to the idea that keychains and domain mappers should be scrapped. :-)

Shane