[ZODB-Dev] Partitioning a container

[sean.upton@uniontrib.com]

If I'm to take a wild-guess, I think this is something that ideally should
be tackled at the application level, but may not rule out other
complementary moves using something like AdaptableStorage or any other
storage or mapping approach appropriate.  I have an application that does a
lot of "virtual" containment (but I haven't tried splitting stuff physically
across storages yet), so here's my thoughts.

For example, take a Zope/CMF site that has an custom "library tool" in it.
A library tool may contain multiple "repositories" (physical containers,
CMFBTreeFolder2 instances).  The library tool would function very similarly
to the skins tool in the CMF, providing __getattr__() hooks to get assets
inside each repository (i.e. "current" & "archive") on behalf of the
application - creating a large virtual store of partitioned (and potentially
diversely mounted?) physical stores.  You would then code "repository-aware"
folders and application objects that used __getitem__() to make it possible
to do this:

Logical URL			Physical Container
=====================================================
news/1999/blah.html 	portal_library/_archive/1999/blah.html
news/2005/foo.html	portal_library/_current/2005/foo.html

Where "1999" & "2005" are folders or placeholder objects that have the
ability to get the assets "blah.html" and "foo.html" by having a
__getitem__() that can utilize the library tool (which will get the "real"
asset from the appropriate "mounted" store on its behalf).

All you application components that could possibly be "virtual containers"
would need a hook like this:

def __getitem__(self, name):
	library = getToolByName(self,'portal_library')
	if library.hasAsset(name):
		return library.getAsset(name).__of__(self)
	raise KeyError, name

The library acts much like the CMF skins tool does, and each "repository"
would be like a skins folder (and each could be a mounted storage). In
theory, if you did your partitioning inside such a "library" tool that
contained multiple stores, you could likely mount each "repository" as a
different sort of storage if you wanted (as long as Implicit.__of__() in
this case is okay working with wrapping objects in an acquisition context of
an object on another physical storage, which I've never tried but assume
would work).

Sean

-----Original Message-----
From: Adrian van den Dries [mailto:adriand@flow.com.au]
Sent: Tuesday, March 04, 2003 2:22 PM
To: zodb-dev@zope.org
Subject: [ZODB-Dev] Partitioning a container


Here's my idea:

I've seen a couple of places with a storage pattern of "no deletes":
ie, when you've finished with an object, you "retire" it and it moves
into an archive.  This means your storage will grow indefinitely, so
it would be good to be able to partition your container into different
storages: current and archive, and tune either to the needs of the
application (temp storage for current, Oracle for archive, etc.)

Here's the catch: the object's path mustn't change: that is, the
container transparently decides the storage from which to pull the
object.  This would satisfy the content management ideal of
"persistent URLs".  For example:

  /news/1999/blah.html -> /news/_archive/1999/blah.html
  /news/2005/foo.html ->  /news/_current/2005/foo.html

Generalising further, even better would be the ability to define your
own partitioning strategy: the example that comes to mind is a large
user homepage farm, where you partition according to a member's
username (first letter, etc.), but the container still transparently
maps the object id.  There are probably many other uses.

I suspect this would potentially touch a lot of code with which I am
not entirely familiar, and I'm not sure where to start. (Mounts,
Acquisition, Traversable come to mind.) If it could be used with the
so-called standalone ZODB, that would make ZODB an *extremely*
powerful object database, but what is the status of mounting storages
in this context?  Also, an implementation at this level precludes the
possibility of using 30x redirects (which aren't the right solution
anyway).

Does this sound feasible?  Tickle anyone's itch?

I suspect AdaptableStorage is the place to experiment with this.

a.

-- 
 Adrian van den Dries                           adriand@flow.com.au
 Development team                               www.dev.flow.com.au
 FLOW Communications Pty. Ltd.                  www.flow.com.au

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev