[ZODB-Dev] Notes on using wrapper storages for record transformation

[Jim Fulton]

The design of ZODB views pickles as opaque to storages. Storages
primarily store strings.  I have always assumed that writing wrapper
storages for compression or encryption would be easy. I was wrong. :)

Storages sometimes need to know about pickles:

A. When doing garbage collection, a storage (or garbage collector)
   needs to extract object references.

B. When doing conflict resolution, the conflict-resolution code needs
   to be able to extract state from object records.

C. When doing blob-aware iteration, a storage needs to be able to
   recognize blob records.

D. When raising conflict errors, the exception instances *want* to get
   the object class name from the database record.

(A) was dealt with by the existing way the referencesf function was
passed into storage pack functions.

I dealt (B) and (C) by expanding and renaming the interface of objects
passed to the storage registerDB call.  These objects now provide an
up-call mechanism from a storage to anything that might wrap it,
including databases, storage servers, and intermediate wrapping
layers.

I punted on (D). You can't always get what you want. :)

I'm mostly convinced what I've done is reasonable, although I'm just a
little uncomfortable.  It has always been the theory that storages
can't count on the contents of database records. Now that is
reality.

I've decided I really want to provide a database compression option. I
could have built something directly into ZODB or used a wrapper
storage. I chose the later because it was more flexible.  I think I
would have run into the same issues without a wrapper storage.

Soon I expect to release a wrapper storage implementation that
provides record compression.

Jim

-- 
Jim Fulton