[ZODB-Dev] SQL persistence based on ZODB4 Persistence/Transaction packages

[Phillip J. Eby]

I'm currently working on a design for doing persistence to/from arbitrary 
databases (but primarily SQL, and secondarily LDAP) using ZODB4's 
Persistence and Transaction packages (ignoring the 'ZODB' package).  This 
persistence layer is for PEAK, the Python Enterprise Application Kit 
("TransWarp in a business suit"), and is intended to support pretty much 
arbitrary underlying database formats.

The detailed design is not finished yet, but I've got the big picture quite 
clear at the moment, based on a "shelf" full of "jars" which implement 
Transaction.IDataManager and Persistence.IPersistentDataManager, and using 
the existing ZODB4 cPersistence API and base classes.

I was skimming through the archives of this list, looking for helpful 
information or news, and noticed a mention of changing API's.  Can anybody 
say more about the planned direction of those changes, or perhaps point me 
to some documents or persons with the info?  I'd like to make sure I'm not 
basing my design work on invalid premises about how things will be 
structured.  Are we talking about major changes at the Python level, or 
just stuff that affects C extensions?  (I don't expect to have any C 
extensions of any consequence, at least initially.)  Any information would 
be helpful.

On a related note, I've noticed that the ZODB4 cache manages its own weak 
references, but it doesn't seem to do anything outside of the gc methods 
that weakref.WeakValueDict doesn't.  Perhaps it would be simpler to just 
subclass WeakValueDict?  You could still keep the '__active' dictionary, 
but just treat it as a set of keys, rather than as a 
dictionary.  __delitem__ would need to remove the key from __active, if it 
existed, __setitem__ would still need to check the state, and setstate() 
would just do a __setitem__, but I think that's about it.  Then you 
wouldn't need to move things from __ghosts to __active and back, and check 
both dictionaries on lookup operations.  The get(), __len__(), and 
__getitem__() methods and the _dictdel class would disappear altogether, 
some of the other methods would get shorter, and lookup performance should 
improve, assuming that one lookup in a larger dictionary is faster than two 
lookups in smaller ones plus some intervening Python logic.

But I actually originally meant to talk about the caching API, not its 
implementation.  :)  Specifically, I'm wondering if there's anything in 
ICache or Cache that is depended on by anything else in the Persistence 
package, or if it's really there for the benefit of the ZODB package.  I 
expect my persistence machinery to have rather different caching needs, so 
I don't plan to be implementing ICache (or even using the Cache 
implementation of it) for most of the caches in the framework.  Mostly, 
I'll be using either straight-up WeakValueDict or very slight modifications 
thereof, and occasionally dictionaries that get cleared at transaction 
boundaries.  (So you can see how I drifted off-topic above...)  Anyway, I 
just wanted to make sure, once again, that I wasn't planning to do Very Bad 
Things in the light of the now and future ZODB4.

Any info that anyone could share would be appreciated.  Thanks!