[Zope3-dev] RDB support

[Phillip J. Eby]

At 12:03 PM 4/10/02 -0600, Casey Duncan wrote:

>ZSQL methods are recordset generators. Perhaps we should abstract that to 
>a common base interface so that anything generating record sets, like 
>catalogish things, sql methods, xml-rpc/wddx and any other tablish things 
>(tinytables?).
>
>Even better I think we might start to think about ORMappings and make 
>recordsets just a simplistic implementation of them. So, we should 
>probably abstract this from a basic interface for mapping objects to records.

I've done a significant amount of work (measured in man-months) on the 
above two notions in TransWarp.  I don't know how applicable the 
implementation is to the Zope 3 core or what other people will want, but 
what I *have* tackled is:

* Uniform read/write API for "records" stored in LDAP, relational, or other 
data sources

* Retrieval and cache lookup by single or multi-field unique keys

* Cache consistency/coherency management so that any retrieval always 
results in the same record object, so long as the record is in cache (and 
it stays in cache at least as long as anything holds a live reference to 
it).  Note that this management is required to prevent the possibility of 
"lost updates" internal to an application which buries the record retrieval 
in its lower implementation layers.

* Handling of inheritance between record types (e.g. for LDAP object types 
and relational inheritance patterns) - specifically, the system ensures 
that regardless of what record type you look something up under, you still 
get the same underlying record object which "knows" its most-specific-types.

* Update queueing and flushing to minimize unnecessary single-field updates 
to the underlying database.

If anybody's interested, check out the TW.Database package from TransWarp 
CVS.  The "Connections" module is of little relevance to this, and the 
"Interfaces" module is lagging behind the implementation at the moment, but 
the "DataModel" and "LDAPModel" modules should provide interesting reading.

The implementation is at present alpha quality, but there is extensive 
design backed by lessons learned from ZPatterns and relational DB 
applications built atop ZPublisher.

The model does not address application-level OR mapping; indeed it doesn't 
even address concepts like associations.  It is strictly a "flat record" 
management layer, upon which higher-order systems can build.  Actually, it 
doesn't really have a notion of a recordset per se.  Recordsets are just 
sequences of records, where each record is sync'd with the cache manager 
for that record type in order to ensure that no matter the query a record 
is returned in (within a given thread/transaction), it's the same record 
object.  Thus, any writes made to it in the current transaction will show 
through to any other reference to that record which is being held by 
another application object.