[ZODB-Dev] Proposal: cross database reference seat belt

[Shane Hathaway]

Problem
-------

In multi-database configurations, ZODB applications can unintentionally 
create cross-database references.  The causes include moving (rather 
than copying) an object between containers, storing an object in a 
session database and later moving it to the main database, and using a 
persistent object for a catalog index when the catalog is located in 
another database.

Unintentional cross-database references can cause significant problems. 
  For example, references from non-volatile objects to volatile session 
objects will break when the session expires, leading to application errors.

In a project I am working on, my team decided that configuring our 
application to use a multi-database was too risky unless we had some way 
to prevent unintentional cross-database references.

Proposed Solution
-----------------

I propose an optional "seat belt" for cross-database references in ZODB. 
  The seat belt, when enabled, will prevent most objects from holding 
any cross-database references.  Application policy will specify which 
cross-database references to allow.  When any cross-database reference 
violates the policy, ZODB will raise an exception to help application 
developers track down the policy violation.

Proposed Mechanism
------------------

The ZODB.DB.DB constructor will accept a new parameter, "check_xrefs", 
that defaults to False.  When check_xrefs is True, the cross-database 
reference seat belt is enabled.  "xref" is short for "cross-database 
reference".

Applications will express cross-database reference policy through a new 
method of persistent objects named "_p_check_xref".  The _p_check_xref 
method accepts a single parameter, the object to be referenced in 
another database.  If the reference should be allowed, the _p_check_xref 
method returns True.  If the _p_check_xref method returns False, the 
object serialization machinery will raise an InvalidObjectReference 
exception, leading to transaction abort.

Furthermore, when the seat belt is enabled, ZODB will raise an 
InvalidObjectReference exception when an object with no _p_check_xref 
method attempts to hold a cross-database reference.

Each database in a multi-database has its own check_xrefs setting.  The 
setting applies only to the objects contained in that database.  This 
allows developers to specify, for example, that arbitrary references 
from the main database to the volatile session database are disallowed, 
while arbitrary references from the volatile session database to the 
main database are allowed.

The proposed mechanism has been implemented in a branch of ZODB named 
"shane-cross-database-seatbelt", checked in at svn.zope.org.  It does 
not change very many lines of code.

Some possible risks of the proposed mechanism:

* An implementation of the _p_check_xref method could wake up ghosts as 
a side effect.  Implementers probably need to avoid that.

* People might want to express the application policy without modifying 
persistent classes.  I chose not to propose that kind of solution, 
opting instead for a solution that follows established ZODB patterns.

Conclusion
----------

The proposed new feature is designed to help developers create more 
robust ZODB applications by enforcing cross-database reference policy. 
I hope it can be included in ZODB 3.9.

Shane