[ZODB-Dev] Storage API change: Checking for reading out-of-date data

[Christian Theune]

Hi,

On 08/30/2010 11:36 PM, Jim Fulton wrote:
> ZODB used multi-version concurrency control to assure that data read
> are consistent.  It doesn't check that or require data read to be up
> to date.  For read-only transactions, this is approriate.
>
> Even for write transactions, not checking whether reads are up to date
> isn't typically a problem, since the important data read is also
> updated and we check for write conflicts.
>
> The approach used by ZODB is a common one and represents a generally
> good tradeoff between consisntency and performance.
>
> The approach, however, can run into probems when data from one object
> are read and used to update a different object.  I've mistakenly
> tended to view this situation as an edge case.  However, BTrees,
> perhaps the most heavily used data structure in ZODB applications,
> follow this data access pattern. In particular, internal nodes are
> read to determine which subnodes data should be written to. An out of
> date internal node can lead to data in BTrees being missplaced.  This
> doesn't happen very often, and when it does happen, it's been pretty
> mysterious.

Ah. This sounds like the isolation issue that was discussed probably 
years ago. There was a scenario where someone (I swear it wasn't me.) 
built an index structure like this:

init:
    self.index = None

unindex:
    self.index.pop(key)
    if not index:
        self.index = None

index:
    if self.index is None:
        self.index = {}
    self.index[key] = value


It sound to me like the problem with the BTree internals you're 
describing has basically the same issue going on and it feels like it's 
a good thing having a new API that allows to deal with those situations 
properly. (Although I'd rather see code like above not getting written 
at all on the application level.)

Christian

-- 
Christian Theune · ct at gocept.com
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1
Zope and Plone consulting and development