[ZODB-Dev] RelStorage now in Subversion

[Shane Hathaway]

Dieter Maurer wrote:
> Shane Hathaway wrote at 2008-1-31 11:55 -0700:
>> ...
>> Yes, quite right!
>>
>> However, we don't necessarily have to roll back the Postgres transaction
>> on every ZODB.Connection close, as we're doing now.
> 
> That sounds very nasty!
> 
> In Zope, I definitely *WANT* to either commit or roll back the
> transaction when the request finishes. I definitely do not
> want to let the following completely unrelated request
> decide about the fate of my modifications.

No, RelStorage doesn't work like that either.  RelStorage opens a second
database connection when it needs to store data.  The store connection
will commit at the right time, regardless of the polling strategy.  The
load connection is already left open between connections; I'm only
talking about allowing the load connection to keep an idle transaction.
 I see nothing wrong with that, other than being a little surprising.

>  and you read older and older data
> which must increase serializability problems

I'm not sure what you're concerned about here.  If a storage instance
hasn't polled in a while, it should poll before loading anything.

> (Postgres might
> not garantee serializability even when the so called isolation
> level is chosen; in this case, you may not see the problems
> directly but nevertheless they are there).

If that is true then RelStorage on PostgreSQL is already a failed
proposition.  If PostgreSQL ever breaks consistency by exposing later
updates to a load connection, even in the serializable isolation mode,
ZODB will lose consistency.  However, I think that fear is unfounded.
If PostgreSQL were a less stable database then I would be more concerned.

> I agree that it can reduce traffic but I am almost convinced that
> the price will be high (in either "cannot serialize concurrent updates"
> or not directly noticable serializability violations).

RelStorage only uses the serializable isolation level for loading, not
for storing.  A big commit lock prevents database-level conflicts while
storing.  RelStorage performs ZODB-level conflict resolution, but only
while the commit lock is held, so I don't yet see any opportunity for
consistency to be broken.  (Now I imagine you'll complain the commit
lock prevents scaling, but it uses the same design as ZEO, and that
seems to scale fine.)

Note that Oracle has an additional isolation mode called "read only".
It provides the same consistency guarantees as the serializable mode,
but apparently takes advantage of read only optimizations to provide
dramatically better throughput than what I've been able to get out of
PostgreSQL.  I would use that isolation mode in Postgres as well if it
had that feature.

Shane