[ZODB-Dev] High Write Applications

[Phillip J. Eby]

At 06:36 PM 7/31/03 +0100, Chris Withers wrote:

>RDBS can handle high write volumes, so why can't ZODB be made to behave 
>similarly?

The root causes is that RDBMS transaction logs record high-level details, 
not blob snapshots.  For example, when an RDBMS logs that you changed row 
#2967's "foo" column to value "bar", it doesn't usually also log the entire 
data page the row was contained in, plus copies of all the b-tree index 
pages that changed as a consequence of the change.  Thus, ZODB's disk usage 
per write transaction generally exceeds RDBMS disk usage for the same 
transaction by at *least* an order of magnitude, even before catalog 
indexes come into play.


>This is just throwing this out here to stimulate some discussion, I'm not 
>bitching, although I would eb over the moon if we can find some way to 
>make ZODB  handle high write volumes...

To do this, ZODB would have to be able to understand and log *differences*, 
rather than just snapshotting object states.  And, it would need to be able 
to manage periodic checkpointing, so that recovering a database wouldn't 
require rerunning all the transactions that had ever been done on it.

By the way, if you've looked at systems like Prevayler, you'll notice that 
what I'm describing is actually pretty similar: high-level parameter log, 
plus periodic checkpoints.  Of course, that is also *exactly* what RDBMS 
systems do, except that an RDBMS's "current state" is on disk as well as in 
RAM, and checkpoints are done by flushing dirty buffers to the on-disk state.