[ZODB-Dev] ZODB4 project plan

[Phillip J. Eby]

At 03:53 PM 12/1/02 +0200, Pieter Nagel wrote:
>On Thu, Nov 28, 2002 at 07:03:50PM -0500, Phillip J. Eby wrote:
>
> > If it's not the transaction system itself that you're testing, why not 
> just
> > abort the "inner" transactions, and don't bother having an "outer"
> > one?  Just curious.
>
>Because of the things one wants to test is whether components commit
>or abort when they are supposed to; and yet retain the ability to rollback
>one's test database to a clean slate at the end of the test by aborting
>the outer transaction.

But doesn't that mean it's the transaction system that you're testing?


>This scheme presupposes a slightly different transaction API: when one
>begins a transaction, the transaction may wind up either being an outer
>transaction, or, if an outer transaction is already open, a new
>subtransaction is opened. One can easily build this on top of the current
>ZODB API.

Hm.  Based on my limited experience with such API's, I find them a bad idea 
because they "allow errors to pass silently".  That is, you can 
accidentally begin an inner transaction an extra time, and then think 
you're committing your transaction when in fact you're just committing the 
inner one.  You might then go on to do several things, and then think 
you're rolling something back, but in fact you're rolling back several.

Therefore, if there is a nested transaction API, I think it should *not* be 
transparent.  Because most people most of the time need only one level of 
transaction, they should not be subject to the possibility of masked errors 
for the few people who really need nesting.

That's why I like the checkpoint/rollback approach currently in ZODB4 - 
it's explicitly different and requires you to be in a started transaction 
already.  Plus, it doesn't hurt to hang onto multiple checkpoints, because 
as long as the outer transaction commits, they're committed.


>This ensures that the tested components end up commiting and aborting
>"real" transactions when the are used in production code.

I still don't see that "components" in the sense of domain objects should 
ever be messing with transactions.  It seems to me that they shouldn't even 
know that there is a such thing as a transaction in the first place, let 
alone try to change its scope.

If an object wants to guarantee that some operation it performs is atomic 
in the absence of an actual error condition, this is generally a 
straightforward thing to do (Command/Undo pattern).  In the presence of an 
actual error condition, a top-level transaction suffices to ensure data is 
restored to its original state.  By "actual error condition" I mean an 
unrecoverable system failure or programming error, not just any exception 
raised by an operation.


>This kind of API also simplifies the style of programming holger krekel
>wrote about earlier in the thread:
>
> > Maybe it's just me but i like to move into a programming paradigm where i
> > can write a "Component" which opens its own transaction regardless if 
> it is
> > part of a outer transaction or not.
>..
> > An example. In a massive multiplayer universe i have lots of logics
> > how the objects in this world interoperate.  Some components determine
> > the outcome of local or wide-area fights, others implement 'research 
> development'
> > and so on.  If the 'local fight' component detects a problem then it
> > should be able to rollback to a consistent state (it may already have
> > modified some objects in its substransaction).  It shouldn't be forced
> > to care about other transactions it is also part of.  When and how to
> > persist the game state seems like an (almost) orthogonal issue to me.

Thing is, if it's orthogonal to storage, then it doesn't need to be part of 
the top-level transaction API; an arbitrarily selected "undo" component 
could do the job.  The "fight" component should simply issue Command 
objects to those local components, and if need be, simply reverse them by 
issuing compensating commands in reverse order.  Such a facility can be 
better tuned to the specific requirements needed for that application, and 
requires no system-level support.  Different components can use different 
undo facilities if they like.  There are a lot of advantages to an 
application-specific command model, as well; commands can be issued by the 
UI, they can be logged, etc.

Also, command-based systems are much easier to write *correctly* than 
transaction systems, because they don't need to guarantee correctness 
despite *any* error.  If an actual error condition occurs, the 
transaction's job is to ensure consistency of the system as a whole, 
including any persistent command queues or logs.

Last, but *far* from least, is that ZODB checkpointing is almost certainly 
going to use more memory or time+disk space to be able to save your 
intermediate states, than an application-specific undo will require, in 
most cases.  ZODB will have to checkpoint the full state of each 
currently-modified object at each nesting level - i.e. make a copy of it 
and put it somewhere.  And it will have to do this even if the new 
subtransaction is only going to change one field of one additional 
object.  If you've already changed fifty objects, you have to copy all of 
them, because you might change one of them in the subtransaction.

A command log, by contrast, will only need to keep enough information to 
restore the fields actually changed in the "subtransaction", and can be 
discarded as soon as it is no longer needed.

The really interesting thing about Holger's example, is that it's something 
that really would be better off with Command objects in the first 
place.  The "local fight" object really sounds like a command object: it 
manages an operation as a "verb", rather than being a "noun".  It calls 
methods and manages the state of an operation over time.  It's a "composite 
command".  And it really is orthogonal to any transaction(s) taking place 
at the application scope.

This is also a lot like workflow applications, which effectively perform 
long-running transactions.  They too issue commands which are logged and 
may need to be undone.  The main difference is that they operate "above" 
the transaction scope, and the individual commands are executed atomically 
as transactions.

I guess my take on all this is that transactions are used to ensure atomic, 
consistent, and durable storage of persistent objects.  Process management 
is the job of the objects themselves, not the transaction system.  Process 
management systems can be mixed and matched for the job or tuned to the 
application domain, while the transaction system needs to be systemwide and 
oh-so-reliable, giving you the freedom to think only about 
application-level concerns.