[ZODB-Dev] Re: ZODB packing error

[Casey Duncan]

On Sun, 4 Apr 2004 23:22:44 -0400
"Tim Peters" <tim at zope.com> wrote:

> [Marius Gedminas, on 0-day packing]
> > ...
> > IOW there is no corruption.  I must have misunderstood the
> > discussions about 0-day packing I skimmed on zope and zodb-dev.
> 
> Oh, I'm not sure about that.  I'm not a ZODB expert (although I expect
> to become one).  Things can certainly get complicated.  For example,
> 
> - Transaction T1 loads an object O with a reference to persistent
> object
>   P, and modifies some other attribute of O (it does not load P
>   itself).
> 
> - Other transactions remove the reference from O to P, and all other
>   references to P, from the database.
> 
> - A pack to current time is done.  Now the only reference to P lives
> in
>   memory, inside T1's view of O.
> 
> - T1 finally tries to commit.
> 
> At this point, I see nothing to prevent O from having a "dangling
> reference" to P -- T1 itself never loaded P into memory, and the
> database no longer contains any trace of P, but O still references it.
>  That probably isn't
> good.

It seems like a possible solution to this is for pack to account for all
open transactions when the pack starts. 

When a pack is executed it must check if there are any transactions
in-progress that began before the time to pack to. If there are such
transactions, one possible solution is to wait for them to complete
before actually packing. A stupid way to implement this for Zope might
be to raise ConflictError in pack and have the pack request retried.
Another option would be to simply have pack sit in a wait loop until the
transactions commit, then pack from the time the transactions are done.
On a busy system this may mean the pack never really executes,
however...

Another approach might be to increase the pack time automagically so
that it falls just before the start time of all transactions that are
in-progress. This would mean that "pack 0" actually might me "pack to 1
minute ago" if a transaction that started a minute ago was still going
when the pack was invoked.

How does pack currently deal with Versions? I suspect those complicate
things.

-Casey