[ZODB-Dev] ZODB packing error

[Tim Peters]

[Tim Peters]
>> I expect the outcome depends on the details of the conflict
>> resolution implemented by O.  If, e.g., O is a BTree, the removal of
>> P (maybe that's one of O's keys) probably wouldn't raise a
>> ConflictError.  But in that case, P would also be removed from the
>> final (post-resolution) committed state, so no dangling reference
>> either.

[Toby Dickenson]
> It is obviously possible to contrive a conflict resolution method
> which causes a dangling reference by ignoring the change made in
> the first transaction. It would be a bug for the BTree conflict
> resolution method to do this....

I would also consider it to be a bug if BTrees did this, but I can't point
to any docs to "back up" that claim.  To me, ZODB endcase behaviors in all
sorts of cases appear to have been made up as it went along, and I'm still
struggling to distinguish implementation accidents from intent.  I'm sure
many of these cases *were* clear in the original author(s)' minds -- but if
it didn't get written down, that's a distinction lacking practical value to
me.

> but are there any cases where this would not be a bug? Does zodb need
> to support this?

Since, AFAICT, it doesn't support it, never did, and there aren't even hints
of plans to make ZODB support it, I have to say "no" to "need" <wink>.

> Previously Jim has argued no. At the time I wasnt conviced, but Ive
> never seen a counterexample.

Me neither, but I have no doubt one could be constructed.  For example, many
kinds of Monte Carlo apps couldn't care less what the state is, exactly,
they only care that it's been "suitably" randomized.  Conflict resolution in
that kind of app could consist of picking any one of three competing states
as "the winner", simply ignoring the other two; the winner could be picked
wholly at random too.

BTW, in my view all potential problems here are due to a clear cause:  we
have a garbage collection procedure that doesn't know the true root set.
The root set is approximated, for good pragmatic reasons, and it's usually
an excellent approximation -- but it's still just an approximation.

>> Let me try to construct an actual dangling reference ... <snip>
>> Back to the second client:
>> >>> conn.sync()
>> >>> O[2] = P      # this client still has a ref to P in memory
>> >>> get_transaction().commit()  # and adds it to the BTree
>> >>> st.close()
>> [and the result is a reference to a persistent object that no longer
>>  exists in the database]

> You stored a reference to a persistent object across transaction
> boundaries.

Yes.

> I dont think we have ever declared whether or not this is permitted.

Insert repeat of blurb about "endcase behaviors" above.

> To me it smells deeply dangerous, and there are several places where
> Zope goes to lengths to avoid doing this. For a start, you've lost
> your I from ACID.

Sorry, I don't follow.  Seems to me the new transaction that implicitly
began after the first commit() is as isolated as the transaction that did
the first commit().  Maybe that's wrong.

> I think I would like to declare this unsupported, but Im not entirely
> comfortable with that either. As Tim demonstrated, it is _so_ easy to
> do.

I would like ZODB to detect dangling references at commit() time, and raise
a subclass of ConflictError then.  DirectoryStorage does do so, right?

> Do we have any causes of 0-day-pack losses that are not arguably
> application bugs?

I don't, but I'm not an expert here.  If anyone does have one, they've been
mighty shy (this thread is over a week old now).