[ZODB-Dev] [zopefoundation/ZODB] 49919d: test for POSKeyError during transaction commit

[Jim Fulton]

On Wed, Feb 5, 2014 at 9:23 AM, Godefroid Chapelle <gotcha at bubblenet.be> wrote:
> Le 05/02/14 13:25, Jim Fulton a écrit :
>
>>>>> Acquisition is added as a test dependency. Any hint how to replicate
>>>>> >> >the bug without acquisition is welcome.
>>>>
>>>> >>
>>>> >>Define a subclass of Persistent which emulates what Acquisition does,
>>>> >> e.g.:
>>>> >>
>>>> >>   from persistent import Persistent
>>>> >>   class Foo(Persistent):
>>>> >>       @property
>>>> >>       def _p_jar(self): # or whatever attribute trggers
>>>> >>           return object()
>>>
>>> >
>>> >What if full replication requires a C extension module?
>>> >
>>> >(I hope that's not true and that it is possible to reproduce the bug
>>> >using some fakes, but I haven't spent the time investigating this.)
>>
>> I'm going to dig into this.  I'm baffled by the assertion that this has
>> anything to do with readCurrent.
>
>
> For sure : the POSKeyError happens during connection.commit when checking
> oids stored in Connection._readCurrent mapping.
>
> (see traceback at http://rpatterson.net/blog/poskeyerror-during-commit)
>
> The _readCurrent mapping is populated only by calls to
> Connection.readCurrent method.
>
> In the Plone code base, the only way I found to get that
> Connection.readCurrent method to be called is by adding a key value pair to
> a BTree.
>
> _BTree_set C function is then called, which in turn calls readCurrent by
> inlining the PER_READCURRENT macro.
>
> This calls the cPersistence.c readCurrent function, which in turn calls
> readCurrent method on the ZODB connection.

Wow.  I had to dig a bit to remind myself (vaguely) why I added this.


>
> When setting a key value pair on a new (not already committed) instance of a
> standard BTree, readCurrent method is not called on the connection.

This is with your change, right?


>
> My understanding is that it is due to the fact that _p_jar and _p_oid are
> only set during transaction commit.

They can be set earlier by calling the connection add method.

This is used often for frameworks that use object ids at the application level.


>
> However, with a new BTree instance that also inherits from
> Acquisition.Implicit, readCurrent method is called on ZODB connection when
> setting key value pair. The only explanation I found is that this instance
> _p_jar attribute has a value (acquired in a way or another ?).

You could also simulate this by adding an object to a connection using
a connection's add method.

Wanna update the test to use this technique instead?

>
> In this case, when readCurrent is called on an object created during a
> savepoint and this savepoint is rolled back, the oid is leftover in the
> Connection._readCurrent mapping. This leads to the POSKeyError when
> committing later as checkCurrentSerialInTransaction cannot check the object
> since it went away at rollback.
>
> This brings us to the fix I propose: calls to readCurrent should not track
> objects with oid equal to z64.

...

> This was a very long explanation which I hope will help to confirm the fix
> or to come up with a better one.
>
> PS: keep in mind that english is not my mothertongue.

:) You do very well.

I think your fix is correct.  As you point out, It doesn't make sense to
guard against conflicts on new objects.

I think a cleaner test could be written using the connection add method.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton