[ZODB-Dev] RelStorage and PosKey errors - is this a risky hotfix?

[Laurence Rowe]

On 24 January 2011 21:28, Shane Hathaway <shane at hathawaymix.org> wrote:
> On 01/24/2011 02:02 PM, Anton Stonor wrote:
>> Hi there,
>>
>> We have recently experienced a couple of PosKey errors with a Plone 4
>> site running RelStorage 1.4.1 and Mysql 5.1.
>>
>> After digging down we found that the objects that were throwing
>> PosKeyErrors  actually existed in the object_state table with pickles
>> etc, however not in the current_object table.
>>
>> After inserting the missing pointers into the current_object  table,
>> everything worked fine:
>>
>>    mysql> SELECT zoid, tid FROM object_state WHERE zoid="561701";
>>
>>    +--------+--------------------+
>>    | zoid   | tid                |
>>    +--------+--------------------+
>>    | 561701 | 255267099158685832 |
>>    +--------+--------------------+
>>
>>    mysql> INSERT INTO current_object(zoid, tid) VALUES('561701',
>> '255267099158685832');
>>
>> Looks like it works -- but is this a safe way to fix PosKeyErrors?
>>
>> Now, I wonder why these pointers were deleted from the current_object
>> table in the first place. My money is on packing -- and it might fit
>> with the fact that we recently ran a pack that removed an unusual large
>> amount of transactions in a single pack (100.000+ transactions).
>>
>> But I don't know how to investigate the root cause further. Ideas?
>
> This suggests MySQL not only lost some data (due to a MySQL bug or a
> filesystem-level error), but it failed to enforce a foreign key that is
> supposed to ensure this never happens.  I think you need to check the
> integrity of your filesystem (e2fsck -f) and database (mysqlcheck -c).
> You might also reconsider the choice to use MySQL.

Must this imply a failure to maintain a foreign key constraint? While
there are FK constraints on current_object (zoid, tid) -> object_state
(zoid, tid) there is no foreign key that might prevent a
current_object row from being incorrectly deleted.

I think that means the possibilities are:

1. The current_object table was not updated properly during a commit
or corrupted so that some rows were lost.

2. Something goes wrong during pack gc (either in the pack logic or on
the database).

3. Database corruption.

Laurence