[ZODB-Dev] RelStorage and PosKey errors - is this a risky hotfix?

[Shane Hathaway]

On 01/24/2011 02:02 PM, Anton Stonor wrote:
> Hi there,
>
> We have recently experienced a couple of PosKey errors with a Plone 4
> site running RelStorage 1.4.1 and Mysql 5.1.
>
> After digging down we found that the objects that were throwing
> PosKeyErrors  actually existed in the object_state table with pickles
> etc, however not in the current_object table.
>
> After inserting the missing pointers into the current_object  table,
> everything worked fine:
>
>    mysql> SELECT zoid, tid FROM object_state WHERE zoid="561701";
>
>    +--------+--------------------+
>    | zoid   | tid                |
>    +--------+--------------------+
>    | 561701 | 255267099158685832 |
>    +--------+--------------------+
>
>    mysql> INSERT INTO current_object(zoid, tid) VALUES('561701',
> '255267099158685832');
>
> Looks like it works -- but is this a safe way to fix PosKeyErrors?
>
> Now, I wonder why these pointers were deleted from the current_object
> table in the first place. My money is on packing -- and it might fit
> with the fact that we recently ran a pack that removed an unusual large
> amount of transactions in a single pack (100.000+ transactions).
>
> But I don't know how to investigate the root cause further. Ideas?

This suggests MySQL not only lost some data (due to a MySQL bug or a 
filesystem-level error), but it failed to enforce a foreign key that is 
supposed to ensure this never happens.  I think you need to check the 
integrity of your filesystem (e2fsck -f) and database (mysqlcheck -c). 
You might also reconsider the choice to use MySQL.

If you're sure you have found all of the corruption, you can do this 
(with all app servers shut down) to re-create the current_object table:

delete from current_object;
insert into current_object (zoid, tid)
     select zoid, max(tid) from object_state;

Shane