[ZODB-Dev] How to fix "CorruptedDataError: Error reading unknown oid. Found '' at 81036527"?

[Jim Fulton]

On Thu, Jul 14, 2011 at 10:12 AM, Andreas Jung <lists at zopyx.com> wrote:
>
>
> Jim Fulton wrote:
>>>
>>> 2011-07-14 13:36:22 INFO Z2 Shutting down
>>
>> That error message seems rather weird. WTF SIGINT?
>
> Likely caused during the Zope shutdown amd unrelated).

I find this a bit unlikely, unless there was also a system shut down.
FileStorage goes to great pains to prevent this sort of corruption.
I don't think it's possible for a process shut down alone to
cause this.

I could imagine this sort of corruption if the system was shut
down before data could be synced to physical disk.

>>
>> Use multi-zodb-check-refs with the -r option to get a database of
>> reverse references.
>> Use that do figure out what's refering to the missing object and fix
>> that object(s) so that
>> it no longer does.
>
> This gives me one reference:
>
> plone at web1:~/poskeyerror-check$ cat Out
>>>> for x in refs.items(): print x
> ...
> ('1', <BTrees.LOBTree.LOBTree object at 0xe7a850>)

That's sort of interesting. So my guess is that the missing object is
a bucket and was the first bucket in the BTree, otherwise there'd also
be a reference from the previous bucket.

>
>
> I followed the documentation at
>
> http://pastebin.com/bL0CbBm2

I don't know where you got that. I didn't write it. :)

> So what do I do with the oid 0xe7a850 in this case?

Well, ideally, you'd update it to not include a reference to the
broken object.

> Trying to load the referenced object gives me of course a PosKey error:

Dang. I would want it to give you a broken object.  I think there
might be a launchpad issue for that.  Not sure if 3.10 has the fix
(not sure what you're using).

I wouldn't normally suggest what I'm about to suggest, but I think you
have enough ZODB foo. :)

I would try adding a new bucket with that oid. Maybe you can even
stuff it in the connection cache to avoid committing it to the
database.  I'm not sure off hand what the steps required would be, but
I think you can figure it out.  Once you get an object in the cache,
you should be able to load the container, modify it to not include the
subobject, and commit it.

You may also need to follow references to the container, using the
reverse-references database to get a handle on what was happening at
the application level.

Alternatively, if you think this was due to a non-graceful shutdown
while committing, there might be a transaction you can undo.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton