[ZODB-Dev] Retrieval of contents of corrupted plone folder

[Tim Peters]

[Jim Rippon]
> ...
> ------ 2005-11-07T15:03:51 ERROR(200) ZODB Couldn't load state for 0x1655
> Traceback (most recent call last):
>   File "/data/newsite/lib/python/ZODB/Connection.py", line 600, in
setstate
>     self._set_ghost_state(obj, p)
>   File "/data/newsite/lib/python/ZODB/Connection.py", line 639, in
set_ghost_state
>     state = unpickler.load()
> EOFError

The pickle for the object with oid 0x1655 is corrupt.  Here's an _example_
of the kind of pickle damage that could cause that:

>>> import cPickle
>>> from cStringIO import StringIO

Create a pickle (serialized string form) of list [1,2,3,4]:

>>> p = cPickle.dumps([1, 2, 3, 4], 1)
>>> p
']q\x01(K\x01K\x02K\x03K\x04e.'

Load the pickle; this should work fine:

>>> f = StringIO(p)
>>> u = cPickle.Unpickler(f)
>>> u.load()
[1, 2, 3, 4]

And it did work fine.  Now replace the pickle's last byte with a NUL byte
(chr(0)) and try again:

>>> p = p[:-1] + '\x00'
>>> p
']q\x01(K\x01K\x02K\x03K\x04e\x00'
>>> f = StringIO(p)
>>> u = cPickle.Unpickler(f)
>>> u.load()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
EOFError

The unpickler complained because it hit the end of the pickle string while
it was still looking for "meaningful" bytes.

Now we don't know what oid 0x1655 _is_ in your app, neither how badly the
pickle is damaged, nor how widespread corruption may be.  This is low-level
corruption, and it's rarely obvious what to do in such cases.  As you
discovered, fsrecover can't help (fsrecover doesn't know anything about
object pickles; it works at a level even lower than that).

Possibilities at this point include:

- Revert to using a recent backup.

- Figure out what oid 0x1655 "means" and repair the problem
  based on your knowledge of your site and of the semantic role
  0x1655 plays in your site.  fsdump.py may help.

- Analyze oid 0x1655's pickle string (the `p` argument passed to
  `_set_ghost_state` in your traceback), hoping that the
  damage is small and obvious (for example, in the case I
  concocted above, the damaged pickle ends with chr(0) instead
  of with '.', and that's easy enough for a pickle expert to
  spot and to fix -- although do note it's unlikely your damage
  is that simple).