[ZODB-Dev] removing invalid object references

[Tim Peters]

[ethan fremen]
> I'm trying to repair a Data.fs that seems to have a few issues. Mostly,
> they involve references to objects that don't exist- this is after doing
> fsrecover.py on the database.

So fstest.py runs without complaint now?

> Here's the report from fsrefs.py:
>
> http://mindlace.net/tmp/badreferences.txt
>
> For the moment, I'm willing to nuke all the message objects that refer to
> invalid attachments; I don't know what (if anything) I should do about
> the IOBuckets.

That was plural, but I only see one missing IOBucket mentioned in the
report.

More troublesome is how that report ends:

Traceback (most recent call last):
  File "fsrefs.py", line 107, in ?
    main(path)
  File "fsrefs.py", line 81, in main
    refs = get_refs(data)
  File "fsrefs.py", line 43, in get_refs
    u.noload()
cPickle.UnpicklingError: invalid load key, ''.

That means at least one object pickle is so corrupt that cPickle can't make
any sense out of it, and so fsrefs.py just died.  There are no tools that
can repair corruption inside object pickles.  If that's necessary, it would
need to be done by hand, using semantic knowledge about the app in context.

But before anything like that, I note that all the objects complained about
before fsrefs.py died were last modified in April of 2004.  IOW, they're
old, so there's a decent chance that packing the file will throw away at
least some of the damaged objects (fsrefs.py has no idea whether objects are
still reachable from the root, it simply looks at the most-recent revision
of every object in the database, whether or not still reachable).

And be sure to make a copy of the .fs file before every attempt at
transformation; keep them all around until you're satisfied the file has
been repaired.