[ZODB-Dev] Corrupted ZODB - tried fsrecover and tranalyzer - any other options?

[zope at meadowbase.com]

I'm running Zope 2.5.1 on OpenBSD. My ZODB grew to about 500 megs - we had
been uploading a lot of large-ish binaryies (sometimes 10's of megs at a
time) and not paying a lot of attention to the database (no packing, no
backups).

This evening a major portion of the database suddenly became corrupted. I
use a bunch of folder-like ZClasses to hold content and files - just a
simple way to assign some basic properties and methods to similar areas of
the database. One of my ZClass definitions is "broken", so all the instances
of the ZClass are "broken" - meaning that I couldn't get to any of my
content.

When I try to browse into the ZClass definition in Products, the debug
output shows:

-------

2003-04-28T05:30:26 ERROR(200) ZODB Couldn't load state for
'\x00\x00\x00\x00\x00\x00\xb7j'
Traceback (innermost last):
  File /usr/local/lib/zope/lib/python/ZODB/Connection.py, line 443, in
setstate
  File /usr/local/lib/zope/lib/python/ZODB/Connection.py, line 123, in
_persistent_load
    (Info: .l)
  File /usr/local/lib/zope/lib/python/ZODB/Connection.py, line 82, in
__getitem__

... huge ugly blob of text...

Could not load oid .l, pickled data in traceback info may            contain
clues

-------

Much of the database is intact, just objects inside those ZClasses were
broken.

It doesn't seem like the thing I was doing triggered the corruption in any
specific way - I had just imported an object from another database, and was
editing and testing a couple Python Script objects.

I took a look with tranalyzer.py, and here are the last few lines of the
output:

-------

TID: 3497BC62E2BAD77 @ 193089105 obs 1 len 6287 By  francis
"/helium/hel-infra/admin_nav/help/url_explanation/changeProperties"
        OID: af2d len 6157

TID: 3497BC67C775A19 @ 193095408 obs 1 len 6291 By  francis
"/helium/hel-infra/admin_nav/help/url_explanation/changeProperties"
        OID: af2d len 6161

TID: 3497C4D03769BB4 @ 193101715 obs 1 len 3149 By  francis
"/helium/home/ZPythonScriptHTML_editAction"
        OID: 8cbd len 3043

TID: 3497C4E75BF6980 @ 193104880 obs 1 len 3184 By  francis
"/helium/home/ZPythonScriptHTML_editAction"
        OID: 8cbd len 3078

TID: 3497C4F7BDA5ECC @ 193108080 obs 1 len 1374 By  francis
"/helium/content_div_loginForm/pt_editAction"
        OID: 8cbf len 1266

TID: 3497C5221AC5A22 @ 193109470 obs 2 len 2160 By  francis
"/helium/manage_addVersion"
        OID: 8cb5 len 1871
        OID: af2e len 157

****** TRUNCATED TRANSACTION at 193111646:

-------

Two things are interesting here:
(1) The last transaction is manage_addVersion. We don't use versions
regularly but I did experiment with them a little in this database.
(2) These transactions are from several months ago - there has been
substantial activity since then.


I copied the database to another file, then ran fsrecover. It gave a bunch
of output (I didn't save it, but can run again if it's helpful) and handed
me a new file. I launched Zope again with the new file, and here's what I
got:
 - The product containing those ZClasses, as well as the entire section
   of the ZODB where I had built content using the ZClasses, was gone.
 - Several objects were marked with version locks - even though I haven't
   used Versions in months.
 - I can browse content but can't add anything, not even a Folder. When I
   try to add a Folder I get ZODB.POSException.ConflictError

So the corrupted pre-fsrecover database is actually more useful than the
recovered one!

I'm *not* ruuning ZCatalog or ZEO.

The lost content is critical to my company, and so I would appreciate input
on:
 - Other approaches (besides fsrecover and tranalyzer.py) to recover a
   corrupted ZODB?
 - Any way to get the content of the objects with the broken ZClass?
 - Any thoughts on what might have caused this (so we can avoid in in the
   future)?

Thanks!

--- Francis Potter
--- Helium LLC