[ZODB-Dev] ZODB Problem

[Tim Peters]

[Alexey V Paramonov]
> I have already solved the problem:

Congratulations!

> I have several backups of Data.fs, one of them (very old) was "alive".
> So, I starget to cut pieces from the end of Data.fs and tried to load
> it, so after a few hours I knew which trasaction was malfromed.

|------------|
|            |
|   Good     |
|            |
|            |
--------------   transaction 00032
| Corrupted  |
|     data   |
|            |
|            |
|            |
|            |
--------------   transaction 00033
|            |
|            |
|   Good     |
|            |
|            |
|            |
--------------

> I decided to cut this transaction from the good database and insert it to
> the existing broken one. The main probem was that theese to pieces had
> different sizes (32 transaction is about 150 MB %| ), so I spent one
> nigth + day figuring out how ZODB stores it's structure and simply set
> "correct" sizes on transactions and data structures. I just filled mising
> pieces with zeros, ran fstest.py on my merged Data.fs, cleaned up
> everything and now Zope eats this file finely. I will not use this file,
> of course, but at least I could export all needed data just pressing
> buttons in Zope :)

Now you know why there's no automated way to recover from arbitrary damage.
Don't know whether "0032" and "0033" are decimal or hex, but it looks like
they're early transactions in the life of this database.  Given that
FileStorage grows only by appending:

    http://zope.org/Wikis/ZODB/FileStorageBackup

do you have any idea how bytes early in the file may have gotten damaged?
There are rare reports of that, but outside of flaky hardware or system
software no cause is evident.  It's scary when once-good data goes bad for
no identifiable reason.