[ZODB-Dev] ConflictError: database conflict error

[Tim Peters]

[Bob Horvath]
> More interesting data....
>
> http://files.horvath.com/tranalyzer.out
>
> ... ends with
>
> Traceback (most recent call last):
>   File "tranalyzer.py", line 333, in ?
>     main(sys.argv)
>   File "tranalyzer.py", line 325, in main
>     t = Transaction(f, off, refs)
>   File "tranalyzer.py", line 170, in __init__
>     self.obs.append(Record(f, refs))
>   File "tranalyzer.py", line 214, in __init__
>     data = f.c(dl - 8)
>   File "tranalyzer.py", line 105, in c
>     self.checkleft(n)
>   File "tranalyzer.py", line 87, in checkleft
>     raise 'Not enough data left, %d < %d' % (self.left, n)
>
>
> Not enough data left, 14766 < 17567408390136

Sorry, I don't know what that's about.  Does it refer to

    http://zope.org/Members/tsarna/Tranalyzer

?  That was Google's top guess.  If so, I haven't used it and don't know
what the message really means.  Offhand I'm not inclined to believe it:
it's saying it wants to read 17567408390136 bytes but only 14766 bytes
remain, while reading up a data record.  But that kind of low-level detail
is checked by fstest.py too, and you didn't get complaints from the latter.
Neither blob of code is easy to understand, but I've spent hours staring at
(and fixing bugs in) fstest.py, so have some trust in that one.

> Reading this stuff now, I wish I would have known it when this all
> started.  Deleting from before the error is not an option.  I would lose
> too much.  It would be easier to start from scratch and rebuild with what
> I want to recreate, but lose a lot of stuff in the process (although
> probably not such a bad idea housecleaning wise).
>
> Are database errors in the middle of a Data.fs hopeless?

Uusally not, but by all accounts they're rarely easy to fix.  While I
participate in these threads so that people don't feel too lonely <wink>, I
should note that I've never repaired an error pointed out by fsrefs.py.
That's simply because, in my time here, Zope Corp has never had an .fs file
that fsrefs.py didn't like (or, if we did, nobody told me about it -- and
that's not because they're shy about dumping .fs problems on me <wink>).

Ethan Fremen finished repaired a damaged .fs file last week, and I suspect
it started in worse shape than yours.  See the list archives for last week.
Maybe you could interest him in taking on yours.  A problem is that once
damage has persisted beyond the point of a backup, there's no purely
mechanical way to repair anything -- someone has to spend (and possibly a
lot of) time figuring out what role(s) the damaged objects play in your
application, and craft ways to worm around the damage that minimize
collateral damage to the rest of the app's data.  That usually requires
knowing a lot about your app and how you use it, which pieces of your data
are and aren't critical, how they all fit together, etc.

It's somewhat akin to, say, pieces of your Unix mbox file getting
overwritten with nonsense.  How do you repair that?  Well, if you don't have
a backup, you stare at all the messages and try to guess the missing
content, based on your extensive unique knowledge of your unique email
usage.