[ZODB-Dev] Re: Can't pack ZODB anymore

[Tim Peters]

[shan sverige]
>> running 'fstest -v' gives only 1 error: time-stamp reduction at
>> 31906881: 0x03537a82059cf1cc <= 0x0355bcc95b5ace33

> I ran fstest (including commenting out the lines Tim suggested) and
> fsrefs and the above error was the ONLY one.

Well, it's the only *kind* of error found.  Because you commented out the
code checking for this error, there may have been any number of later
isnstances of the same thing.

> Is there a way to resolve this ?

Sorry, I really don't know, but I'll suggest something below.  Have you
determined *why* your system clock got set backwards by months?  That's the
real cause, and *may* point to deeper problems with your system than just
your visible ZODB headache.

> even if it means converting to Directorystorage+reiserFS I'd do
> it. I'm completely depending on my Data.fs for all my content.
>
> I saw that in the latest ZODB 3.2.3 from yesterday the cause of the
> problem has been taken away,

No, please be clear on that the cause of the problem was insane behavior
from your system clock.  There's nothing ZODB can do to repair that.  3.2.3
takes extreme steps to try to recover from an insane clock when it detects
one, but it logs a panic-level message about it, which you should take
seriously.

> but that doesn't solve my existing problem does it?

Nope, it doesn't do anything to help your pre-existing problem.

But I noticed that the BaseStorage (a class from which FileStorage derives)
copyTransactionsFrom() method appears to contain code trying to recover from
out-of-order tids.  I have no idea why that code is there (it's a peculiar
thing for this method to worry about!), and, since to the best of my
knowledge it's never been tested in real life, I can't swear it will fix
your problem.  But it looks like the chances are good, especially since
fsrefs had no complaints.

Here after I got into a binary editor and damaged a .fs deliberately,
changing a tid to be out of order:

$ fstest.py Damaged.fs
Damaged.fs time-stamp reduction at 594004:
     0x0356d2f846037288 <= 0x0356d2f846101c66

Now I get into Python and use copyTransactionsFrom() to copy it to a new .fs
file:

$ python
...
>>> import ZODB
>>> from ZODB.FileStorage import FileStorage
>>> bad = FileStorage('Damaged.fs')
>>> good = FileStorage('Undamaged.fs')
>>> good.copyTransactionsFrom(bad, verbose=True)
2004-07-30 23:51:06.375000
0000000000000000  59

... lots of output snipped ...

2004-07-30 23:52:16.421000
0000000000000011  561
Time stamps out of order 2004-07-30 23:52:16.421000,
                         2004-07-30 23:52:16.409406
2004-07-30 23:52:16.421000
0000000000000011  567
Time stamps back in order 2004-07-30 23:52:16.515000
2004-07-30 23:52:16.515000
0000000000000011  573

... more output snipped ...

>>> good.close()


Now I've got a new .fs identical in size to the original one:

$ wc *.fs
    814    6165  603510 Damaged.fs
    814    6165  603510 Undamaged.fs
   1628   12330 1207020 total

I won't show details here, but the content is identical too, except for a
few bytes in the formerly out-of-order tid (which has been changed to be
back in order).

The real difference is that fstest is happy now:

$ fstest.py Undamaged.fs

$

I manually corrupted a tid by a fraction of a second, so the out-of-order
timestamp was easy to repair locally.  Your clock jumped by months, and this
technique may have to change a very large number of transactions to get them
all back in order again.  That may have bad effects of its own -- I can't
guess, and there are no guarantees this will work for you.  I think it's
well worth trying, though.