[ZODB-Dev] Preliminary notes from fixing a bad data.fs

[Dieter Maurer]

Tim Peters wrote at 2005-1-7 17:17 -0500:
> ...
>> ConflictError: database conflict error (oid 0x05, serial this txn started
>> with 0x00 1900-01-01 00:00:00.000000, serial currently committed
>> 0x0339fe733cf5ddf7 2001-01-20 21:39:14.287598)
>
>Woo hoo!  Congratulations again -- for the past couple days we've been
>fruitlessly speculating here about how we could possibly get a conflict
>error that claimed the "before" serial was 0.  There seem to be a couple
>reports of that per year, but doesn't look like anyone ever got anywhere
>with it.

If the "Data.fs" is indeed seriously (!) corrupted, a zero
serial might be read from it -- when the serial stored with
an object state happens to be (wrongfully) zero.

I do not think this is probably, though.
And I seem to remember that some FileStorage tools make a consistency check
(the serial stored with the object must equal the transaction id).

> ....
>> The Zope started fine when I deleted Data.fs.index.
>
>It's peculiar,

Indeed.

>in part because an .index file doesn't have any info about
>serial numbers.  It essentially just maps oids to .fs file offsets.  The
>serials are stored in the .fs file.  Maybe, after packing, you got a new .fs
>file but were still using the old .index file?  Then the latter would
>contain now-bogus offsets.  It would be easy to read up any sort of nonsense
>then.  It's going to be hard to tell unless you have a way to reproduce
>this.

Usually, reading the index makes a heuristic check whether
the index matches the storage. I had the impression that
the check is quite reliable -- but a heuristic may err ;-)
However, the probability that the changes to the storage
done by packing compromise the match with the index without
that the heuristics detect it seems very low to me.
Moreover, packing stores a new index file (determined during packing).

> ....
>Speaking of the dangers of silent data loss, here's a bit of the code that
>tries to write an .index file:
>
>        try:
>            try:
>                os.remove(index_name)
>            except OSError:
>                pass
>            os.rename(tmp_name, index_name)
>        except: pass
>
>So if anything at all goes wrong in this part (insufficient permission, bugs
>in the code that leave index_name or tmp_name wrong, ...), you won't hear
>anything about it.

I hit precisely this code when I recently tried to understand
why our index files were not written reliably -- and I added
at least logging for problems (in our Zope sources).

It turned out that the problem has not been there but
in "zdaemon". When "zdaemon" shuts down a service, it
sends it a SIGTERM, then waits for up to "backoff-limit" seconds
(10 s by default) for the service to stop and sends it a
SIGKILL, if it does not. Thus, when writing the index(es) does
not finish within 10 s, the new index is not installed -- without
anyone learning about it...

I have a feature request with solution in the collector
to give "zdaemon" a separate configuration option
"shutdown-timeout" to control shutdown behaviour.
Hopefully, it finds its way into the official Zope sources ;-)

-- 
Dieter