[Zope-Checkins] CVS: ZODB/src/ZODB/FileStorage -
tim.one at comcast.net
Tue Mar 16 18:51:37 EST 2004
Update of /cvs-repository/ZODB/src/ZODB/FileStorage
In directory cvs.zope.org:/tmp/cvs-serv30994/src/ZODB/FileStorage
Bugfix for pack(). Approximately forever, we've seen extremely rare
failures in the ZEO FileStorage pack-while-writing tests, where they die
CorruptedError: ... transaction with checkpoint flag set
By coincidence, the same kind of death during pack() was reported on
zodb-dev today(!). The new checkPackLotsWhileWriting test provokes this
on my box reliably (failed each time it was run).
The problem appears to be that pack's idea of where a FileStorage
ends was obtained merely by seeking to the end of the file. But if
a transaction is in progress, there can be an extra "not really there
yet" transaction at the end of the file, and when either buildPackIndex()
or findReachableFromFuture() bumped into a such a thing, the CorruptedError
above got raised. This is always rare, and was exceedingly rare before
because only one pack per test was tried. The new test tries packing
repeatedly while a thread keeps hammering the database, so is much more
likely to provoke the problem.
The fix amounts to passing FileStorage._pos to the pack code, telling
the latter directly where the legit transactions in the file end.
=== ZODB/src/ZODB/FileStorage/FileStorage.py 1.10 => 1.11 ===
--- ZODB/src/ZODB/FileStorage/FileStorage.py:1.10 Tue Mar 16 17:41:22 2004
+++ ZODB/src/ZODB/FileStorage/FileStorage.py Tue Mar 16 18:51:06 2004
@@ -1319,13 +1319,15 @@
raise FileStorageError, 'Already packing'
self._pack_is_in_progress = True
+ current_size = self.getSize()
p = FileStoragePacker(self._file_name, stop,
opos = p.pack()
if opos is None:
=== ZODB/src/ZODB/FileStorage/fspack.py 1.8 => 1.9 ===
--- ZODB/src/ZODB/FileStorage/fspack.py:1.8 Tue Mar 16 16:46:46 2004
+++ ZODB/src/ZODB/FileStorage/fspack.py Tue Mar 16 18:51:06 2004
@@ -411,14 +411,15 @@
# stop is the pack time, as a TimeStamp.
# la and lr are the acquire() and release() methods of the storage's lock.
# cla and clr similarly, for the storage's commit lock.
- def __init__(self, path, stop, la, lr, cla, clr):
+ # current_size is the storage's _pos. All valid data at the start
+ # lives before that offset (there may be a checkpoint transaction in
+ # progress after it).
+ def __init__(self, path, stop, la, lr, cla, clr, current_size):
self._name = path
self._file = open(path, "rb")
self._stop = stop
self.locked = 0
- self._file.seek(0, 2)
- self.file_end = self._file.tell()
+ self.file_end = current_size
self.gc = GC(self._file, self.file_end, self._stop)
More information about the Zope-Checkins