[ZODB-Dev] FileStorage database unrecoverable?

[Shane Hathaway]

Greg Czajkowski wrote:
> Hi all, I would appreciate any help.
> 
> my database grew over 2GB on NT causing it to become
> corrupted. I am running ZODB 3.1.1 on NT.
> ZODB.FileStorage.packed_version = 'FS21'
> 
> I am using FileStorage because it packs python
> strings,lists,dict,ints in the least amount of space.
> How can I prevent issues when my database grows larger
> than 2GB with the ZEO client/server dying? Should I
> use DirectoryStorage instead? Does it pack the data
> just as efficiently? 
> 
...
> 
> Loading the storage fails:
> ------------------------------------------
> storages = {'1' :
> FileStorage(r"C:\ZODB_BACKUP\data.fs")}
> Traceback (most recent call last):
>   File "<interactive input>", line 1, in ?
>   File "C:\Python21\ZODB\FileStorage.py", line 272, in
> __init__
>     r = self._restore_index()
>   File "C:\Python21\ZODB\FileStorage.py", line 473, in
> _restore_index
>     tid=self._sane(index, pos)
>   File "C:\Python21\ZODB\FileStorage.py", line 374, in
> _sane
>     if file.tell() < pos:
> IOError: (0, 'Error')

Yikes, it looks like the file is actually 2^31 bytes or larger, making 
it so that even file.tell() fails.  In the short term, if you trim the 
file to 2147483647 bytes (2^31 - 1), you'll probably be able to open it 
or use fsrecover.  In the long term, the file.tell() call probably ought 
to be surrounded by an IOError catcher.  Here's a way to truncate in 
Python (make a backup first ;-) ):

f = open('/path/to/data.fs', 'w+b')
f.truncate(2147483647)
f.close()

Once recovered, you can pack more often, move to a platform that 
supports large files with Python 2.1, or move to Python 2.2, which 
implements large file support on Windows.

Shane