[ZODB-Dev] repozo full backup and index files

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tres Seaver wrote:
> Hanno Schlichting wrote:
>> On Fri, May 14, 2010 at 10:24 AM, Christian Theune <ct at gocept.com> wrote:
>>> Hmm. If the full backup is just a regular FS file then you could start
>>> with the naive approach and just open/close it once after performing a
>>> backup as that would create the index file.
>> Sure. That would be an easy but also rather inefficient way. In my
>> case it takes only about 15 minutes of rather heavy I/O to create the
>> file, but for larger files this time usually goes up.
> 
>> >From what I understand repozo does the following:
> 
>> 1. it opens the real file storage file as a read-only FileStorage
>> 2. calculates the byte position of end of the last complete transaction
>> 3. closes the file storage
>> 4. opens the fs file as a normal binary file
>> 5. copies over all bytes up to the calculated position into a temp file
>> 6. closes the fs file and temp file and renames the temp file
>> according to some timestamp
> 
>> What I'm wondering is, if you could copy or otherwise create the index
>> as part of the second step. In step 5 we only deal with bytes and
>> cannot parse those, but in step 2 we have a proper ZODB FileStorage
>> object and can to some degree decide which transaction we want.
> 
> This strategy should be workable for both full and incremental backups:
>  the storage opened in step 1 has an attribute, '_index', which is an
> fsIndex instance.  We should be able to save the index into the
> repository direcotry with a timestamped filename, something like::
> 
>    index_file = os.path.splitext(dest)[0] + '.index'
>    fs._index.save(fs._pas, index_file)
> 
> At restore time, we would look for the index corresponding to the full
> or incremental being restored, and copy it into place followi[ng the
> successful copy of the data.
> 
> As usual, the tedious part about implementing this feature will be
> adding / updating tests.

I would be willing to make a stab at this, if we can hold off on a
3.10.0 beta until I've had a chance to try it.



Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkvtX0gACgkQ+gerLs4ltQ4dWQCgs3Icg+omT7O3xfxCPoIiIyCm
zuEAoKprV5ggwEx9xZHC7bWVX5Zfuogc
=AKk0
-----END PGP SIGNATURE-----