[ZODB-Dev] repozo full backup and index files

Tres Seaver tseaver at palladion.com
Fri May 14 09:59:34 EDT 2010


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hanno Schlichting wrote:
> On Fri, May 14, 2010 at 10:24 AM, Christian Theune <ct at gocept.com> wrote:
>> Hmm. If the full backup is just a regular FS file then you could start
>> with the naive approach and just open/close it once after performing a
>> backup as that would create the index file.
> 
> Sure. That would be an easy but also rather inefficient way. In my
> case it takes only about 15 minutes of rather heavy I/O to create the
> file, but for larger files this time usually goes up.
> 
>>From what I understand repozo does the following:
> 
> 1. it opens the real file storage file as a read-only FileStorage
> 2. calculates the byte position of end of the last complete transaction
> 3. closes the file storage
> 4. opens the fs file as a normal binary file
> 5. copies over all bytes up to the calculated position into a temp file
> 6. closes the fs file and temp file and renames the temp file
> according to some timestamp
> 
> What I'm wondering is, if you could copy or otherwise create the index
> as part of the second step. In step 5 we only deal with bytes and
> cannot parse those, but in step 2 we have a proper ZODB FileStorage
> object and can to some degree decide which transaction we want.

This strategy should be workable for both full and incremental backups:
 the storage opened in step 1 has an attribute, '_index', which is an
fsIndex instance.  We should be able to save the index into the
repository direcotry with a timestamped filename, something like::

   index_file = os.path.splitext(dest)[0] + '.index'
   fs._index.save(fs._pas, index_file)

At restore time, we would look for the index corresponding to the full
or incremental being restored, and copy it into place followi[ng the
successful copy of the data.

As usual, the tedious part about implementing this feature will be
adding / updating tests.



Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkvtV0EACgkQ+gerLs4ltQ4JFgCfYZ3PV4pZCP7NInG1lW36F0ke
sRMAoL24vUmOIpb2W2XOps7I9TuxmGRk
=aMi4
-----END PGP SIGNATURE-----



More information about the ZODB-Dev mailing list