[ZODB-Dev] [ZEO] Dealing with long startup times with out-dated index files/Index snapshots?

[Andreas Jung]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi there,

we had in the past several issues with hour-long outages of our
ZEO servers. The reasons are pretty easy: with have several big
storages (60-80GB) and in case of a cluster failures we were not able
to shutdown the ZEO servers in a clean way. So at startup time the
complete index files had to be recreated which takes pretty long
for our storages.

The question came up if it would be possible to write regular snapshots
of the in-memory index back to disk (e.g. every 15 minutes) and then
to modify
the ZEO startup procedure to optionally start a ZEO server with such a
snapshoted
index file? This may cause some data loss (transactions between the last
snapshot and the crash would be ignored - or perhaps the ZEO server
could take
the snapshot file and read all transactions happened after the
snapshot until
the crash in order to update the index with the completed transactions
after
the snapshot...

When I talk of ZEO here then this related almost everything to
FileStorage
in generell since the zodb index file is a FIleStorage feature.

Just some ideas...would that be doable?

Andreas

- -- 
ZOPYX Limited              \ zopyx group
Charlottenstr. 37/1         \ The full-service network for your
D-72070 Tübingen             \ Python, Zope and Plone projects
www.zopyx.com, info at zopyx.com \ www.zopyxgroup.com
- ------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAktpNlcACgkQCJIWIbr9KYwRaACbBcyT/QTWgMhIHL6Viw56rL7e
v+sAn3sCqUkBaVy6aT6AF03+VJ2rl6ZF
=elE4
-----END PGP SIGNATURE-----

-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 316 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zodb-dev/attachments/20100203/f2245535/attachment.vcf