[ZODB-Dev] Re: ZODB + ZEO performances problem

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thierry Florac wrote:
> Le mercredi 16 juillet 2008 à 21:22 -0400, Tres Seaver a écrit :
>>> Application setup includes a ZEO server managing two databases (a "main"
>>> one for common data and a "catalog" to store catalog), and two Zope
>>> clients (one as main frontend and another one for uncommon
>>> administration tasks, both with standard threads count of 4).
>>> Database sizes (when packed) are of 280 and 400 MB ; Zope clients caches
>>> are of 300 and 500 MB, for 40000 and 80000 objects.
>>>
>>> All data are stored on two mirrored (via RAID 1) Hitachi SAN with quite
>>> good writing speed.
>> Because you need fast *seek* speed, too, I would recommend trying your
>> app with the ZODB on local disk, first.  In fact, I never recommend
>> putting the main ZODB on a non-local disk, for just this reason.  The
>> usage pattern of FileStorage (one big file, writing only at the end, but
>> with lots of random-access reads throughout) is nearly worst-case for
>> such deployments.
>>
>> I have one client whose setup is similar (Linux, but using a "big iron"
>> SAN), where the first phase of incremental repozo backup (the part
>> checking whether incremental is possible) takes longer by a factor of
>> two than just doing a full backup.  Putting it on local disk both makes
>> writes go faster and makes incremental backup feasible.
> 
> Actually, I've already seen that "big reads/writes" are efficient on the
> SAN (more than 40 MB/s) but small ones (many 'seek' and 'read' of 8 Ko
> as made by ZEO) are not efficient at all ; I'll try to see if putting
> ZEO databases as well as Zope clients caches on local disks (which are
> also mirrored) can be more efficient (a hourly cron job can then copy
> ZEO databases on the SAN for data security, I think...).

Rather than copying the open file, which can potentially lose a
transaction which occurs during the copy, I would use the 'repozo'
script to create backup sets onto the remote drive.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIf0tO+gerLs4ltQ4RAgCQAJsFH019JfoVpqrsMCI0FGHE6aG7iwCeIMi7
rYIjpPBQ36tP3YTk8Yr3B+c=
=CXZK
-----END PGP SIGNATURE-----