[ZODB-Dev] RE: [Zope-CMF] Big CMF sites / Storage

[Toby Dickenson]

On Wed, 30 Jan 2002 17:10:42 -0800, sean.upton@uniontrib.com wrote:

>I've posted this to ZODB-dev for further discussion about scaling big
>ZODBs...

I benchmarked some storage options earlier this year. results at
http://www.zope.org/Members/htrd/benchmarks/storages

> a bit of background on my most current project: a CMF site with
>potential for 1million+ objects.
>
>Regarding hardware... I'm trying to forecast what to buy, and this is =
what
>I'm guessing at the moment...  All boxes are likely to be Dual Athlon MP
>boxes (1.2 & 1.56 GHz), with likely to be 1GB on Zope clients and 3GB on=
 the
>ZSS box; the ZSS will be running a RAID10 of 4 10kRPM drives (via Mylex =
170
>16MB cache).

I suspect it is unlikely that your storage server would make good use
of 3GB. At the moment all storages need *alot* of RAM for packing,
this is probably the only time you would need more than 128MB if using
BerkeleyStorage.

(Later this year my BerkeleyStorage will be hitting a RAM-pack
ceiling; so this is unlikely to go unfixed)

>  The ZEO client cache will be run on a software RAID0 of two
>volumes on 7200RPM IDE disks...

How much bandwidth do you have between ZEO servers and clients? If
they are on a LAN then a large ZEO client cache will have more
problems, and few advantages. The cache has to be revalidated at the
start of each connection.

>If I understand correctly, FileStorage might make more sense from a
>performance perspective

=46ileStorage is 'damn fast'. Im currently using BerkeleyStorage which
is thought to be roughly 10x slower. However I have never seen that
make a difference to overall *system* performance (even when looking
carefully for that difference).


The only thing you didnt mention was separating front-end http traffic
from back-end ZEO traffic onto different NICs. ZEO suffers badly if
network latencies become high because of the large number of
round-trips. This may be worth considering if your http traffic is
high enough.



Toby Dickenson
tdickenson@geminidataloggers.com