ZEO cache best practices, was Re: [ZODB-Dev] ZEO 3.2 (Zope 2.7) ->3.6 (Zope 2.9) upgrading: Much slower startup due to cache file creation

[Paul Winkler]

On Tue, Apr 18, 2006 at 02:12:15PM -0400, Tim Peters wrote:
> [Paul Winkler]
> > Interesting. Is there a recommended way now to judge whether your
> > ZEO cache is "big enough"?
> 
> Was there a recommended way before?  If so, it probably sucked too ;-)

Heh. Well, I had a site that under some usage patterns would
occasionally slow to a crawl with cache flips every few minutes.  That
was with the old default 20 MB cache size.  I think I left it at
500 MB or so and that site's been fine since. But the performance
demands were pretty low.

> The best approach for any knob is to try different settings with your
> actual app running an actual workload, measure whatever it is you're
> trying to optimize, and pick the next setting to try accordingly. 
> Lather, rinse, repeat.

Sure. But when you're in a hurry to fix a particular symptom, and
the first thing you try apparently makes it go away permanently,
sometimes that's good enough :-)
 
> ZEO supports a way to create a dump file summarizing "interesting"
> cache events, and there's a cache simulator program that uses that
> file as input to predict how various cache statistics (like overall
> hit rate) would change _if_ you had specified a different cache size. 
> That goes much faster than actually running the whole application
> again, but the reported results are an approximation.  I know several
> (but not many) people have tried this post-MVCC, and the few I heard
> back from said it was helpful.  You can read about it in ZODB's
> doc/ZEO/trace.txt.  cache.txt in the same directory gives a brief
> overview of the post-MVCC ZEO cache design.

Thanks, those are very good documents!

Out of curiosity, do you have any guesstimates on how much
overhead enabling the cache trace can incur?

-- 

Paul Winkler
http://www.slinkp.com