ZEO cache best practices, was Re: [ZODB-Dev] ZEO 3.2 (Zope 2.7) ->3.6 (Zope 2.9) upgrading: Much slower startup due to cache file creation

[Tim Peters]

[Tim Peters]
>> That's an example:  the post-MVCC ZEO cache is a single file, and
>> there are no cache flips; flips are unique to the pre-MVCC two-file
>> ZEO cache design.

[Paul Winkler]
> Interesting. Is there a recommended way now to judge whether your
> ZEO cache is "big enough"?

Was there a recommended way before?  If so, it probably sucked too ;-)
 It depends so much on the specifics of your app's "typical behavior",
and how you answer the question "'big enough' for _what_?".  For
example, Martin said he never saw cache flips under ZEO 3.2, and that
says he's probably got unusual (in the statistical sense, relative to
the universe of ZEO users) goals here.  If so, "an answer" that made
him happy probably wouldn't mean much to you.

> I used to grep the logs for cache flips to see how often they were happening.
> (I didn't have a formula for what to do with that information, it was a pretty fuzzy
> process that typically ended up as "make the cache really huge and
> forget about it").

The best approach for any knob is to try different settings with your
actual app running an actual workload, measure whatever it is you're
trying to optimize, and pick the next setting to try accordingly. 
Lather, rinse, repeat.

ZEO supports a way to create a dump file summarizing "interesting"
cache events, and there's a cache simulator program that uses that
file as input to predict how various cache statistics (like overall
hit rate) would change _if_ you had specified a different cache size. 
That goes much faster than actually running the whole application
again, but the reported results are an approximation.  I know several
(but not many) people have tried this post-MVCC, and the few I heard
back from said it was helpful.  You can read about it in ZODB's
doc/ZEO/trace.txt.  cache.txt in the same directory gives a brief
overview of the post-MVCC ZEO cache design.