ZEO cache best practices, was Re: [ZODB-Dev] ZEO 3.2 (Zope 2.7) ->3.6 (Zope 2.9) upgrading: Much slower startup due to cache file creation

[Tim Peters]

[Paul Winkler]
> Heh. Well, I had a site that under some usage patterns would
> occasionally slow to a crawl with cache flips every few minutes.  That
> was with the old default 20 MB cache size.  I think I left it at
> 500 MB or so and that site's been fine since. But the performance
> demands were pretty low.

Yes, 20MB is _very_ small.  It may have seemed conservatively safe 10
years ago when disks were much smaller, but now it's ludicrously
small.

OTOH, the larger the ZEO cache file, the longer it may take startup or
reconnection cache verification to complete, so there's always some
reason not to do the obvious thing <0.3 wink>.

[on doc/ZEO/trace.txt]
> Thanks, those are very good documents!
>
> Out of curiosity, do you have any guesstimates on how much
> overhead enabling the cache trace can incur?

It's definitely intended that you be able to use ZEO's cache tracing
on a production box.

No particular memory burden (trace records are written directly to a
disk file, not in any way cached in memory (apart from that the OS may
use RAM to buffer disk writes, of course).

The speed burden should be minor:  producing a trace record requires a
trivial amount of computation, and then whatever time it takes to pass
a binary string of a few dozen bytes off to the platform output
routines (note that this is summary info, and trace records have a
fixed small size independent of object sizes).  Given that a ZEO cache
hit has to do hundreds or thousands or ... bytes worth of file I/O
anyway (to read up the object pickle), and a cache hit is the cheapest
thing you can do with ZEO, it's relatively minor additional work.

Depending on cache activity, trace files can grow quickly (megabytes
per hour is common, and I contrived tests that produced hundreds of
megabytes per hour), and that's probably the biggest thing to look out
for.  For example, if the trace file is on a small partition, don't be
suprised if a traced ZEO craps out with an "out of disk space" error
when trying to append a new trace record to the file.