ZEO cache best practices, was Re: [ZODB-Dev] ZEO 3.2 (Zope 2.7) ->3.6 (Zope 2.9) upgrading: Much slower startup due to cache file creation

[Tim Peters]

...

[Tim Peters]
>> OTOH, the larger the ZEO cache file, the longer it may take startup or
>> reconnection cache verification to complete, so there's always some
>> reason not to do the obvious thing <0.3 wink>.

[Jim Fulton]
> This is only a problem if:
>
> 1. a persistent cache is used or
>
> 2. the client gets disconnected from the server, without
>     restarting, long enough for the server to commit enough
>     transactions for the server to be able to do enough quick
>     verification.

IME, the most frequent (or maybe just the loudest ;-)) complaint was
neither of those:  it's when multiple ZEO clients all use very large
non-persistent ZEO client caches, and the ZEO _server_ goes down. 
When the ZEO server comes back up, it (of course) has lost all the
in-memory caching done by the previous incarnation of the server to
support quick verification, so all the clients "take forever" to do
full verification, and all the clients appear non-responsive for the
duration.

IIRC, that's the rationale for adding a new "yes, I know I'm
non-persistent, but please stop trying to help me anyway"
never-verify-always-start-over-from-scratch ZEO client cache option. 
Ask Andrew Sawyers to be sure ;-)

> We seem to have a lot of problems with persistent caches for
> some reason, so I tend to recommend against their use.  I'm
> not sure what's going on there.

I believe there are multiple causes, from undetected cache-file
corruption after a non-clean shutdown (there's little redundancy to
_check_ in the pre- or post-MVCC cache designs, although I tried to
add a little more sanity-checking in the post-MVCC cache), to users
switching the database their ZEO client connects to without deleting
the persistent cache file(s) or changing any of the stuff on the
client that gets folded into that file's name.  The key seems to be
that almost all cases of persisent-cache problems I'm aware of "got
fixed by magic" just by deleting the cache file(s) are restarting.

> I still find them useful in situations in which the connection to the server is slow.
>
> the second case should be rare.
>
> I would definately error in the direction of using a larger cache

Me too.