[ZODB-Dev] zeo client persistent cache files?

[Tim Peters]

[Chris McDonough]
> I'm currently writing an application that puts large "blobs" into ZODB.
> To that end, I'm using the ZODB 3.2 release that is part of Zope 2.7.3.

That's ZODB 3.2.4.

> Each blob is broken into many separate database records to conserve
> client app RAM. Due to the nature of this application, I'm curious about
> the purpose of persistent ZEO client cache files.

My understanding is that your understanding of the purpose is correct.

> I had always assumed the intent of creating a persistent client cache
> file was that if you shut down the program which acts as the ZEO client
> and restarted it, that "less communication" would need to happen on the
> next run of the client program between the client and the server.  The
> cache file could be used to obtain object state as long as individual
> object revisions in the cache matched those on the server.  As a result,
> the only communications between the client and the server would be to
> ensure that the cached data wasn't stale and that the actual data needn't
> go over the wire.
>
> But in some cursory tests, I find that this appears to not be the case.

Sorry, it's impossible to tell from here.  Missing info:

+ How big is your ZEO client cache (measured in bytes).
+ How big is your ZODB pickle cache (measured in # of objects).
+ How big are your ~700 objects in aggregate (measured in bytes).

> If I start the ZEO client with a persistent cache file, and download a
> set of objects from the database using the client (about 700 objects,
> which, when reconstituted, make up a "file"), I get an aggregate transfer
> speed from my app to a disk file of about 5MB/sec.  During this request,
> I see a bunch of zeoLoad requests show up in the ZEO log.  A subsequent
> request for the same set of objects in the same client process lifetime
> gives me an aggregate transfer speed of about 70MB/s. This is due to the
> data being returned from cache: the ZEO log file on "debug" shows no ZEO
> loads during this request, so I know the data is coming from cache under
> this circumstance.  This is good.

There are always two caches in play when using ZEO.  A Connection object has
"a pickle cache", which is an in-memory cache.  By default, it holds 400
objects across transaction boundaries, regardless of how much RAM they
consume.  If you're loading the same 700 objects repeatedly across
transactions, then about 400 of them are gotten from this memory cache.
Note that the pickle cache persists even across closing and then opening a
"new" Connection (Connections are actually cached in a pool when they're
closed, by the DB object, and reused by subsequent DB.open() calls).

The ZEO client cache is a second-level cache, seeing only requests not
satisfied by the pickle cache.  It's a disk-based cache, and defaults to a
maximum size of 20MB, only half of which is fully usable at any time.  The
default size is absurdly small given modern disk capacities.  Since you
didn't say how big your objects are, I can't guess how much disk cache might
be appropriate.

> Then I shut down the client program and immediately restart it.  No other
> processes are accessing the ZEO server, so no objects could have changed
> on the server.  My first request for that same set of objects is gives me
> an aggregate transfer rate of 5MB/s and the ZEO log file shows lots of
> zeoLoad calls which appear to have the same general makeup as the kinds
> of requests that would have come in if the data was not in in the client
> cache.
>
> I would have expected the actual transfer rate to be somewhat higher;
> maybe not the "70MB/s" optimum that appears to be the limit of
> cache-to-app transfer, but somewhere between that and the "worst case"
> 5MB/s of needing to load every object from the server again.

Your first scenario probably satisfied most requests out of memory (the
pickle cache), not from the ZEO disk cache.  I can guess that because the
default pickle cache holds 400 objects, and you said you have 700.  There's
not enough info to guess what to expect from the ZEO cache for the other 300
objects.

> Am I misunderstanding the intent of persistent cache files?  Do I need to
> shut down the client application in a certain way to ensure that the
> cache files can be used on the next client application invocation?
>
> FWIW, "client application" I refer to here is Zope 2.7.3.  Maybe it's
> broken and doesn't shut itself down properly?

I don't know.  Try boosting the size of your ZEO cache and see whether that
helps?  Hell, give it a gigabyte -- that's a great use for disk space, if
you have some to spare.  It's even possible (but I think unlikely) that, if
you're running on a fast local network, it's not any faster to get an object
out of the ZEO client cache than to refetch it over the network.  Either way
ends up reading the object pickle from disk, and also ends up needing to
unpickle it, so the difference is largely network overheads.  The pickle
cache avoids all of disk, unpickling, and network overheads, so to the
extent the pickle cache came into play in your first scenario, it greatly
accelerated object reuse.

It's also possible there's a new bug in ZODB 3.2.4, and it's also possible
there's an old bug in ZODB 3.2.4, and it's also possible there's a new or
old bug in Zope 2.7.3.  I don't believe anything relevant changed in ZODB in
3.2.4, except that the number of bytes ZEO needs to send over the network is
potentially much smaller (up to a factor of 4) under 3.2.4 compared to 3.2.3
or earlier.

A peculiarity with pragmatic importance:  because the ZEO cache is a
second-level cache, it's predictable that it *doesn't* hang on to the most
popular objects:  since all but the first request for a very popular object
is satisfied by the pickle cache, such an object looks *unpopular* to the
ZEO cache (which sees only the first request for it).  So if you've got a
highly skewed access pattern, with relatively few very popular objects, it's
predictable that a persistent ZEO cache won't do you much good.  But I don't
think that applies to the scenario you're sketching here.