[ZODB-Dev] Speeding up ZODB (was "redis cache for RelStorage")

[Paul Winkler]

On Fri, May 06, 2011 at 12:19:39PM +0200, Pedro Ferreira wrote:
> A quick check with nethogs shows values of network usage oscillating
> between 100 and 200 KB/s. But I guess that if I were loading an
> excessive amount of data, this value would be higher, no?

So throughput isn't the problem.

I don't think you've mentioned what the latency is like between your
ZEO server and clients.   What's the ping time look like?

You mentioned pages that load lots of objects (200 objects each
referring to a "handful" of sub-objects).  If they're not in the
client cache, each of those will require a separate fetch from the ZEO
server.

As an example, let's say your average object size is only 1k, but due
to network latency it takes 5ms per fetch.  Let's say your slow page
loads 1000 objects, and 100 of them are not in the cache. That's only
100k of data, but you've spent 500ms waiting on the network.

If this is indeed the problem, one symptom would be vastly better
performance on "warm" pages when all the needed objects are in the
client cache.  Is this a Zope 2 app? Have you checked the control
panel to see what the cache stats look like?

-- 

Paul Winkler
http://www.slinkp.com