[ZODB-Dev] [X-Post] Figure out bottle neck in a repoze.bfg based web app

[steve]

Hi Laurence,

Thanks for your reply and sorry for the delayed response (been fighting fires).

On 01/24/2012 09:18 PM, Laurence Rowe wrote:
> On 24 January 2012 13:50, steve <steve at lonetwin.net> wrote:
>> Hi All,
>>
>> <...snip...>
>> Any pointers/comments would be appreciated.
> 
> (Following up only on zodb-dev as I'm not subscribed to the other lists.)
> 
> I'm guessing, but I suspect your load tests may only be reading from
> the ZODB so you rarely see any cache misses.
> 

Yes, that's correct, we do test (mostly) read operations.

> The most important tuning paramaters for ZODB in respect to memory
> usage are the number of threads and the connection_cache_size. The
> connection_cache_size controls the number of persistent objects kept
> live in the interpreter at a time. It's a per-connection setting and
> as each thread needs its own connection. Memory usage increases
> proportionally to connection_cache_size * number of threads. Most
> people use either one or two threads per process with the ZODB. I know
> plone.recipe.zope2instance defaults to two threads per process, though
> I think this is only to avoid locking up in the case of Plone being
> configured to load an RSS feed from itself.
> 
> The Python Global Interpreter Lock prevents threads from running
> concurrently, so with ZEO running so many threads per process is
> likely to be counter-productive. Try with one or two threads and
> perhaps up the connection_cache_size (though loading from the zeo
> cache is very quick you must ensure your working set fits in the
> connection cache or else you'll be loading the same objects every
> request). If your CPU usage goes down a lot it means your spending
> time waiting for objects to be loaded in the connection and you might
> want to run a couple more processes than you have cpus if you are
> running one thread per process.
> 

Thanks for that information. That is helpful. Yes, I shall try and experiment
with the connection_cache_size.

> I also suggest you check that your mod_wsgi config is correctly
> specifying WSGIProcessGroup, see:
> http://www.martinaspeli.net/articles/update-repoze-under-mod-wsgi-is-not-slow
> and http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIDaemonProcess.
> 

Yes, it is.

> To get more information about what is going on, try porting
> https://github.com/collective/collective.stats to work as WSGI
> middleware instead of hooking into Zope2's ZPublisher.
> 

Hmm, this is interesting too. I'll try to see if I can port and fit it into our app.

Thanks for taking the time.

cheers,
- steve

-- 
random spiel: http://lonetwin.net/
what i'm stumbling into: http://lonetwin.stumbleupon.com/