[ZODB-Dev] Speeding up ZODB (was "redis cache for RelStorage")

[Jim Fulton]

On Fri, May 6, 2011 at 6:19 AM, Pedro Ferreira
<jose.pedro.ferreira at cern.ch> wrote:
> Hello.
>
>> Is your storage server CPU bound?
>
> load average: 1.47, 1.34, 1.20
>
> mpstat:
> """
> 11:40:13 AM  CPU   %user   %nice    %sys %iowait    %irq   %soft %steal
>  %idle    intr/s
> 11:40:13 AM  all    5.82    0.00    0.69    1.11    0.01    0.12    0.00
>  92.25    548.52
> """
>
> I guess it's not very high, for an 8 core machine.

Nope. None of these stats are high.


>> Starting with ZODB 3.10, ZEO
>> storage servers are multi-threaded. They have a thread for each
>> client.  We have a storage server that has run at 120% cpu on a 4-core
>> box.  Also, if you use zc.FileStorage, packing is mostly done in a
>> separate process.
>
> I didn't know this package even existed. I will give it a try, but in
> any case our problems don't seem to be related to packing.

Nope.  I doubt your problems even relate to ZEO. :)

But it illustrates that storage servers can benefit from multiple
processors.

>> A major source of slow down *can* be disk access times. How's IO wait
>> on your server?
>
> As you can see from the mpstat snapshot above, just around 1%. I have
> checked iostat as well, and the number of transactions per second seems
> to be very low considering the maximum allowed by the hardware.
>
>> Look at how your application is using data. If you have requests that
>> have to load a lot of data, maybe you can refactor your application to
>> load fewer.
>
> A quick check with nethogs shows values of network usage oscillating
> between 100 and 200 KB/s. But I guess that if I were loading an
> excessive amount of data, this value would be higher, no?

Right. I'm skeptical that you have a storage problem.

What makes you think you have a storage problem? :)

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton