[ZODB-Dev] Speeding up ZODB (was "redis cache for RelStorage")

[Hanno Schlichting]

On Thu, May 5, 2011 at 10:43 AM, Pedro Ferreira
<jose.pedro.ferreira at cern.ch> wrote:
> Since we are talking about speed, does anyone have any tips on making
> ZODB (in general) faster?

Query fewer objects from the database. Make sure you don't store lots
of tiny persistent objects in the database, I'd aim for storing data
in chunks of 8-32kb or use blobs for larger objects. Remember that
ZODB is a key/value storage for the most part. Model your data
accordingly.

> In our project, the DB is apparently the
> bottleneck, and we are considering implementing a memcache layer in
> order to avoid fetching so often from DB.

Before you do that, you might consider switching to RelStorage, which
already has a memcached caching layer in addition to the connection
caches.

But remember that throwing more caches at the problem isn't a
solution. It's likely the way you store or query the data from the
database that's not optimal.

> However, we were also
> wondering if we could in some way take advantage of different computer
> hardware - since the ZEO server is mostly single-threaded we thought of
> getting a machine with higher clock freq and larger cache rather than a
> commodity 8-core server (which is what we are using now).

The ZEO server needs almost no CPU power, except for garbage
collection and packing. During normal operations the CPU speed should
be irrelevant.

> Any tips on the kind of hardware that performs best with ZODB/ZEO? Are
> there any adjustments that can be done at the OS or even application
> layer that might improve performance?

Faster disks. Whatever you can do to get faster disks will help
performance. But that's general advise that applies to all database
servers. You can also throw more memory at the db server, so the
operating systems disk cache will kick in and you'll actually read
data from memory instead of the disks.

Hanno