[ZODB-Dev] Speeding up ZODB (was "redis cache for RelStorage")

[Pedro Ferreira]

Hello,

> Query fewer objects from the database. Make sure you don't store lots
> of tiny persistent objects in the database, I'd aim for storing data
> in chunks of 8-32kb or use blobs for larger objects. Remember that
> ZODB is a key/value storage for the most part. Model your data
> accordingly.

That's hard to do for a project that is already 8 or 9 years old, as you
can see in the attached file, we've got have many cases that fall
outside your limits. I've noticed, for instance, that pages that involve
the loading of >200 MaKaC.review.Abstract objects have an awful
performance record (maybe because we then load for each object a handful
of other referenced persitent objects).

> Before you do that, you might consider switching to RelStorage, which
> already has a memcached caching layer in addition to the connection
> caches.

But isn't RelStorage supposed be slower than FileStorage/ZEO?

> But remember that throwing more caches at the problem isn't a
> solution. It's likely the way you store or query the data from the
> database that's not optimal.

I agree, many things could be improved regarding the data structures we
use. However, it is also true that we have a large number of objects
that are rarely changed, and that there is no need to fetch from the DB
if we can keep them in memory.

> The ZEO server needs almost no CPU power, except for garbage
> collection and packing. During normal operations the CPU speed should
> be irrelevant.

Yes, it is consistent with our load average history. It slightly
increases during DB packing, but otherwise stays around 1.

Thanks a lot for your answers.

Cheers,

Pedro

-- 
José Pedro Ferreira

Software Developer, Indico Project
http://indico-software.org

+-----------+
+  '``'--- `+  CERN - European Organization for Nuclear Research
+ |CERN|  / +  1211 Geneve 23, Switzerland
+ ..__. \.  +  IT-UDS-AVC
+  \\___.\  +  Office: 513-R-0042
+      /    +  Tel. +41227677159
+-----------+
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: stats.txt
Url: http://mail.zope.org/pipermail/zodb-dev/attachments/20110506/0e3cbcdd/attachment-0001.txt