[ZODB-Dev] ZODB Benchmarks

[Roché Compaan]

Well I finally realised that ZODB benchmarks are not going to fall from
the sky so compelled by a project that needs to scale to very large
numbers and a general desire to have real numbers I started to write
some benchmarks.

My first goal was to get a baseline and test performance for the most
basic operations like inserts and lookups. The first test tests BTree
performance (OOBTree to be specific) and insert instances of a persitent
class into a BTree. Each instance has a single attribute that is 1K in
size. The test tries out different commit intervals - the first
iteration commits every 10 inserts, the second iteration commits every
100 inserts and the last one commits every 1000 inserts. I don't have
results for the second and third iterations since the first iteration
takes a couple of hours to complete and I'm still waiting for the
results on the second and third iteration.

The results so far is worrying in that performance deteriorates
logarithmically. The test kicks of with a bang at close to 750 inserts
per second, but after 1 million objects the insert rate drops to 260
inserts per second and at 10 million objects the rate is not even 60
inserts per second. Why?

In an attempt to determine if this drop in performance is normal I
created a test with Postgres purely to observe transaction rate and not
to compare it with the ZODB. In Postgres the transaction rate hovers
around 2700 inserts throughout the test. There are periodic drops but I
guess these are times when Postgres flushes to disc. I was hoping to
have a consistent transaction rate in the ZODB too. See the attached
image for the comparison. I also attach csv files of the data collected
by both tests.

During the last Plone conference I started a project called zodbbench
available here:

https://svn.plone.org/svn/collective/collective.zodbbench

The tests are written as unit tests and are run with a testrunner
script. The project uses buildout to make it easy to get going.
Unfortunately installing it with buildout on some systems seems to lead
to weird import errors that I can't explain so I would appreciate it if
somebody with buildout fu can look at it. 

What I would appreciate more though is an explanation of the drop in
performance or alternatively, why the test is insane ;-)

-- 
Roché Compaan
Upfront Systems                   http://www.upfrontsystems.co.za
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zodb-vs-postgres-txn-rate.png
Type: image/png
Size: 39656 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zodb-dev/attachments/20071029/809b74ad/zodb-vs-postgres-txn-rate-0001.png
-------------- next part --------------
A non-text attachment was scrubbed...
Name: postgres.csv
Type: text/csv
Size: 13470 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zodb-dev/attachments/20071029/809b74ad/postgres-0001.bin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zodb.csv
Type: text/csv
Size: 13730 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zodb-dev/attachments/20071029/809b74ad/zodb-0001.bin