[ZODB-Dev] New ZODB-related project on svn.zope.org

[Tim Peters]

There's a new "zodbbench" project, at

    svn://svn.zope.org/repos/main/zodbbench/trunk

Those on the zodb-checkins mailing list may already have noticed that
zodbbench checkins show up on that list too.

Sad but true:  we've never had a way to quantify ZODB performance across
releases (or across random changes).  This became especially acute when MVCC
was introduced, because a common hope & belief was that MVCC would
significantly speed many typical ZODB apps.  But, to date, to the best of my
knowledge, that's never been measured.  A primary goal of zodbbench is to
craft a sufficiently realistic and complex benchmark that the effect (if
any) of MVCC can be quantified.

I'm also interested in adding very simple benchmarks:  just about anything
both measurable and maintainable (no, I don't want massive applications
sucking in the Zope du jour -- someone who wants that is free to start a
zopebench project instead) is potentially interesting.

OTOH, benchmark interpretation is almost always full of surprises, and I
don't want to get dragged into endless investigation of platform-dependent
anomalies either.  For example, there's one simple benchmark in the project
right now, in zodbbench/benchmarks/one_minute_commit.py.  That just counts
how many IIBTree commits it can do in one minute, to a FileStorage.  Here
are typical reports from a fast (3.4GHz P5) WinXP Pro SP2 box:

Benchmark one-minute commit, version 0.9
Python version: 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit
(Intel)]
ZODB version: 3.5.1
Did 2665 commits in 60.0 seconds, for 44.41 txn/sec.

Benchmark one-minute commit, version 0.9
Python version: 2.3.5 (#62, Feb  8 2005, 16:23:02) [MSC v.1200 32 bit
(Intel)]
ZODB version: 3.2.10b2
Did 2619 commits in 60.0 seconds, for 43.65 txn/sec.


Running both more than once shows there's no reason to suspect that either
3.5.1 or 3.2.10b2 is significantly faster at this task on this box
(variability across runs is high).  OTOH, here's a typical run from a slower
Linux box:

Python version: 2.4.2 (#1, Sep 30 2005, 11:24:23)
[GCC 3.3.2 20031022 (Red Hat Linux 3.3.2-1)]
ZODB version: 3.5.1
Did 26157 commits in 60.0 seconds, for 435.95 txn/sec.


Oops!  That's about 10x better.  Long-time zodb-dev readers will surely
suspect (as I do <wink>) that what we're _really_ measuring here is the
speed of Python's os.fsync() across operating systems.  And/or disk driver
implementations:  on that same Windows box, if I put the Data.fs file on a
USB2-connected external hard drive instead, the performance zooms:

Benchmark one-minute commit, version 0.9
Python version: 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit
(Intel)]
ZODB version: 3.5.1
Did 22697 commits in 60.0 seconds, for 378.28 txn/sec.


So even very simple benchmarks:

- Can tell you interesting things.

- Can tell you _useful_ things (e.g., if I were running ZODB
  in an intense application on this Windows box, I have reason
  to doubt that its internal hard drive is a good place for the
  Data.fs file).

- Can bury you over your eyeballs in endless mysteries ;-)


If you want to add code to the project, feel free.  It's under the ZPL, so
if you have zope.org commit privileges already (for Zope or ZODB) you're all
set.

There's not much structure in the project yet, and it's all subject to
massive rearrangement.  Briefly, everything is arranged as classes in
packages now:

    zodbbench/trunk
        zodbbench/
            __init__.py
            utils.py
            benchmarks/
                __init__.py
                ... individual benchmark modules and packages ...

I expect that every benchmark will make heavy use of zodbbench.utils.  That
detects the ZODB version in use, and supplies ZODB-independent ways to spell
basic things like "commit the current transaction".  It also hides some
x-platform spelling differences.

If you want to know how much of a performance boost MVCC gives, the answer
has always been 6.  A goal of this project is to attach units to that number
;-)