[Zope-ZEO] ZEO and ZODB timing results

[Greg Ward]

On 28 April 2000, Jim Fulton said:
> > Didn't see anything very
> > simple that would speed things up; your 35% improvement is still
> > impressive!
> 
> I want more! :)

Me too!

After much careful honing of our database test suite, I'm finally
gathering some numbers to compare an in-memory "database", plain ZODB,
and ZEO.  There's a possibility that we'll be able to do some test runs
with Versant, but the validity of the comparison would really only be to
order-of-magnitude, as we'll have to use severely hacked-up code to be
fair to Versant.  I'd also like to use ZODB with an alternate storage,
probably Berkeley DB, and Jim's latest ZEO code with 35% (or better?)
improvement.  But for now, it's just the three basics: memory, ZODB, and
ZEO.

For the Digital Creations guys, here's a brief summary of our problem
domain and testing methodology.  The system we're trying to build is a
"virtual fab" for distributed semiconductor fabrication.  The main
classes in the system represent microfabrication processes in their many
guises:
  * theoretical abstractions ("there exists such a thing as plasma
    etching")
  * concrete offerings ("the fab at UC Berkeley offers plasma
    etching of aluminum at $485 for 12 wafers")
  * customer orders ("Joe Blow wants UC Berkeley to do plasma etching
    of his 15 wafers to a depth of 1.5 microns")

To a first approximation, there are classes for each of these three
concepts, and classes to handle the extra information assicated with
each such object.  There are also collections of processes, and
necessary bureacracy such as users and groups.

For testing databases, we wrote a severely slimmed-down version of the
"virtual fab" that weighs in at 13 main classes and ~850 lines of
Python.  (Of those 850 lines, I'd say that about 250 deal with the
bureaucracy of persistence.  That bureaucratic overhead would probably
drop by 50% if we only cared about supporting *one* persistence
mechanism, but we have generalized it to deal with an "in-memory"
database, ZODB, ZEO, and possibly big commercial databases --
specifically we have toyed with POET and Versant.)

To actually test the databases, we wrote a handful of scripts that do
some (perceived) common tasks:
  * create groups and users
  * create "process capabilities" (that's the second type of fabrication
    process, the "concrete offering" above)
  * query the "process library" (the collection of all process
    capabilities)
  * generate process sequences (a process sequence is a list of process
    steps; a process step is the "customer order" sense above)
  * query process sequences

For timing purposes, I separated the writing tasks into "creation" and
"commit" phases, because obviously commit is non-trivial when committing
several hundred objects to either ZODB or ZEO -- and committing to an
in-memory "database" is always trivial (it's a no-op).

Ultimately, I'd like to run data sets of various sizes: small, medium,
and large.  For now, I've only run a "small" data set, because I don't
have the patience to wait for ZEO to write more than a couple of hundred
objects at a time -- it's just too slow on writing to do serious bulk
writes over a ZEO connection.  Of course, I could automate the testing
some more, but I thought I'd let some results leak out first.  Also, no
fancy statistics are needed to reveal ZEO's performance problem.

So, the "small" data set is:
  * create 20 users and 10 groups
  * create 250 process capabilities (actually converted from our
    existing process library)
  * query those 250 process capabilities in a variety of ways
  * generate 30 process sequences of 10-20 process steps each
  * query those process sequences (list all belonging to a particular
    group)

Without further ado, here are the timings:

task:                   in-memory       ZODB             ZEO
create users/groups     0.04            0.15            4.90
commit users/groups     0.01            0.06            5.12
convert processes       2.20            3.34            3.42
commit processes        0.01            7.86          260.00
query processes         0.66            1.19           20.00
generate sequences      1.20            1.25            2.80
commit sequences        0.00            1.76          207.00
query sequences         0.00            0.00            0.07

Comments:
  * all times are in seconds of elapsed real time.
  * for ZEO; elapsed time  was typically much greater than CPU time
  * ZEO times are a single run; memory and ZODB are averages of 3 runs each
  * for ZEO "query processes", the bulk of the time was the first query
    (fill the cache, presumably)

Both of the queries are brute-force, walk-the-object-graph algorithms:
no indexing.

All tasks for a given trial were run within a single process, ie. I
imported the script and called its 'main()' -- this is necessary to do
any sort of in-memory "database" testing, but it also more closely
reflects the anticipated real-world setup, where we'll have a
long-running server process of some sort sitting behind our web server
doing this stuff all day long.  For the ZODB trial (repeated 3 times), I
blew the database away each time.  (I didn't repeat the ZEO trial,
because I'm just not that patient.)

All trials were run with Python 1.5.2 on Solaris 2.6, using the ZEO code
that Michel supplied to Andrew a few weeks back.  The ZODB trials were
with a FileStorage in /tmp, which on Solaris is blazingly fast (I think
it's essentially a complicated ramdisk).  The ZEO trial was over a 100
Mb Ethernet via TCP; the server used a FileStorage on a regular local
disk (ie. not NFS, but not the speedy /tmp either).

The final size of the .fs file containing all the data is about 1 MB.  I
don't have firm figures on just how many objects, or how many objects of
each class, the database contains.  Jim, do you (or anyone else at DC)
have a script that walks a ZODB and prints that sort of summary
information?  Or will I have to write my own?

Stuff on my immediate agenda:
  * write that walk-the-database-and-get-some-stats script (or steal
    someone else's, if it exists)
  * try ZODB with a Berkeley DB storage
  * get my hands on Jim's latest optimized ZEO code and give that a try
  * see if we can run a few trials with Versant, however badly
    we have to hack up our code to make it work

Hope this helps --

        Greg
-- 
Greg Ward - software developer                gward@mems-exchange.org
MEMS Exchange / CNRI                           voice: +1-703-262-5376
Reston, Virginia, USA                            fax: +1-703-262-5367