[Zope-ZEO] Re: ZEO and ZODB timing results
Jim Fulton
jim@digicool.com
Tue, 02 May 2000 17:55:28 -0400
Greg Ward wrote:
>
> On 28 April 2000, Jim Fulton said:
> > > Didn't see anything very
> > > simple that would speed things up; your 35% improvement is still
> > > impressive!
> >
> > I want more! :)
>
> Me too!
>
> After much careful honing of our database test suite, I'm finally
> gathering some numbers to compare an in-memory "database", plain ZODB,
> and ZEO. There's a possibility that we'll be able to do some test runs
> with Versant, but the validity of the comparison would really only be to
> order-of-magnitude, as we'll have to use severely hacked-up code to be
> fair to Versant. I'd also like to use ZODB with an alternate storage,
> probably Berkeley DB, and Jim's latest ZEO code with 35% (or better?)
Looking more like 65% at this point.
> improvement. But for now, it's just the three basics: memory, ZODB, and
> ZEO.
>
> For the Digital Creations guys, here's a brief summary of our problem
> domain and testing methodology. The system we're trying to build is a
> "virtual fab" for distributed semiconductor fabrication. The main
> classes in the system represent microfabrication processes in their many
> guises:
> * theoretical abstractions ("there exists such a thing as plasma
> etching")
> * concrete offerings ("the fab at UC Berkeley offers plasma
> etching of aluminum at $485 for 12 wafers")
> * customer orders ("Joe Blow wants UC Berkeley to do plasma etching
> of his 15 wafers to a depth of 1.5 microns")
>
> To a first approximation, there are classes for each of these three
> concepts, and classes to handle the extra information assicated with
> each such object. There are also collections of processes, and
> necessary bureacracy such as users and groups.
>
> For testing databases, we wrote a severely slimmed-down version of the
> "virtual fab" that weighs in at 13 main classes and ~850 lines of
> Python. (Of those 850 lines, I'd say that about 250 deal with the
> bureaucracy of persistence. That bureaucratic overhead would probably
> drop by 50% if we only cared about supporting *one* persistence
> mechanism, but we have generalized it to deal with an "in-memory"
> database, ZODB, ZEO, and possibly big commercial databases --
> specifically we have toyed with POET and Versant.)
>
> To actually test the databases, we wrote a handful of scripts that do
> some (perceived) common tasks:
> * create groups and users
> * create "process capabilities" (that's the second type of fabrication
> process, the "concrete offering" above)
> * query the "process library" (the collection of all process
> capabilities)
> * generate process sequences (a process sequence is a list of process
> steps; a process step is the "customer order" sense above)
> * query process sequences
>
> For timing purposes, I separated the writing tasks into "creation" and
> "commit" phases, because obviously commit is non-trivial when committing
> several hundred objects to either ZODB or ZEO -- and committing to an
> in-memory "database" is always trivial (it's a no-op).
>
> Ultimately, I'd like to run data sets of various sizes: small, medium,
> and large. For now, I've only run a "small" data set, because I don't
> have the patience to wait for ZEO to write more than a couple of hundred
> objects at a time -- it's just too slow on writing to do serious bulk
> writes over a ZEO connection. Of course, I could automate the testing
> some more, but I thought I'd let some results leak out first. Also, no
> fancy statistics are needed to reveal ZEO's performance problem.
>
> So, the "small" data set is:
> * create 20 users and 10 groups
> * create 250 process capabilities (actually converted from our
> existing process library)
> * query those 250 process capabilities in a variety of ways
> * generate 30 process sequences of 10-20 process steps each
> * query those process sequences (list all belonging to a particular
> group)
>
> Without further ado, here are the timings:
>
> task: in-memory ZODB ZEO
> create users/groups 0.04 0.15 4.90
There's something wrong here. I can't imagine why ZEO
would be so much slower at this phase, since the storage is
hardly involved.
> commit users/groups 0.01 0.06 5.12
> convert processes 2.20 3.34 3.42
> commit processes 0.01 7.86 260.00
> query processes 0.66 1.19 20.00
> generate sequences 1.20 1.25 2.80
> commit sequences 0.00 1.76 207.00
> query sequences 0.00 0.00 0.07
>
> Comments:
> * all times are in seconds of elapsed real time.
> * for ZEO; elapsed time was typically much greater than CPU time
> * ZEO times are a single run; memory and ZODB are averages of 3 runs each
> * for ZEO "query processes", the bulk of the time was the first query
> (fill the cache, presumably)
This probably unfairly penalizes ZEO. In a realistic usage scenario,
most queries will be fed from the local cache.
> Both of the queries are brute-force, walk-the-object-graph algorithms:
> no indexing.
>
> All tasks for a given trial were run within a single process, ie. I
> imported the script and called its 'main()' -- this is necessary to do
> any sort of in-memory "database" testing, but it also more closely
> reflects the anticipated real-world setup, where we'll have a
> long-running server process of some sort sitting behind our web server
> doing this stuff all day long. For the ZODB trial (repeated 3 times), I
> blew the database away each time. (I didn't repeat the ZEO trial,
> because I'm just not that patient.)
>
> All trials were run with Python 1.5.2 on Solaris 2.6, using the ZEO code
> that Michel supplied to Andrew a few weeks back. The ZODB trials were
> with a FileStorage in /tmp, which on Solaris is blazingly fast (I think
> it's essentially a complicated ramdisk). The ZEO trial was over a 100
> Mb Ethernet via TCP; the server used a FileStorage on a regular local
> disk (ie. not NFS, but not the speedy /tmp either).
Why wouldn't you do the FileStorage and ZEO tests on the same disk?
You certainly seem to have put FileStorage at an unfair advantage.
> The final size of the .fs file containing all the data is about 1 MB.
Wow. These times are really bad. I ran a test that created a 100MB
database on a crappy old 100Mhz mobile non-MMX pentium-based laptop.
My time with the ZEO you have was about 2400 seconds. I'd really like to
get a copy of your test script so I try to understand why your numbers
are so bad. Obviously, I'd like you to try out a new ZEO as soon as it's
available.
> I
> don't have firm figures on just how many objects, or how many objects of
> each class, the database contains. Jim, do you (or anyone else at DC)
> have a script that walks a ZODB and prints that sort of summary
> information?
You can take the len() of a database to get the number of objects.
ZEO currently only gets the len() at startup, but you can ask
the FileStorage and the number of objects would be the same either way.
> Or will I have to write my own?
You'll have to write your own.
> Stuff on my immediate agenda:
> * write that walk-the-database-and-get-some-stats script (or steal
> someone else's, if it exists)
Why do you want this information?
> * try ZODB with a Berkeley DB storage
Cool. What are you hoping to gain?
> * get my hands on Jim's latest optimized ZEO code and give that a try
Yup.
> * see if we can run a few trials with Versant, however badly
> we have to hack up our code to make it work
I'd be interested to see the results of this.
Jim
--
Jim Fulton mailto:jim@digicool.com
Technical Director (888) 344-4332 Python Powered!
Digital Creations http://www.digicool.com http://www.python.org
Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email
address may not be added to any commercial mail list with out my
permission. Violation of my privacy with advertising or SPAM will
result in a suit for a MINIMUM of $500 damages/incident, $1500 for
repeats.