[ZODB-Dev] Re: ZODB Benchmarks

[Shane Hathaway]

Roché Compaan wrote:
> On Mon, 2008-02-25 at 07:36 +0200, Roché Compaan wrote:
>> I'll update my blog post with the final stats and let you know when it
>> is ready.
>>
> 
> I'll have to keep running these tests because the more I run them the
> faster the ZODB becomes ;-) Would you have guessed that the ZODB is
> faster at both insertion and lookups than Postgres?
> 
> http://www.upfrontsystems.co.za/Members/roche/where-im-calling-from/zodb-benchmarks-revisited

For some workloads, ZODB is definitely faster.  However, I think your 
analysis needs to provide more detail:

- How much RAM did ZODB and Postgres consume during the tests?

- How often are you committing a transaction?

- Did you use optimal methods of insertion in Postgres, such as COPY? 
Also note that a standard way to insert a lot of data into a relational 
database is to temporarily drop indexes and re-create them after 
insertion.  Your original test may be more valid than you thought.

- Did you use optimal methods of retrieval in Postgres?  It is 
frequently not necessary to pull the data into the application.  Copying 
to another table could be faster than fetching rows.

- What is the ZODB cache size?  How much does the speed change as you 
change the cache size?

- How much disk space does each database consume when there are 10M objects?

> Lookups are even faster then what I originally reported. Lookup times
> averages at 2 milliseconds (0.002s) on 10 million objects. 

Maybe if you set up a ZODB cache that allows just over 10 million 
objects, the lookup time will drop to microseconds.  You might need a 
lot of RAM to do that, though.

> I think somebody else should run these tests as well and validate the
> methodology behind them, otherwise I'm spreading lies.

You're not far from something interesting.

On a related topic, you might be interested in the RelStorage 
performance charts I just posted.  Don't take them too seriously, but I 
think the charts are useful.

http://shane.willowrise.com/archives/relstorage-10-and-measurements/

Shane