[ZODB-Dev] Storage Benchmarks

[Toby Dickenson]

Ive recently been performing some benchmarks and systems tests comparing 
FileStorage (the usual version and Jim's fsIndex branch) against Berkeley 
Full and Packless.

Full details are at:
http://www.zope.org/Members/htrd/benchmarks/storages

A summary of the highlights:

* Storage choice had little impact on overall system throughput  
Specifically, I didnt see the performance problems that Chris W reported 
(although my databases are somewhat smaller).

* Jim's fsIndex branch to FileStorage was the suprise high-performer. It has 
the lowest memory usage of all storages tested.

* All three BerkeleyStorage implementations have very high memory 
requirements for packing: higher than FileStorage. Im not convinced that 
these current implementations deserve their 'high scalability' reputation.


Im most familiar with BerkeleyStorage Full, where the memory usage issues 
come from using a python dictionary with one element per object in the 
database.... obviously not a scalable strategy.

I am suprised that I had not heard about this limitation before..... Does 
anyone with a 'large' BerkeleyStorage know how much memory it uses when 
packing?