[ZODB-Dev] Storage Benchmarks

[Jeremy Hylton]

>>>>> "SU" == sean upton <sean.upton@uniontrib.com> writes:

  SU> On a related note, anybody have any feelings on the stability of
  SU> the fsIndex branch?  I'm really tempted to use this given I have
  SU> a process that consumes several hundred MB on a big transaction
  SU> adding and indexing ~330,000 objects to a BTreeFolder, though
  SU> I'm not sure whether or not the current 22bytes per object in
  SU> the index is that big of a deal (my ZODB has 1,072,082 objects
  SU> between this folder and the Catalog I am using)...  If this buys
  SU> me something (significantly less RAM usage, better performance,
  SU> better Zope/ZSS restart times), I'll check it out of CVS (I'm
  SU> using Zope 2.4.2 at the moment). Thoughts?

I'm convinced it is pretty stable.  I had hoped to get it included in
Zope 2.5, but we decided against that because we had missed the first
beta release.  It will be in StandaloneZODB 1.0 final.

The fsIndex branch is a hack of sorts, but a good one :-).  The
FileStorage doesn't need to have its entire index in memory at one
time.  The fsIndex reduces the size of the index, but still keeps it
all in memory.  A better long-term solution is to keep the index on
disk and cache parts of it in memory; i.e. use a BTree.

I'm confident that fsIndex branch works because it is so simple.  It
replaces the basic FileStorage index with an index based on BTrees.
It implements a simple mapping API that is easy to test.  The only
cause for concern would be the BTrees code itself.  But if the BTrees
code has a problem, the FileStorage index is the least of your worries
:-).

Jeremy