[ZODB-Dev] FileStorage size/density survey

[Guido van Rossum]

> Greg Ward wrote:
> > On 16 January 2003, Chris Withers said:
> > 
> >>No it's not, Jim did a custom BTree thing for this which recently got 
> >>merged back in, IIRC...
> > 
> > It's not there in the 3.1 branch, which is what I'm using.  
> 
> Eeep! Can someone who knows (I'm guessing Jeremy or Jim) let us know
> where this code is and where it's supposed to be?
> 
> > And it *appears* to make things both slower and bigger, which
> > isn't much help.
> 
> It reduced memory usage quite a lot for me back when I was testing
> it as just a patch.
> 
> > Anyways, I just wanted to come up with a third way, to see if a
> > dedicated in-memory data structure could work.  Jury's still out --
> > haven't done enough testing yet.

Look in your FileStorage.py for the method _newIndexes().  If it looks
like this:

    def _newIndexes(self):
        # hook to use something other than builtin dict
        return fsIndex(), {}, {}, {}

you have the code that uses a BTree for the index.  There should also
be a code block starting like this:

        if isinstance(index, DictType) and not self._is_read_only:
            # Convert to fsIndex
            newindex = fsIndex()

somewhere in _restore_index().

This is revision 1.105.2.12 of FileStorage.py.  The branch is
ZODB3-3_1-branch.

--Guido van Rossum (home page: http://www.python.org/~guido/)