[ZODB-Dev] Space used by IOBTrees

[Martijn Faassen]

Martijn Faassen wrote:
> Andreas Jung wrote:
> > A XML document with about 26.000 nodes (1.3 MB data) is represented
> > in an application by a nested structure of nodes where the childs
> > are stored in IOBTrees (17.000 nodes out of the 26.000 nodes are leafs).
> > The complete object allocates about 20MB inside the ZODB. analyze.py
> > shows that there are about 9600 IOBTree objects inside the ZODB with an
> > average size of 1850 bytes. The amount of data stored as attributes
> > of a single treenode instance is very small (not more than 100-200 bytes).
> > So why is the pickle of an IOBTree nearly 2KB large instead of several
> > hundred bytes?
> 
> I was going to confirm a problem here, but then I updated my Zope3
> checkout and recompiled the C extensions, and suddenly a lot of
> incredible bloat that I was finding in my tests just evapored. Now my
> XML documents in Forest including some indexes seem to get a bloat factor
> of between 2 and 3 on my initial testing, which is quite tolerable.

I did some more testing. I've now loaded the well known Shakespeare
XML documents as well as the 'religion' XML documents (old testament,
new testament, book of mormon and quran) into a ZODB using Forest and
the space use is quite tolerable. 

In XML form it's about 15 megs. In ZODB form (after a pack) they're
about 27 megs. This includes a number of indexes well (to help Forest's
fast XPath query engine).

Regards,

Martijn