[ZODB-Dev] Nested BTrees?

[Toby Dickenson]

On Wednesday 08 January 2003 12:07 pm, Andreas Jung wrote:
> I got some sources where an XML document is mapped to a hierarchy
> of corresponding objects inside the ZODB (up to some hundred or thousan=
d
> objects). The disavantage is clear: it takes very long to create the
> hierarchy.=20

It is not clear to me why this large heirarchy should be unnecessarily sl=
ow to=20
create. Have you profiled where the time is going?

My main application creates similar structures of 100's of nodes. These t=
ricks=20
proved useful to me:

1. use subtransactions to keep memory usage small. Manually deactivate no=
des=20
after commiting them to a subtransaction if you guess they are not worth=20
keeping in the ZODB memory cache.

2. serialize additions to avoid conflicts

3. custom __getstate__

> I am thinking about an efficient way to keep the object
> hierachy inside a single Zope object as nested BTree structure.
> Does the ZODB store the complete datastructure (with nested BTrees)
> when changing a single BTree of the datastructure or does store only th=
e
> changed BTree?

The granularity of a BTree is between 30 and 500 nodes, depending on type=
=2E=20
Specifically, `grep DEFAULT_MAX_BUCKET_SIZE BTrees/*.c`=20

To get a better idea of this behavior you could try running on=20
DirectoryStorage, and check what gets written in each transaction using l=
s,=20
or this tool: http://dirstorage.sourceforge.net/dumpdsf.html

--=20
Toby Dickenson
http://www.geminidataloggers.com/people/tdickenson