[ZODB-Dev] BTree memory bomb

[Jim Fulton]

Tim Peters wrote:
> [Jeremy Hylton]
> ...
> 
>>In your example, the keys 10,000,000 through 10,009,999 each store a
>>128KB string as the value.  If a single bucket holds 50 items (I'm making
>>that number up), it will use roughly 40MB of memory.  The program is
>>going to make thousands of buckets that require this much memory or more,
>>so it's not going to be effective at limit memory consumption.
> 
> 
> Concretely, an OOBTree generally holds at most 30 entries per bucket, and
> when entries are inserted in sorted order to begin with (as the test driver
> does) it averages closer to 15 entries per bucket (if your actual app has
> integer keys and values, an IIBTree is a better choice, where the
> entries/bucket max changes from 30 to 120).
> 
> So if this program ran to completion, it would build about
> 
> 
>>>>10000**2/15
> 
> 6666666
> 
> OOBuckets.  If FileStorage is used, that also means we'll have an
> approximately 7-million entry in-memory dict mapping the ~7 million bucket
> object ids to their data records' current offsets in the .fs file. 

This isn't a dict. It is a fsBTree, which should use around 10 bytes per object,
so around 70 megabytes.  That's not a huge amount of memory by today's
standards.

Of course, the serial cache could, in a sample program like this,
use a lot more memory, because it does use a dictionary.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org