[ZODB-Dev] Data.fs size grows non-stop

[Laurence Rowe]

2009/12/9 Pedro Ferreira <jose.pedro.ferreira at cern.ch>:
> Hello,
>> Just zodbbrowser with no prefix:
>>
>>   http://pypi.python.org/pypi/zodbbrowser
>>   https://launchpad.net/zodbbrowser
>>
>> It's a web-app: it can connect to your ZEO server so you can inspect the
>> DB while it's being used.
>>
> We tried this, but we currently get an error related with the general
> security policy for zope.app. Maybe we need to install Zope?
> This would be a very handy tool.
>> I'd suggest dumping the last few transactions with one of the ZODB
>> scripts (fsdump.py perhaps) and seeing what objects get modified.
>>
> That's what we've being doing, and we got some clues. We've modified
> Jim's script in order to find out which OIDs are being rewritten, and
> how much space they are taking, and this is a fragment of it:
>
> OID class_name total_size percent_size n_pickles min_size avg_size max_size
> '\x00\x00\x00\x00%T\x89{' BTrees.OOBTree.OOBucket 17402831841 30% 8683
> 1977885 2004241 2026518
> '\x00\x00\x00\x00%T\x89|' BTrees.OOBTree.OOBucket 14204430890 24% 8683
> 1616904 1635889 1651956
> '\x00\x00\x00\x00\x04dUH' MaKaC.common.indexes.StatusIndex 11955954522
> 20% 28513 418230 419315 420294
> '\x00\x00\x00\x00%\xa0%\x7f' BTrees.OOBTree.OOBucket 3532998238 6% 11238
> 307112 314379 320647
> '\x00\x00\x00\x00%\xa0%\x80' BTrees.OOBTree.OOBucket 2193843302 3% 11238
> 190816 195216 199007
> '\x00\x00\x00\x00\x04\x8e\xb6\x04' BTrees.OOBTree.OOBucket 1728216003 3%
> 1953 880615 884903 887285
> [...]
>
> As you can see, we have an OOBucket occupying more than 2MB (!) per
> write. That's almost 17GB only considering the last 1M transactions of
> the DB (we get ~3M transactions per week). We believe this bucket
> belongs to some OOBTree-based index that we are using, whose values are
> Python lists (maybe that was a bad choice to start with?). In any case,
> how do OOBuckets work? Is it a simple key space segmentation strategy,
> or are the values taken into account as well?
> Our theory is that an OOBTree simply divides the N keys in K buckets,
> and doesn't care about the contents. So, since we are adding very large
> lists as values, the tree remains unbalanced, and since new contents
> will be added to this last bucket, each rewrite will imply the addition
> of ~2MB to the file storage.

BTree buckets have no concept of the size of their contents, they
split when their number of keys reaches a threshold (30 for OOBTrees).

> Will the replacement of these lists with a persistent structure such as
> a PersistentList solve the issue?

The list would then be stored as a separate persistent object, so
changes to the bucket would not rewrite the entire list object. The
downside of this is that your application may become slower as reading
the contents of the index will incur additional object loads. Zope2's
ZCatalog stores index data as tuples in BTrees, but only a small
amount of metadata is stored (so the buckets are maybe 30-60KB). It
sounds like you are storing a large amount of metadata in the index,
or perhaps inadvertently indexing something. I've seen similar
problems caused by binary data ending up in a text index (where a
'word' ended up being several megabytes). Load the object to check the
problem is large values, rather than large keys.

Laurence