[Zope] Re: Zcatalog bloat problem (berkeleydb is a solution?)

Toby Dickenson tdickenson@geminidataloggers.com
Tue, 26 Jun 2001 12:55:59 +0100


On Tue, 26 Jun 2001 06:45:54 -0400, Chris McDonough
<chrism@digicool.com> wrote:

>I can see a potential reason for the problem you explain as "and I
>remind you that as the folder get populated, the size that is added to
>each transaction grows, a folder with one hundred objects adds some
>100K"... It's true that "normal" folders (most ObjectManager-derived
>containers actually) cause database bloat within undoing storages when
>an object is added or removed from it.

What Chris describes would be a prudent change anyway, however I dont
think it is the root of this problem. The tranalyzer output shows the
following line for the Folder. At a length of 363 I guess it is pretty
empty. Even if this object grows to 100k (when adding the 100th item)
it is not the single biggest cause of bloat to the total transaction
size.

(incidentally, it *was* the cause of the bloat problems that led me to
develop this patched tranalyzer)

>> OID: 40817 len 363 [OFS.Folder.Folder]


The following entries I do find interesting. They are all somewhat
larger that I remember seeing before.

Are you indexing *large* properties (or storing large metadata
values)? It may be interesting to see the raw pickle data for these
large objects...... my patched tranalyzer can do that too.

>> OID: 37ac4 len 25537 [BTrees.IIBTree.IISet]
>> OID: 37aca len 13947 [BTrees.IIBTree.IISet]
>> OID: 388ff len 24707 [BTrees.IIBTree.IISet]
>> OID: 37ab8 len 39336 [BTrees.IOBTree.IOBTree]
>> OID: 3c610 len 33864 [BTrees.IOBTree.IOBucket]



Toby Dickenson
tdickenson@geminidataloggers.com