[ZODB-Dev] Analyzing the committed data during one transaction

[Dieter Maurer]

Andreas Jung wrote at 2008-11-17 14:21 +0100:
> ...
>> On Nov 15, 2008, at 10:03 AM, Andreas Jung wrote:
>>> is there a way to analyze the data committed during one transaction?
>>> The current usecase is Plone. A simple change to a document causes a
>>> large transaction (between 30k and 100k even for a one-char change). I
>>> am interested to know how many of this data belongs to the
>>> portal_catalog/index and how many are actually changes to the content
>>> object itself (or subobjects).
> ...
>I am basically looking for "real-time monitoring" solution (hacked into 
>the sources).

Why should this be "real-time"?
What magic should intervene when it sees a large transaction?

I have approached analysis tasks like this in two ways:

  *  Use "fsdump" to dump a readable presentation of parts of
     the storage file and look at the relevant transactions
     (Jim has described a programmatic alternative for this).

  *  Instrument "ZEO.ServerStub.ServerStub" to log relevant information
     about ZEO command parameters.

In both cases, I have loaded interesting objects from the ZODB
and looked into them. Very often, the class and content can tell you what
this object is for. E.g. Indexes and catalog metadata are quite easily
recognizable.

This way, I found the horrible inefficent TextIndexNG2
search (before "StupidStorage") behaviour and the huge transaction sizes
due to oversized catalog metadata.

>Rephrasing my question in a different way: how can I get 
>hold of the "parent" object based a persistent object with a _p_jar
>connection? E.g. when the ZODB commits a subobject (e.g. a BTree or a 
>bucket) of portal_catalog/Indexes/some_index then I would like to get 
>hold of the the related parent "first-class" Zope 2 object (e.g. derived 
>from SimpleItem/Item or something like that).

As Jim pointed out, this is difficult.
But often, class and content can tell you what an object is part of.



-- 
Dieter