[ZODB-Dev] Data.fs size grows non-stop

[Kapil Thangavelu]

i would think it would be more useful immediately to account for the rate of
growth rather than trying to account for all objects in the zodb. i've
attached a quick script you can use to get some introspection on the last
few transactions of a large data.fs file. it operates in read only mode,
just pass the Data.fs file as an argument.  For each of the last 20
(default) transactions it will report the objects (by class name) in the
transaction and their sizes aggregated by class. hopefully based on the
classes being written you can identify where the large writes are
happening.

cheers,

kapil



On Mon, Dec 7, 2009 at 3:08 PM, Jose Benito Gonzalez Lopez <
jose.benito.gonzalez at cern.ch> wrote:

> Hi,
>
> Actually, we have two different versions (current and beta) of the
> software,
> working against  the same ZODB. We are just developing new features
> on top of the beta version.
> Fearing that it was due to some changes done by a programmer
> we have disabled the "beta version" but the growing of the DB is still
> there...and since our last email we've got 5GB more (71GB now) .... We
> are a bit worried
> since the disk space is 102GB and  the application is widely used here
> at CERN and
> for outsiders too.
>
> So, up-to-now we are afraid that it is not a problem with any
> modifications in the code....and
> no clue about it.
> We are trying to pack to see the difference with the data.fs file.
>
> Jim, I am still running your script "class_stats.py".....and it seems it
> is going to take a while yet....
>
> Thank you so much for your help. More ideas appreciated!
>
> Best regards,
> Jose
>
>
>
>
> Alan Runyan wrote:
> >> I'd just like to add that there's some changes that can be related to
> this:
> >>
> >> - we had some classes inheriting from Persistent that now inherit from
> >> something else as well (but no extra arguments are being added, AFAIK);
> >> - we added some zope.interface definitions to some Persistent classes;
> >>
> >> maybe this causes some kind of behavior that we were not aware of?
> >>
> >
> > I doubt it.  Thousnads of people are doing this and do not report the
> same
> > behavior.
> >
> > What is more likely is that a programmer is changing a persistent object
> > very often.  One of the downsides of the ZODB is that it is so
> transparent
> > it is possible to unwillingly make database changes.
> >
> > A design pattern for RDBMS is to have 2 pools. READ pool and WRITE pool.
> > Often the READ pool comes from some replica and WRITE is to the master.
> > I'm unsure this pattern would work for ZODB.  I know Malthe was thinking
> > about this but unsure if he had anything concrete.
> >
> > cheers
> > alan
> >
>
> _______________________________________________
> For more information about ZODB, see the ZODB Wiki:
> http://www.zope.org/Wikis/ZODB/
>
> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> https://mail.zope.org/mailman/listinfo/zodb-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zodb-dev/attachments/20091207/531fc1e4/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fstail2.py
Type: application/octet-stream
Size: 2919 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zodb-dev/attachments/20091207/531fc1e4/attachment.obj