[ZODB-Dev] Re: memory exhaustion problem

[Flavio Coelho]

Some progress!

Apparently the combination of:
            u._p_deactivate()
            transaction.savepoint(True)
            transaction.commit()

helped. Memory  consumption keeps growing but much more slowly (about 1/5 of
the original speed). Please correct me if I am wrong, but I believe that
ideally memory usage should stay constant throughout the loop, shouldn't it?

Moreover, I shouldn't need to commit either, since I am not modifying the
objects...

thanks,

Flávio


On Jan 17, 2008 2:45 PM, Flavio Coelho <fccoelho at gmail.com> wrote:

> Calling obj._p_deactivate() didn't help either....
>
> I made a few simple tests:
>
> if I run a empty loop (with just a pass statement), its fine. However If I
> something as simple as printing an attribute of the object, there goes my
> memory....
>
> thanks,
>
> Flávio
>
>
> On Jan 17, 2008 1:52 PM, Tres Seaver <tseaver at palladion.com> wrote:
>
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Flavio Coelho wrote:
> > > Hi,
> > >
> > > I wrote a simple script to move data from a ZODB database to sqlite.
> > >  My data is in a OOBTree, I chose this because supposedly you can
> > bring the
> > > buckets to memory one at a time....
> > >
> > > So what I am doing is basically this: I iterate over my objects and
> > write
> > > them one-by-one to the other db.
> > > for k,u in user_root['userdb'].items(): # I have tried iteritems()
> > here too
> > > but the results are the same....
> > >     # write the data to sqlite
> > >
> > > My problem is that my database is big and  as the memory consumption
> > > increases as the loop progresses until all my memory is exhauted and
> > the OS
> > > goes into swap. It seems that the objects from previous iterations are
> > not
> > > been cleared from memory.
> > > Am I doing something wrong, or is it impossible to iterate over a ZODB
> > > database which is bigger than you memory?
> >
> > The connection normally only tries to enforce its cache size limit at
> > transaction boundaries:  if you are iterating manually in a script, you
> > need to free things up manually, e.g. by calling '_p_deactivate' on the
> > object when you are done with it.
> >
> >
> > Tres.
> > - --
> > ===================================================================
> > Tres Seaver          +1 540-429-0999           tseaver at palladion.com
> > Palladion Software   "Excellence by Design"    http://palladion.com
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.6 (GNU/Linux)
> > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
> >
> > iD8DBQFHj3nK+gerLs4ltQ4RAtyIAKDYrXI9Zs8uJhagbudeJG36LdQvaACfe3VU
> > AHgNdjbyIUkRgUkjZsizJMw=
> > =d42v
> > -----END PGP SIGNATURE-----
> >
>
>
>
> --
> Flávio Codeço Coelho
> ----------------------------------------------------------------
> "My grandfather once told me that there were two kinds of people: those
> who do the work and those who take the credit. He told me to try to be in
> the first group; there was much less competition."
> Indira Gandhi
> ====================================
> registered Linux user # 386432
> get counted at http://counter.li.org
> ----------------------------------------------------------------
>



-- 
Flávio Codeço Coelho
----------------------------------------------------------------
"My grandfather once told me that there were two kinds of people: those who
do the work and those who take the credit. He told me to try to be in the
first group; there was much less competition."
Indira Gandhi
====================================
registered Linux user # 386432
get counted at http://counter.li.org
----------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zodb-dev/attachments/20080117/1af151c3/attachment-0001.htm