[ZODB-Dev] Re: memory exhaustion problem

[Flavio Coelho]

Calling obj._p_deactivate() didn't help either....

I made a few simple tests:

if I run a empty loop (with just a pass statement), its fine. However If I
something as simple as printing an attribute of the object, there goes my
memory....

thanks,

Flávio

On Jan 17, 2008 1:52 PM, Tres Seaver <tseaver at palladion.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Flavio Coelho wrote:
> > Hi,
> >
> > I wrote a simple script to move data from a ZODB database to sqlite.
> >  My data is in a OOBTree, I chose this because supposedly you can bring
> the
> > buckets to memory one at a time....
> >
> > So what I am doing is basically this: I iterate over my objects and
> write
> > them one-by-one to the other db.
> > for k,u in user_root['userdb'].items(): # I have tried iteritems() here
> too
> > but the results are the same....
> >     # write the data to sqlite
> >
> > My problem is that my database is big and  as the memory consumption
> > increases as the loop progresses until all my memory is exhauted and the
> OS
> > goes into swap. It seems that the objects from previous iterations are
> not
> > been cleared from memory.
> > Am I doing something wrong, or is it impossible to iterate over a ZODB
> > database which is bigger than you memory?
>
> The connection normally only tries to enforce its cache size limit at
> transaction boundaries:  if you are iterating manually in a script, you
> need to free things up manually, e.g. by calling '_p_deactivate' on the
> object when you are done with it.
>
>
> Tres.
> - --
> ===================================================================
> Tres Seaver          +1 540-429-0999          tseaver at palladion.com
> Palladion Software   "Excellence by Design"    http://palladion.com
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iD8DBQFHj3nK+gerLs4ltQ4RAtyIAKDYrXI9Zs8uJhagbudeJG36LdQvaACfe3VU
> AHgNdjbyIUkRgUkjZsizJMw=
> =d42v
> -----END PGP SIGNATURE-----
>



-- 
Flávio Codeço Coelho
----------------------------------------------------------------
"My grandfather once told me that there were two kinds of people: those who
do the work and those who take the credit. He told me to try to be in the
first group; there was much less competition."
Indira Gandhi
====================================
registered Linux user # 386432
get counted at http://counter.li.org
----------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zodb-dev/attachments/20080117/6f190b47/attachment.htm