Some progress!<br><br>Apparently the combination of:<br> u._p_deactivate()<br> transaction.savepoint(True)<br> transaction.commit()<br><br>helped. Memory consumption keeps growing but much more slowly (about 1/5 of the original speed). Please correct me if I am wrong, but I believe that ideally memory usage should stay constant throughout the loop, shouldn't it?
<br><br>Moreover, I shouldn't need to commit either, since I am not modifying the objects...<br><br>thanks,<br><br>Flávio<br><br><br><div class="gmail_quote">On Jan 17, 2008 2:45 PM, Flavio Coelho <<a href="mailto:fccoelho@gmail.com">
fccoelho@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Calling obj._p_deactivate() didn't help either....<br>
<br>I made a few simple tests:<br><br>if I run a empty loop (with just a pass statement), its fine. However If I something as simple as printing an attribute of the object, there goes my memory....
<br><br>thanks,<br><br>Flávio<div><div></div><div class="Wj3C7c"><br><br><div class="gmail_quote">On Jan 17, 2008 1:52 PM, Tres Seaver <<a href="mailto:tseaver@palladion.com" target="_blank">tseaver@palladion.com</a>> wrote:
<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
-----BEGIN PGP SIGNED MESSAGE-----<br>Hash: SHA1<br><div><br>Flavio Coelho wrote:<br>> Hi,<br>><br>> I wrote a simple script to move data from a ZODB database to sqlite.<br>> My data is in a OOBTree, I chose this because supposedly you can bring the
<br>> buckets to memory one at a time....<br>><br>> So what I am doing is basically this: I iterate over my objects and write<br>> them one-by-one to the other db.<br>> for k,u in user_root['userdb'].items(): # I have tried iteritems() here too
<br>> but the results are the same....<br>> # write the data to sqlite<br>><br>> My problem is that my database is big and as the memory consumption<br>> increases as the loop progresses until all my memory is exhauted and the OS
<br>> goes into swap. It seems that the objects from previous iterations are not<br>> been cleared from memory.<br>> Am I doing something wrong, or is it impossible to iterate over a ZODB<br>> database which is bigger than you memory?
<br><br></div>The connection normally only tries to enforce its cache size limit at<br>transaction boundaries: if you are iterating manually in a script, you<br>need to free things up manually, e.g. by calling '_p_deactivate' on the
<br>object when you are done with it.<br><br><br>Tres.<br>- --<br>===================================================================<br>Tres Seaver +1 540-429-0999 <a href="mailto:tseaver@palladion.com" target="_blank">
tseaver@palladion.com</a><br>Palladion Software "Excellence by Design" <a href="http://palladion.com" target="_blank">http://palladion.com</a><br>-----BEGIN PGP SIGNATURE-----<br>Version: GnuPG v1.4.6 (GNU/Linux)
<br>Comment: Using GnuPG with Mozilla - <a href="http://enigmail.mozdev.org" target="_blank">http://enigmail.mozdev.org</a><br><br>iD8DBQFHj3nK+gerLs4ltQ4RAtyIAKDYrXI9Zs8uJhagbudeJG36LdQvaACfe3VU<br>AHgNdjbyIUkRgUkjZsizJMw=
<br>=d42v<br>-----END PGP SIGNATURE-----<br></blockquote></div><br><br clear="all"><br></div></div><div><div></div><div class="Wj3C7c">-- <br>Flávio Codeço Coelho<br>----------------------------------------------------------------
<br>"My grandfather once told me that there were two kinds of people: those who do the work and those who take the credit. He told me to try to be in the first group; there was much less competition."
<br>Indira Gandhi<br>====================================<br>registered Linux user # 386432<br>get counted at <a href="http://counter.li.org" target="_blank">http://counter.li.org</a><br>----------------------------------------------------------------
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Flávio Codeço Coelho<br>----------------------------------------------------------------<br>"My grandfather once told me that there were two kinds of people: those who do the work and those who take the credit. He told me to try to be in the first group; there was much less competition."
<br>Indira Gandhi<br>====================================<br>registered Linux user # 386432<br>get counted at <a href="http://counter.li.org">http://counter.li.org</a><br>----------------------------------------------------------------