[ZODB-Dev] Iterating over large ZODB

[Jim Fulton]

Chris Withers wrote:
> Hi All,
> 
> I want to do something like the following in an external method:
> 
> stack = [container.getPhysicalRoot()]
> 
> while True:
>    object = stack.pop(0)
>    print object.getPhysicalPath()
>    print object.meta_type
>    if object._isPrinicipiaFolderish:
>       stack.extend(object.objectValues())
>    if not stack:
>        break
> 
> ...but I'm worried that will drag all the objects in the ZODB into memory.
> 
> Now, given the ZODB in question is Zope.org, that's something i'd like 
> not to do ;-)
> 
> How can I dump objects out of memory every 1000 or so times round the loop?

Look at the strategy used by the Zope find facility.

Before using an object, it checks whether it was active before use.
If it wasn't, it deactivates it when it's done with it.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org