[ZODB-Dev] sticky objects

[Tim Peters]

[Simon Burton]
> Following on from my "memory bomb" quandry, I have been further stress
> testing my computer with ZODB :)
>
> I am needing to scan through a PersistantList of Persistant objects each
> of which has non-persistant (2nd class) data. As the list is built, the
> memory usage is low, but after a scan it pulls the whole DB into memory.

Yes, that much is expected.  If you don't do anything to force otherwise,
persistent objects can become ghostified only at transaction boundaries
(this isn't a promise (let alone a threat <wink>), it's simply an
implementation artifact common across all ZODB releases to date).

> It seems that these Persistant objects become sticky, and no amount of
> commit/cacheMinimize/db.close/_p_invalidate/del will get rid of them.
>
> Below is a simple example of what I am talking about.
>
> Am I doing something stupid ? Is this just too pathological for ZODB ?

Which version of ZODB, which version of Python, which operating system,
which C library, and exactly how are you measuring memory use?  All of those
feed into what you see, although I _suspect_ the platform C malloc/free
implementation has the biggest influence on what you see in this program.

The answers for what I happened to try just now:

    ZODB 3.3a1 (because I'm guessing you used some flavor of 3.3, and
                3.3a1 is the version of that I had instantly at hand)
    Python 2.3.5c1 (just because it was most convenient)
    WinXP Pro (ditto)
    C library inherited from the python.org Windows 2.3.5c1 Python
        (some version of MS's MSVC 6.0 C runtime)
    Task Manager's Mem Usage and VM Size statistics (which happened
        to be roughly equal to each other throughout this program run)

> #!/usr/bin/env python2.3
>
> from ZODB import FileStorage, DB
> from persistent import Persistent

That line made me suspect you're using some flavor of ZODB 3.3.  Some
massive memory leaks were fixed over the long string of 3.3 pre-releases,
especially in 3.3b2, but I don't recall any getting fixed after 3.3 final
was released.

> from persistent.list import PersistentList
>
> class PItem(Persistent):
>   def __init__(self, s):
>     Persistent.__init__(self)
>     self.s = s
>
> def main():
>   storage=FileStorage.FileStorage('test.fs')
>   db = DB(storage, cache_size=400)
>   connection = db.open()
>   root = connection.root()
>
>   data = PersistentList()
>
>   root[0] = data
>
>   x = 'a'*10240
>
>   for i in range(2):
>     for j in xrange(10000):
>       data.append( PItem( x ) )
>     get_transaction().commit()
>     print len(data), "items"

Detail:  because the object x is reused, every PItem instance created here
refers to exactly the same 10KB string in memory.  You lose this property
when reading PItem objects back from the database:  2nd-class objects are
stored and recreated by value, not by reference.  IOW, when you read these
things in again, each PItem object will refer to a distinct 10KB string.
That's where the bulk of the memory gets used in the next section (about
2*10000 strings at about 10000 bytes each ~= 200MB).

>   # Memory = 10 Mb

On my box it was closer to 12MB at this point.

>   raw_input( 'enter to scan' )
>   for item in data:
>     s = item.s
>   # Memory = 200 Mb

About 210MB for me.

>   raw_input( 'enter to invalidate all' )
>   for item in data:
>     item._p_invalidate()
>   data._p_invalidate()
>   connection.cacheMinimize()
>
>   # Memory = 200 Mb

Back to 12MB for me here.   Also ~12MB if all the _p_invalidate calls were
commented out (just leaving cacheMinimize()).  Also ~12MB for the complement
(comment out cacheMinimize(), leave all the _p_invalidate calls).  Stayed
near 210MB only if I commented out all of it.

>   raw_input( 'enter to quit' )
>
> if __name__=='__main__':
>   main()

In my run, it's obvious that neither ZODB nor Python held on to the memory.
If your platform C free() doesn't "give the memory back to the operating 
system" here, I don't know that there's anything you can do about that short
of using a different C or libc.