[ZODB-Dev] memory or cache leak?

[Andrew Dalke]

Jeremy Hylton <jeremy@zope.com>:
>    - Commented out the Catalog-related stuff; it's not part of
>      StandaloneZODB and didn't appear to be used by the app.

Agreed.  It's leftovers from the original code I was hacking
down to get a reproducible.

>If I change the arguments to cacheMinimize() and cacheFullSweep() from
>0 to 4, the memory usage seems to be capped at 37MB.  The cache size
>still seems to grow frequently, but it occasionally goes down.

Just *how* did you decide to try out 4?  The docs at

http://www.zope.org/Documentation/Developer/Models/ZODB/ZODB_Architecture=
_DB
_cacheMinimize--PyInt_Doc.html
say

] The age parameter is used to indicate that objects that haven=92t been
] accessed in age seconds should be deactivated.

so I only tried 0 and 1 ... and time.time() in case it was more like
pack and took a floating point timestep in seconds from epoch start.

Looking at the comments in the source code, I see cacheFullSweep does
less work than cacheMinimize.  If I call cacheFullSweep(4) every 100
records through the database then everything runs just fine, and the
cacheSize never gets above about 20,000.

Yippee!

With my real data set I need to do the sweep about every 100 records.
I didn't want to expose that requirement to client code so my
interface always calls cacheFullSweep(4) every 100 calls to
Molecule.__getitem__.  Seems to work, and the overhead of calling
the function when it isn't needed is about 1 millisecond.

[in another email, Jeremy said]
> Perhaps [the code around line 155 of cPickleCache.c] should
> become this --
>
> if (dt < 0)
>     dt =3D 0;
> else {
>     dt /=3D 3;
>     if (dt =3D=3D 0)
>         dt =3D 1;
> }
>
> I can't imagine anyone depends on 1 or 2 being treated as 0.

I completely agree.  'Course, the other solution is to fix the
documentation to describe this.

Thanks everyone for helping me work around this problem.  I
hope you all figure out what's going on.

                    Andrew
                    dalke@dalkescientific.com