[Zope-dev] Very severe memory leak

Shane Hathaway shane at zope.com
Mon Aug 25 15:57:50 EDT 2003


Leonardo Rochael Almeida wrote:
> On Sat, 2003-08-23 at 22:18, Shane Hathaway wrote:
>>When you flush the cache, those DateTimes should disappear.  If they 
>>don't, the leak is keeping them.
> 
> They are disappearing. Too bad they return immediately after, as they're
> comming from a very heavy catalog query result.

Hmm, that tends to indicate you really need that much memory.

>>1) The ZODB cache size.  The meaning of this number changed dramatically 
>>in 2.6.  Before 2.6 it was a very vague number.  In 2.6 it's a target 
>>number of objects that Zope actually tries to maintain.  Before 2.6 it 
>>might have made sense to set the ZODB cache size to some arbitrarily 
>>high number like 100,000; in 2.6 you want to start at about 2000 and 
>>adjust from there.  There are tools in 2.6 for helping you adjust the 
>>number.
> 
> We're keeping it at 15,000. Most threads keep around 12k and 13k. One of
> them sometimes exagerates at 100k or so. Hmm, funny... when I look at
> them now, they're all under 5k...

Maybe the individual ZODB objects are excessively large.  A few months 
ago we found a bug in the new zope.org software that made it store file 
uploads as a single chunk.  A lot of those files were over 1 MB in size. 
  This exhausted 1 GB of RAM pretty quickly.  Splitting the files into 
chunks, like Zope usually does automatically, solved the problem.

>>2) The number of ImplicitAcquisitionWrappers present in the system.  I 
>>have found it to be a reliable indicator of whether you have a leak or 
>>not.  Expect this number to stay under 400 or so.  If it grows 
>>gradually, there's a leak.  Watch the refcounts screen.
> 
> 
> It keeps under 200. At least when I'm looking :-)

That's another strong indication that it's not actually leaking.  Or 
perhaps it's a class of leak I have not yet encountered. ;-)

>>4) Don't use 2.6.1.  Use 2.6.2, which has fixes for known leaks.  It is 
>>actually already tagged in CVS as "Zope-2_6_2", and it's what zope.org 
>>is now running.  Various unrelated things prevented a formal release 
>>this week.
> 
> 
> You guys are sure about this? The client is very much against running
> things from CVS because they just don't want to go thru another upgrade
> processes when the "real" 2.6.2 shows up.

Well, 2.6.2b6 has been released (but not announced), and I've been 
promised there will be no more changes before 2.6.2.

>>If all else fails, grep all Python modules for "sys._getframe()" and 
>>"sys.exc_info()".  These are the primary causes of memory leaks in 
>>Python 2.1 and below.  If you're brave, you can just run Zope under 
>>Python 2.2, which fixes those particular leaks AFAIK.
> 
> 
> Do I need to be particularly brave to run 2.6.2 (as tagged in the CVS)
> under Python2.2? is it still an "unsupported" combination? I know it's
> 2.7's job to be 2.2 compliant, but I've seen reports of more and more
> people running Zope 2.6.x under Python 2.2.

2.6 is also intended to be compatible with Python 2.2.  It's just that 
here at ZC we haven't gone through the rigorous analysis we've been 
planning.  Hmm, come to think of it, the community could actually do 
that analysis.  It wouldn't be terribly difficult.  Any volunteers?

>>Finally, there's always hope. :-)  The latest thing I've been doing is 
>>running Zope in a debug build of Python.  A debug build makes a magical 
>>"sys._getobjects()" available,
> 
> Where do you discover these things? :-)

I learned this one from Jeremy Hylton.  He posted some info about it on 
his weblog after he mentioned it.

http://www.python.org/~jeremy/weblog/030410.html

>>allowing you to inspect all live objects 
>>through a remote console.
> 
> Interesting, which one do you use? does Boa have a remote console? or is
> it just plain monitor port?

I rolled my own remote console.  The monitor port is too buggy and Boa 
is overkill for this.

>>Since debug builds aren't much slower than 
>>standard builds, you can even run a debug build in production for short 
>>periods of time.  I've been building a small library of functions for 
>>working in this mode, and if you need them, I'll pass them along.  I'd 
>>have to warn you that they are anything but intuitive in their purpose 
>>and use, though.
> 
> 
> I'll let you know if it gets to that :-)
> 
> For now we'll be stress testing to try and locate the URLs that cause
> the most leakage.

That's an excellent approach.

Shane




More information about the Zope-Dev mailing list