[ZODB-Dev] Investigating a Zope reference leak... tracking object creation

[Ben Last (ZODBDev)]

Dear Zopistas

I've been trying to track down what's either a memory or a reference leak in a Zope 2.7.3 (Python 2.3.4) system (two servers,
production one running Zeo, development running Zope direct to ZODB).

The symptoms are that after several days of normal running on the agressively proxy-cached production site or a few hours on the
development site under simulated load, the number of OFS.Image.Image references reaches very high levels, such as 17761.  In the
cache at that time are just 4345 images.  There are only 1332 images on the whole site, so assuming (as I understand it) that each
of the 5 Zope thread has its own cache, there should be a maximum of (1332 * threads) = 6660 Images.  The site never creates new
Images, it just loads existing ones.

In a bid to see where these references are being created, I installed LeakFinder, but that doesn't help me because Images are
persistent objects, and thus their __init__ methods are not called (LeakFinder appears to patch just the __init__ and __del__
methods for tracking).  Thus I've been on a quest to find a place to stick a patch that will show me, for a persistent object, from
where a reference to it is being created.

I've tried __setstate__, and overridden it for OFS.Image.Image to print a traceback to stdout.  Then I run the development Zope site
with bin/runzope and watch.  This shows me Images being created in the expected places, but not in quantities that would explain
those huge refcounts.

So... is there a better place than __setstate__ to identify where references to Images are being created?  I don't fully understand
the way in which persistent objects are actually created and populated.
I'd be very grateful for any help on this!

Regards to all
Ben

Ben Last
Technical Director
SleepyDog