[Zope-Coders] BTrees memory leak?

[Matthew T. Kromer]

Jeremy Hylton wrote:

>If I run the BTrees test cases in a loop, they appear to leak ~18MB
>per run.  This should be repeatable from a StandaloneZODB build:
>
>[do a checkout]
>cd StandaloneZODB
>python setup.py build
>python test.py -L BTree
>
>On Linux, it reports on memory usage each time through the loop.
>
>Jeremy
>

BTW, I instrumented the BTree code, replacing Py_INCREF and Py_DECREF 
with versions that would log to stderr.  After selecting grepping for 
OOSets (which testrunner is leaking) and running it through a simple 
filter program, it doesn't seem that I see any!  This makes me highly 
suspicious of the test suite itself.


Since ONLY the BTree modules were instrumented, I wouldn't see refcount 
changes caused by modules.

Here's an example log file record:

djinn(177)$ more testout.run2.OOSet
INCREF OOSet @ 081cbca0 to 2 from BTrees/BTreeTemplate.c 340
INCREF OOSet @ 081cc568 to 2 from BTrees/BucketTemplate.c 435
INCREF OOSet @ 081cad50 to 2 from BTrees/BucketTemplate.c 435


and here's the log checker program, which just keeps stacks of INCREFs, 
and tosses the entry when it sees a DECREF going to 0.


djinn(358)$ cat logcheck.py
#!/usr/bin/env python2.1

import xreadlines
import sys
import string


file = open(sys.argv[1])

objs = {}


for line in xreadlines.xreadlines(file):
    (how, what, at, where, to, count, frm, file, lineno) = 
string.split(line)
    if how == "INCREF":
        a = objs.get(where,[])
        a.append(line)
        objs[where] = a
    elif how == "DECREF":
        a = objs.get(where,None)
        if count == '0' and a is not None: del objs[where]

print "INCREFS which did not get DECREF'd to 0:"
for k in objs.keys():
    for l in objs[k]:
        print l