[ZODB-Dev] zodb-3.4.0 leaks ZEO.cache.Entry objects?

[Tim Peters]

[Tim Peters]
>> The refcount on Entry keeps growing.  I suspect, but don't yet know,
>> that this is because FileCache._makeroom()'s should have another line:
>>
>>             if e is not None:
>>                 del self.key2entry[e.key]
>>                 self._evictobj(e, size)

[Chris Bainbridge]
> I tried this and it worked - the Entry objects disappear, along with
> tuple.

Yup, it's the right thing to do.  I'm making new ZODBs today to repair this,
and ...

> The number of lists is still growing though.

ClientCache._evicted() left behind an empty list (of non-current tid ranges)
when the last non-current revision of an object was evicted from cache.
That could be a slow leak, since when an oid is never referenced again, the
empty list nevertheless stuck around forever.  I'll fix that too.  It makes
for a visible but relatively minor improvement in this test driver (which is
unrealistic in its use of a single gigantic BTree accessed uniformly at
random); I expect it's of more benefit in real apps (accessing many objects
in "lumpy" ways).

> With your example I get linear unbounded growth with
> BTrees._OOBTree.OOBucket and list objects.

I do not with lists, not even before the list change above.  No offense, but
the code you're using is too fancy (or not fancy enough ...) to trust, and
it's notoriously difficult to get a sane count of objects in use while
simultaneously building container objects to keep track of them.  It's fun
to debug such pits, but I don't have time for that today.  I used this much
simpler (and self-contained) instrumentation code instead:

    lcount = bcount = 0
    for obj in gc.get_objects():
        if isinstance(obj, list):
            lcount += 1
        elif isinstance(obj, OOBucket):
            bcount += 1
    print lcount, bcount

The number of list objects bounces up and down as time goes on.  The number
of OOBuckets does increase the whole time, but that's expected:  the driver
creates a tree with up to 1000000000 (a billion) distinct keys, and the # of
buckets will keep growing and growing.  This is true even if you use a
FileStorage directly (leaving ZEO out of it entirely).

More insight can be obtained like so:

    lcount = bcount = 0
    bclass = {}
    for obj in gc.get_objects():
        if isinstance(obj, list):
            lcount += 1
        elif isinstance(obj, OOBucket):
            bcount += 1
            bclass[obj._p_state] = bclass.get(obj._p_state, 0) + 1
    print lcount, bcount, bclass

After the program has run long enough to reach the default target size (400)
of the ZODB (memory; "pickle" -- not ZEO!) cache, the output changes
character:

...

343 398 {0: 398}
344 398 {0: 398}
345 398 {0: 398}
343 399 {0: 399}
344 399 {0: 399}
345 399 {0: 399}
346 399 {0: 399}
347 399 {0: 399}
346 400 {0: 399, -1: 1}
346 400 {0: 399, -1: 1}
346 400 {0: 399, -1: 1}
347 400 {0: 399, -1: 1}
349 401 {0: 399, -1: 2}
351 401 {0: 399, -1: 2}
352 401 {0: 399, -1: 2}
354 401 {0: 399, -1: 2}
355 402 {0: 399, -1: 3}
356 402 {0: 399, -1: 3}
357 402 {0: 399, -1: 3}
358 402 {0: 399, -1: 3}
359 402 {0: 399, -1: 3}
360 402 {0: 399, -1: 3}
361 402 {0: 399, -1: 3}
362 402 {0: 399, -1: 3}
363 402 {0: 399, -1: 3}
332 402 {0: 399, -1: 3}
329 403 {0: 399, -1: 4}
330 403 {0: 399, -1: 4}
331 403 {0: 399, -1: 4}
332 403 {0: 399, -1: 4}
334 403 {0: 399, -1: 4}
335 403 {0: 399, -1: 4}
336 403 {0: 399, -1: 4}
337 403 {0: 399, -1: 4}
338 403 {0: 399, -1: 4}
340 403 {0: 399, -1: 4}
342 403 {0: 399, -1: 4}
343 403 {0: 399, -1: 4}
344 403 {0: 399, -1: 4}
346 403 {0: 399, -1: 4}
347 403 {0: 399, -1: 4}
347 403 {0: 399, -1: 4}
348 403 {0: 399, -1: 4}
348 403 {0: 399, -1: 4}
349 403 {0: 399, -1: 4}
350 403 {0: 399, -1: 4}
351 403 {0: 399, -1: 4}
352 403 {0: 399, -1: 4}
353 403 {0: 399, -1: 4}
354 403 {0: 399, -1: 4}
355 403 {0: 399, -1: 4}
356 403 {0: 399, -1: 4}
356 403 {0: 399, -1: 4}
357 403 {0: 399, -1: 4}
357 403 {0: 399, -1: 4}
358 403 {0: 399, -1: 4}
358 403 {0: 399, -1: 4}
359 403 {0: 399, -1: 4}
360 403 {0: 399, -1: 4}
358 404 {0: 399, -1: 5}
359 404 {0: 399, -1: 5}

That is, although the number of OOBuckets keeps increasing, after we get to
near 400 of them, the excess become ghosts (_p_state == -1).  I'm sure it's
not always a good thing that memory-cache gc never purges ghosts, but AFAIK
all versions of ZODB have always worked that way.  Ghosts will go away when
nothing else (outside the memory cache) references them, but it's in the
nature of BTrees and the extreme access pattern in this test driver that
interior BTree nodes stay alive for a very long time, and reference ghost
leaf buckets for the duration.

In short, that the count of OOBuckets in RAM keeps increasing follows from
the way the test driver was coded, and is independent of storage choice; all
but about 400 of them are ghosts at any given time, though (also independent
of storage choice).