[ZODB-Dev] memory or cache leak?

[Andrew Dalke]

Toby Dickenson <tdickenson@devmail.geminidataloggers.co.uk>
> The test case Andrew posted used a persistent dictionary.
> That dictionary object is likely to be a non-ghost, since
> it is frequently accessed. As long as it is, all of its
> contained objects will be in the cache too, although with
> most of them as ghosts.

> This doesnt cause a great deal of memory bloat (maybe a few hundred
> bytes per ghost?) but does make the cahce less efficient.

> Using some form of BTree will fix this problem.

True, my test case used a PersistentMapping, but I did try an
OOBTree in my real data set and had the same problems.  Also, one
of the other posts was

Michel Pelletier
] I got the same result.  I tried:
]
]   - commit subtransactions instead of transactions every % 1000.
]
]   - Use BTrees instead of PMs
]
] Memory still grew very fast.

Most importantly, if I force a cacheFullSweep(4) then even with
a PersistentMapping I have no memory problems -- I can do everything
in about 100MB.

That method call is the only difference in the code between
the version that uses 1.5GB of RAM then gets a MemoryError and one
that uses 100MB and finishes.

So I assume the problem I'm seeing is somehow cache related and
independent of PersistentMapping vs. BTree.

                    Andrew
                    dalke@dalkescientific.com