[ZODB-Dev] ZODB Ever-Increasing Memory Usage (even with cache-size-bytes)

[Jim Fulton]

On Mon, May 10, 2010 at 8:20 PM, Ryan Noon <rmnoon at gmail.com> wrote:
> P.S. About the data structures:
> wordset is a freshly unpickled python set from my old sqlite oodb thingy.
> The new docsets I'm keeping are 'L' arrays from the stdlib array module.
>  I'm up for using ZODB's builtin persistent data structures if it makes a
> lot of sense to do so, but it sorta breaks my abstraction a bit and I feel
> like the memory issues I'm having are somewhat independent of the container
> data structures (as I'm having the same issue just with fixed size strings).

This is getting tiresome.  We can't really advise you because we can't
see what data structures you're using and we're wasting too much time
guessing. We wouldn't have to guess and grill you if you showed a
complete demonstration program, or at least one that showed what the
heck your doing.

The program you've showed so far is so incomplete, perhaps we're
missing the obvious.

In your original program, you never actually store anything in the
database. You assign the database root to self.root, but never use
self.root. (The variable self is not defined and we're left to assume
that this disembodied code is part of a method definition.) In your
most recent snippet, you don't show any database access. If you
never actually store anything in the database, then nothing will be
removed from memory.

You're inserting data into wordid_to_docset, but you don't show its
definition and won't tell us what it is.

Jim

--
Jim Fulton