[ZODB-Dev] Zope memory usage

[Alan Runyan]

There was a recent modification to limit the ZODB cache to a set size.  i.e.
Limit the size of memory usage to 128MB.

The original feature was implemented here:
  http://svn.zope.org/ZODB/branches/dm-memory_size_limited-cache/

You can get the feature+3.8 branch of the ZODB from:
  http://svn.zope.org/ZODB/branches/zcZODB-3.8/

The changes are also on trunk (will be ZODB 3.9).

The goal of the modification is to try to put upper bounds on the size of
the database cache.  Before this "size limited cache" the cache would grow
and if you had very large objects in the zodb cache -- your memory usage would
be surpringsly high.

Please use this feature in your testing at upfront.  And let Roche know this
feature has landed (if we was not monitoring svn checkins).  The feature needs
to be test driven.  Give it a go and report your experience.

cheers
alan


On Wed, Sep 17, 2008 at 5:10 AM, Izak Burger <izak at upfrontsystems.co.za> wrote:
> Hi all,
>
> I'm sure this question has been asked before, but it drives me nuts so I
> figured I'll ask again. This is a problem that has been bugging me for
> ages. Why does zope memory use never decrease? Okay, I've seen it
> decrease maybe by a couple megabyte, but never by much. It seems the
> general way to run zope is to put in some kind of monitoring, and
> restart it when memory goes out of bounds. In general it always uses
> more and more RAM until the host starts paging to disk. This sort of
> baby-sitting just seems wrong to me.
>
> It doesn't seem to make any difference if you set the cache-size to a
> smaller number of objects or use a different number of threads. Over
> time things always go from good to bad and then on to worse. I have only
> two theories: a memory leak, or an issue with garbage collection (python
> side).
>
> It is possible that this is not a bug, my reasoning goes like this: The
> OS exposes a "virtual memory" picture to the application, ie, the
> application does not know how much of the RAM is real and how much is
> paged out. All it does is call malloc every now and then. Combined with
> a kernel that allows overcommit (which is in general a good thing), I
> see this going only one way.
>
> I'm hoping someone on this list has some insights into the issue.
>
> regards,
> Izak
> _______________________________________________
> For more information about ZODB, see the ZODB Wiki:
> http://www.zope.org/Wikis/ZODB/
>
> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> http://mail.zope.org/mailman/listinfo/zodb-dev
>



-- 
Alan Runyan
Enfold Systems, Inc.
http://www.enfoldsystems.com/
phone: +1.713.942.2377x111
fax: +1.832.201.8856