[ZODB-Dev] ZODB Ever-Increasing Memory Usage (even with cache-size-bytes)

[Ryan Noon]

First off, thanks everybody.  I'm implementing and testing the suggestions
now.  When I said ZODB was more complicated than my solution I meant that
the system was abstracting a lot more from me than my old code (because I
wrote it and new exactly how to make the cache enforce its limits!).

> The first thing to understand is that options like cache-size and
> > cache-size bytes are suggestions, not limits. :)  In particular, they
> > are only enforced:
> >
> > - at transaction boundaries,
>

If it's already being called at transaction boundaries how come memory usage
doesn't go back down to the quota after the commit (which is only every 25k
documents?).

With regards to returning memory to the OS, I don't really care if it
reports less, but it really seems like it's overallocating if the OS kills
it on an 8GB machine with a 512mb quota.

Tres:

With your first point:

Yeah, I wrote that late last night and I just realized it's getting
evaluated stupidly on the setdefault call.  I was trying to be cute with
Python dict methods that I hadn't used before.  Stupid me.

With regards to your second point:

I read the loop optimization wiki page over at python.org too many times and
I get itchy whenever there's method lookup inside of a loop. I need to
remember I'm dealing with a database here and IO is gonna be the bottleneck
anyway.

With regards to your third point:

I actually ran into the same change notification problem when I was rolling
my own OODB and I assumed ZODB had done something tricky because my changes
were showing up upon reopening the db even when I'd done the append and not
told ZODB about the change. I'll fix it to make that more explicit...I think
the "magical" effects I'd seen were related to my problem with too much damn
caching.  The "array" type is from the stdlib array module that I'm just
appending my IDs to as longs.  I figured it'd be more compact and would
serialize faster.


Btw, the final commit is outside the loop. (not shown). =)

Cheers,
Ryan

-- 
Ryan Noon
Stanford Computer Science
BS '09, MS '10
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zodb-dev/attachments/20100510/7d00a776/attachment.html