[ZODB-Dev] RE: Memory out of control when *NOT* changing objects

[Tim Peters]

[Malcolm Cleaton]
>>> But, why is this necessary? I thought the only circumstance that would
>>> bring swap death to the ZODB was a super-sized transaction, full of
>>> changed objects.

[Tim Peters]
>> Why would you think that?  In the absence of docs, I'm just curious
>> about where people get their ideas.

[Malcolm]
> The usual poor chain of deduction based on simplifications and
> misconceptions, I guess :)

There's no shame in that, and thanks for responding.  I was mostly worried
that you got bad info from a web page or writeup somewhere.  Most things you
read are wrong <wink>.

> I knew that objects were liable to be deactivated at any time by ZODB
> memory management, from documentation on "_v" volatile attributes of
> persistent classes.

That's theory versus practice:  while it's true that ZODB "reserves the
right" to make _v attributes vanish at any instant, no implementation of
ZODB to date actually does.  It's good that the *intent* is understood
anyway, since future implementations may work differently.  Then again,
since comprehensive current docs don't really exist, we usually back off
from changing behavior that depends on "attractive accidents".

> When I read that large transactions caused memory problems, I assumed
> this was solely because modified objects can't be deactivated, so the
> memory management can't win.

It's true that modified objects won't be deactivated mid-transaction now.
It's possible to deactivate them, using implementation tricks, but it's sure
not recommended.  The idea of "subtransactions" seems to exist mostly to
allow breaking a large transaction into many smaller ones, precisely to
limit memory burden -- modified objects can become deactivated after a
subtransaction commit, even though the containing transaction is still in
progress.

It's also true that objects merely loaded but not changed won't be
deactivated mid-transaction now either.  I predict the latter will change
someday.  ZODB 3.3 would not have given you the symptom you saw under 3.2,
but it still doesn't deactivate anything mid-transaction (3.3 avoids your
problem via a different model for how transactions and connections interact
-- in 3.3 connections "join" transactions, so a transaction knows which
connections have been active in it, and informs all such connections about
transaction boundarieds, whether or not an object from the connection was
modified).

> So I trusted the magic of automated memory management to ghost away
> enough objects for me, and to 'just work', so long as I didn't change and
> not commit a lot of things in one go.

If it's any consolation, in 3.3 you would have gotten away with it.

> I'll go for calling sync after committing the transactions, I think.

That should work in 3.2 (or even 3.1).  It's not documented, though <wink>.
A basic hangup is that cache cleanup is synchronous -- it has to be
explicitly invoked, from time to time.  So the question "when does it
happen?" can only be answered now by staring at the implementation to see
when cache cleanup gets invoked.  It so happens that sync() does invoke it,
and it makes *sense* for sync() to invoke it (so that's unlikely to change),
but AFAICT there's no logical necessity for sync() to invoke it (so it's not
impossible for it to change).

Calling Connection.cacheGC() is marginally safer on those grounds.