[ZODB-Dev] Re: Connection pool makes no sense

[Dieter Maurer]

Tim Peters wrote at 2005-12-30 14:51 -0500:
>[Dieter Maurer]
>> They did not tell us about their application. But, Zope's database
>> adapters work like this. They use the ZODB cache (and its pool) as an RDB
>> connection pool. Thus, if the ZODB caches are not released, the RDB
>> connections won't.
>
>I believe you, but I'm not sure what to make of it; for example,
>
>1. The OP is the only person who has this problem?
>
>2. Other people have this problem too, and don't know what to do about
>   it either, but never complain?

I expect (but the original poster may step in) that the problem
could occur in Zope only when the number of threads exceeds
the pool size (or additional connections are used in an application
specific way) as otherwise, there are no "dropped" connections.

Because formerly, it made no sense to have more worker threads than
that given by the pool size, this situation is likely to occur
rarely.

> ...
>>> If not, you may have better luck on the zope-db list (which is
>>> devoted to using other databases with Zope):
>
>> The problem is not with the RDB but with the ZODB connections that are
>> magically not garbage collected. He will probably not find help on
>> "zope-db".
>
>That suggestion was based on a guess that #3 (above) is most likely.  Of
>course I don't know, but #1 and #2 seem unlikely on the face of it.  If
>other people using RDB don't have this problem, then zope-db is the right
>place to ask how they manage to avoid it.

If the poster has no evidence that the ZODB connections are definitely kept
but just see that the relational database connections remain open,
the reason might indeed be at a completely different place:

  There are some reports on "zope-db" about DAs leaking relational
  database connections.
  
  The problem is not related to ZODB connection handling.
  Instead, the relational database connection is kept open
  even if the DA object was invalidated (and especially cleared).


We observed such behaviour with Zope 2.7/ZODB 3.2 and "ZPsycopgDA".

   When we used "resetCaches" (which should in principle release
   the old caches (I also added an "minimizeCache" because
   the cyclic gc does not work with Zope 2.7's ExtensionClass objects)),
   a new set of Postgres connections was opened without
   the old ones being closed.

   We worked around this problem by:

     *  avoiding to use "resetCaches"

     *  restarting Zope once a weak to get rid of
        stale Postgres connections

> ...
>OTOH, if that's not
>what's going on here, I'd expect to have heard about this here more than
>once in the last 5 years ;-)

The older ZODB code (ZODB 3.2 and before) was seriously flawed
with respect to cache release handling.

Fortunately, it was very rare that caches need to be released.

I found the flaws because we used temporary connections extensively
and, of course, their caches need to go away with the temporary
connection. I had a hard fight to get rid of the memory leaks
induced by those flaws.


Now (ZODB 3.4 and above) caches might get released more often.
True, the garbage collector now has a chance to collect cycles
including "ExtensionClass" objects, but it is very easy to
defeat the GC -- an object with a "__del__" method is sufficient.

> Perhaps because the OP is unique in allowing
>hundreds (or thousands -- whatever) of Connections to be active
>simultaneously?  Don't know.

He did not say that.

  If his hypothesis is indeed true and some connections exceeding
  the pool size are kept indefinitely, then already slightly
  exceeding the pool size may lead to an unbound number of connections
  provided the "exceedance" occurs frequently enough.


>I suggested before that forcing calls to gc.collect() would give more
>evidence.  If that doesn't help, then it's most likely that the application
>is keeping Connections alive.  Since I'm not familiar with the RDB code, I
>suppose it's even possible that such code uses __del__ methods, and creates
>cycles of its own, that prevent cyclic gc from reclaiming them.  In that
>case, there are serious problems in that code.

There is also another "gc" attribute holding the garbage cycles
not released. The poster may examine them to check whether they
contain indeed ZODB connection objects.

-- 
Dieter