[ZODB-Dev] Re: Connection pool makes no sense

[Florent Guillaume]

>>>A little bit of history...
>>>We have zope as an application server for heavy loaded tech process. We
>>>have high peaks of load several times a day and my question is about how
>>>can we can avoid unused connections to remain in memory after peak is
>>>passed?
>>>Before ZODB-3.4.1 connection pool was fixed size of pool_size and that
>>>caused zope to block down while load peaks.
>>>ZODB-3.4.2 that is shipped with Zope-2.8.5 has connection pool that does
>>>not limit the opened connections, but tries to reduce the pool to the
>>>pool_size and this behavior is broken IMO.
>>>
>>>Follow my idea...
>>>After peak load I have many (thousands of connections) that have cached
>>>up different objects including RDB  connections.
> 
> Hundreds... my mistake.
> 
>>Huh are you sure? That would mean you have thousands of threads. Or 
>>hundreds or ZEO clients. Or hundreds of ZODB mountpoints.
>>
>>By itself Zope never uses more than one connection per thread, and the 
>>number of thread is usually small.
>>
>>If you see many RDB connections, then it's a RDB problem and not a ZODB 
>>problem. Something not releasing RDB connections quick enough, or 
>>leaking RDB connections.
> 
> 
> Not agree. Can you answer the question? Does self.all.remove(c) mean
> that we WANT to destroy connection instance?

The self.all.remove(c) in _ConnectionPool attempts to destroy the 
connection. If something else has a reference to it once it's closed, then 
that's a bug, and it shouldn't. It should only keep a weak reference to it 
at most.

> If not then where in ZODB source code i can see connection destruction?
> Clearing cache and calling _v_database_connection.close() method?

Sorry I don't know what a _v_database_connection is, it's not in ZODB or 
transaction code. If it's RDB code I can't help you.

> You've just caught me on "thousands" but gave no comments on deletion of
> connection instances... but this is the clue to the topic.

Even hundreds of ZODB connections is absurd.
Again, with 4 threads you should never get more than 4 Filestorage 
connections plus 4 TemporaryStorage connections.

Florent

-- 
Florent Guillaume, Nuxeo (Paris, France)   CTO, Director of R&D
+33 1 40 33 71 59   http://nuxeo.com   fg at nuxeo.com