[ZODB-Dev] server stops handling requests - nowhere near 100% CPU or Memory used

[Marius Gedminas]

On Fri, Apr 20, 2012 at 10:06:40AM -0400, Claudiu Saftoiu wrote:
> >
> > Ah, so it seems that, as I leave the server running longer & longer,
> > more & more threads are taken up with a `.recv()` call. I think one of
> > my clients opens requests and does not read them/close them.

Does not write, rather.

> > Eventually
> > all the threads are blocking in that fashion.
> >
> > I will fix my clients. But, is there a server-side fix to this (again,
> > using Repoze.BFG)?
> >
> 
> Additional info: in particular the blocked threads' stack dumps look like
> this:
> Thread 140605868680960:
>   File "/usr/lib/python2.6/threading.py", line 504, in __bootstrap
>     self.__bootstrap_inner()
>   File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
>     self.run()
>   File "/usr/lib/python2.6/threading.py", line 484, in run
>     self.__target(*self.__args, **self.__kwargs)
>   File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
> line 878, in worker_thread_callback
>     runnable()
>   File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
> line 1052, in <lambda>
>     lambda: self.process_request_in_thread(request, client_address))
>   File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
> line 1068, in process_request_in_thread
>     self.finish_request(request, client_address)
>   File "/usr/lib/python2.6/SocketServer.py", line 322, in finish_request
>     self.RequestHandlerClass(request, client_address, self)
>   File "/usr/lib/python2.6/SocketServer.py", line 617, in __init__
>     self.handle()
>   File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
> line 442, in handle
>     BaseHTTPRequestHandler.handle(self)
>   File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
>     self.handle_one_request()
>   File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
> line 437, in handle_one_request
>     self.wsgi_execute()
>   File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
> line 287, in wsgi_execute
>     self.wsgi_start_response)
>   File
> "/home/tsa/env/lib/python2.6/site-packages/repoze/zodbconn/connector.py",
> line 21, in __call__
>     result = self.next_app(environ, start_response)
>   File
> "/home/tsa/env/lib/python2.6/site-packages/repoze/zodbconn/cachecleanup.py",
> line 25, in __call__
>     return self.next_app(environ, start_response)
>   File
> "/home/tsa/env/lib/python2.6/site-packages/repoze/retry/__init__.py", line
> 65, in __call__
>     chunk = original_wsgi_input.read(rest)
>   File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
> line 474, in read
>     data = self.file.read(length)
>   File "/usr/lib/python2.6/socket.py", line 377, in read
>     data = self._sock.recv(left)
> 
> Was my assessment of the situation accurate?

It appears so.  Here's the code of repoze.retry:
https://github.com/repoze/repoze.retry/blob/master/repoze/retry/__init__.py

Line numbers differ somewhat; the original_wsgi_input.read(rest) is on
line 71 on that github page right now.  It does look like this is the
bit that reads the HTTP request from the client.

> > Something to time out the connection after 60 seconds or so if nothing has
> > happened?

You're using paste.httpserver.  It has support for killing hung threads:
http://pythonpaste.org/modules/httpserver.html#paste.httpserver.server_runner
has a list of options.

You could also put nginx in front, to take care of ill-behaved HTTP
clients.

Marius Gedminas
-- 
Never be afraid to tell the world who you are.
                -- Anonymous
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://mail.zope.org/pipermail/zodb-dev/attachments/20120420/f0d5fe3a/attachment.sig>