[Zope-dev] How to make Zope fail nicely under high load?

[Romain Slootmaekers]

Bjorn Stabell wrote:

> Hi,
> 
> We've run into an interesting problem when load-testing a Zope site
> behind Apache, a problem that we've also encountered in real life.
> 
> Basically, when the load gets high, Zope has a huge backload of work
> (several minutes of requests), making the average latency for each
> request many minutes.  What are effective ways to do this kind of
> overload management so that the backlog of work doesn't get that big?
> 
> The ideal would be for requests to fail immediately if the backlog of
> work is more than a certain number of requests (or even better,
> estimated time to process).
> 
> 
> Here's what we've tried:
> 
> Naively, we thought we could just set the socket.listen() backlog in
> Apache and Zope to a lower value, but in TCP connect()'s apparently
> don't fail if the server's listen backlog is full; instead requests are
> retried, resulting in a client side managed "listen backlog", also
> giving the same long latency.  (If someone knows this stuff, please
> confirm/deny these allegations against TCP :)
> 
> It appears the way to control it would for Apache or Zope to return "503
> Service Unavailable" when the load is too high, but we haven't found a
> good way to do this; Zope doesn't appear to have any mechanism for it,
> and Apache's ProxyPass doesn't either.  I guess load balancers would,
> but that's a bit overkill since we run the server on one machine.
> 
> 
> Regards,

Naturally,

It would be really nice to have a solution inside zope where you can 
configure stuff like the maxinum number of concurrent sessions, and a 
hook for handling the refusal of a new session.
This so called graceful degradation functionality could also be used for

-) graceful shutdown:
now, a shutdown of a zope server is brutal. For some applications you 
want people to be able to finish their business, while at the same time 
you want that no new users can get into the system.

-) licensing policies:
having web services that are used by at most X people at the same time.

These things have to be handled at the Zope level, since neither the 
apache, nor the networking layer have any clue about things like user 
session, aso.

Ok,
This said.
who writes the proposal? ;)

Romain Slootmaekers.