[Zope] Zope Zeo Performance

[Dieter Maurer]

FuBuJo wrote at 2008-3-14 13:31 +0000:

You need to be a bit more careful in your description.

For example the diagram "Apache -> Zeo -> Zope(ZODB)" is
very confusing. It is very rare that Apache speaks to Zeo.

The confusion between Zope and Zeo may go straight through your
description such that it is often unclear whether you really
mean Zope when you write Zope and Zeo when you write Zeo.

More below.

> ...
>The traffic is heavy write traffic (I read some of Dieters posts and am testing
>that out as well). Once overall load hits about 100 people or so the Zeo's start
>dying

Here again, you use a wrong word: "dying" would mean that your ZEO
process terminates but below to say that it gets slower.

>- heavy load, slow response, python takes all CPU/Memory.

Which "python"? The "python" executing Zeo? Or the one executing Zope?

> Then when
>traffic is removed from the ZEO instance ... the system remains CPU bound by the
>python process ... and you have to bounce Zope(Zeo instance) and Apache to free
>it.

Which system? The one running ZEO (the ZEO server) or the one running
Zope?


>The ZODB reports heavy Clients waiting ... but doesn't budge on load.
You see this in the ZEO logfile?
Then, it is ZEO which reports the waiting -- not the ZODB.


>So ... anyone have any suggestions.

We are having similar problems -- I call them commit congestions.

As far as we understand it by now, it is a multiple cause problem.
Commit congestions can be caused on the client (=Zope) side and on the
server (=ZEO) side.


A client drastically increases the probability for commit congestions
when he does expensive things while he helds the commit lock, i.e.
during the second phase of the two phase commit protocol.
We have identified three causes:

  *  garbage collections

     During a garbage collection the garbage collector holds
     the GIL and blocks all Python activity.
     We found that a single generation 2 (i.e. full) garbage
     collection can take between 10 and 20 s.
     We had a bad text index implementation
     that caused excessive object creation and thereby lots
     of garbage collections.

     Our measure has been to drop the bad index implementation
     and reconfigure the garbage collector to reduce the
     garbage collection frequency by a factor of 1000

  *  "stat"s in the second commit phase.

     In our system, "stat"s for NFS served files could take up to
     27 s. It is a complete mystery why. Local IO, too, occasionally
     seemed to need excessive time. This, too, is still mysterious.
     We may have some hints: some ranking bugs in a search engine
     could cause millions of IO operations within a short timeframe
     and may have significantly affected the Linux IO behaviour.

  *  invalidation message reception and correspondng client cache updates
     during the second commit phase


Other causes for commit contention come from the (Zeo) server:

  *  "FileStorage.pack" unnecessarily holds the commit lock
     during large periods of the copying phase, drastically
     increasing the probability for commit contentions

  *  during some pack phase (reachability analysis),
     access to the storage file is high volume and erratic.
     This drastically reduces the performance of the storage
     and make commit contentions likely.

  *  other heavy use of the file system can affect the IO performance
     available for storage access and can increase the
     likelyhood for commit contentions.

>I can throw 10 more Apache/Zeo instances as it - but not sure if that's the
>right approach.

It is not. Commit contention is a synchronization problem.
It does not go away but is likely to increase when you scale
your frontends up.

>So I guess here's my questions.
>
>1. Is there a Zeo Client limit you can have when connecting to a Zope(Zeo
>Server) instance?

There is no limit in principle -- but as you can see,
lots of clients can affect performance.

Invalidation message processing poses a load on the server
which grows linearly with the number of clients (each client
must get all invalidations).

Most other Zeo load contributions are more dependent on the actual
number of requested operations (reads, writes, commits)
and less on the number of clients that request these operation
(of cause, more clients can generated more requests).

>2. Are there any special setting to allow for 'many' Zeo
>clients connecting to Zeo server?

Reconfigure the Python garbage collector such that it runs far
less often.

Get rid of components that (unnecessarily) create lots of Python objects.

Check whether you do unnecessary operations during the second commit
phase.

Place your ZODB storage files intelligently in the file system
such that other high volume IO operations do not badly affect
IO on the storage.



-- 
Dieter