[Zope] zope performance issues

[Chris McDonough]

On Wed, 2004-05-19 at 01:31, Trevor Warren wrote:
> --- Tim_Terlegrd <tim at se.linux.org> wrote:
> > > Note that Python (and therefore Zope) usually
> > cannot use
> > > multiprocessor systems efficiently.
> > > This is because in a multi-threaded Python
> > application
> > > at most one thread can execute Python code -- even
> > on
> > > a multiprocessor system.
> [snip]
>  
>  This is disastrous. What engineering marvel
> necessitated such an architecture. Would luv to be
> educated on this front.

Actually, Tim's statement isn't entirely true.  "Python code" consists
of calls into the interpreter but it also obviously uses a bunch of C
code and OS libraries do a lot of its work (opening files, sockets,
doing regular expressions, sleeping, etc.)  When Python code executes
and it hits a patch of code in which it's safe to "release the global
interpreter lock" (which can happen a lot), the process can execute code
on multiple CPUs.  So even a single Python process does and can execute
code on multiple CPUs.  I think someone stated after a bunch of research
at some point (which I've lost) that running a single Zope process for
some common application on a dual-processor machine made effective use
of about 75% of each processor's resources.

It *would* be far more disastrous if it actually made any measurable
difference to Zope performance in a production environment, given that
ZEO exists and you can use as many processes as you want on a single box
or across as many boxes as you like to serve up Zope content.

>  So this means if i have a Single CPU system, the max
> i can get from Zope multithreaded towards handling 100
> diff threads simultaenously would be really meagre as
> compared to any other application server architectured
> to handle multiple threads simultaenously.

Performance doesn't always equate to being able to spawn and service
some number of threads.  In fact, Zope's underlying web server
architecture doesn't use threads at all.  It's async-based.  Zope itself
uses a number of worker threads to service requests that get queued up
by the async server.  This is indeed for Zope where all the time is
spent, so creating more threads may help.  But other systems, like
Squid, which can handle an ungodly number of requests per second doesn't
need any extra threads to do so.

That distinction is pedantic in this case, because you're right,
increasing the number of threads (up to some point) should increase
*Zope* performance.  I haven't done any structured testing of
performance vs. number of threads, however, so where the "knee" is, I
don't know.  It's also probably very application-dependent.

>  Does this esplain the 30TPS as compared to the 100+
> TPS for the simple short circuit(no sql) tests
> conducted on my SMP system here at the labs???.

Maybe.  I don't know what defines a "transaction" (you'd need to post
the code and the steps you took to get the numbers).  But actually, if
you're following your own rules about testing the system "in situ", you
would probably not connect Zope to any SQL server.  Zope doesn't need a
SQL server to store data.  So the "shortcut" case might actually be the
"real" case if you took the SQL server out of the mix.

That said, there's no denying it: a single Zope process is slow.  I hope
no one here will try to convince you otherwise.  I suspect that if you
compare one Zope process to one Weblogic process, Weblogic will beat the
snot out of it for many tests.  But if you compare the performance of 40
instances of Zope on 20 machines serviced by a ZEO server to whatever
you'd need to do to Weblogic to get it scaled across that much hardware
sharing the same data storage, I suspect Zope would beat the living
daylights out of it from a cost/performance perspective.  Scaling Zope
adequately requires you to believe that hardware is cheap but thinking
is expensive.  If you're unable/unwilling to take that leap of faith,
you're probably better off sticking with a Java appserver.

- C