[Zope] Defining High Load in Heavily DB Driven site (was: [Z ope] zope2.2.0 and what is high load)

[Chris McDonough]

Here's my understanding of how it works:

- The number of threads is decided by NUMBER_OF_THREADS in z2.py or the
command line argument -t[whatever].

- You can give ZServer/Zope a hundred threads if you want to, but if you
want more DB connections to service those threads with, you need to
change the pool_size in DB.py.  

Because Dario's site is hypothetical at the moment, it's probably not a
good idea to start out by changing these parameters.  He should probably
just try it out at default, knowing that he can (carefully) bump these
values up as necessary if he notices problems.  It's sort of pointless
to guess about how to increase speed until you know you need it to be
increased.  Zope can handle this kind of load, it's just a matter of
hitting the sweet spot.  In order to do that, you need to do some
profiling work, unfortunately.

> -----Original Message-----
> From: Paul Everitt [mailto:paul@digicool.com]
> Sent: Friday, August 11, 2000 9:03 AM
> To: Dario Lopez-Kästen
> Cc: zope@zope.org
> Subject: Re: [Zope] Defining High Load in Heavily DB Driven site (was:
> [Zope] zope2.2.0 and what is high load)
> 
> 
> "Dario Lopez-Kästen" wrote:
> > 
> > Hello!
> 
> Hello Dario!
> 
> > I need to find more info on how Zope handles threading with 
> relation to
> > database connections.
> 
> How about:
> 
>   http://www.zope.org/Documentation/Developer/Models/ZODB
> 
> The jargon of threads and connections (and then threads again) can be
> pretty confusing in Zope.
> 
> First, ZServer is the thing in front listening for socket 
> requests.  It
> has a connection pool (select based) which maps into Python threads in
> the Zope application.  (ZServer is part of the Zope process, I'm
> discussing logical partitioning.)
> 
> Each concept of a "thread" in Zope is actually a "database connection"
> to the ZODB.  Threads get a private copy of the transaction 
> space, thus
> greatly simplifying (to the point of removal) the need for programmers
> to think about thread safety.
> 
> If you're talking about the thread concurrency a connection to Oracle,
> then that's a bit different.  I _think_ that ultimately this 
> maps to the
> number of Zope database connections.
> 
> This conversation is probably better in the zope-dev list.
> 
> > I am currently considering and evaluating Zope as one of 
> the options we have
> > to build a really large, completely databasedriven 
> "enterprise scale"
> > web-platform. I am a bit worried about this "maximum of 7 
> threads per db
> 
> This isn't a maximum.  It's just the default.  It's easy to 
> increase the
> number of connections.
> 
> > connection" limitation mentioned previously, and I need to 
> know a) what does
> > this mean in terms of accesibility, b) how does it affect 
> performace, and c)
> > how does Zope work (in detail) with external database connections.
> > 
> > We are going to be using Oracle as our DB backend, all our 
> served data will
> > be database *only* (some of it will be fairly large), and 
> efficient database
> > connections is *crucial* to what we intend to do.
> 
> Since you're not using any persistent objects in Zope (unless you do
> caching of the SQL data, which you probably should), then you 
> don't have
> to worry about process growth from having more database 
> connections for
> Zope.  Jack it up to 25.
> 
> Since much of the time in a Zope hit is taken by rendering to HTML, a
> number like 25 probably means you can process a hundred simultaneous
> Oracle requests.  That works out to be an average of around 
> ten million
> hits per day.  If you want more, increase the number.
> 
> > We are going to have between 10-20k users at most and I 
> expect the maximun
> > of simultaneous connections to be beween 400 - 900 under 
> heavy load; "normal
> > load" will probably oscillate around 25-200 simultaneous 
> connections (these
> > numbers are crude guesses, but take into account increased 
> usage over time
> > as more services are provided in the future).
> > 
> > Bandwith problems are not an issue, nor is computing power.
> > 
> > I would appreciate any pointers to documentation or people to ask.
> 
> Is this an intranet or a public site?
> 
> --Paul
> 
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists - 
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )
>