[Zope] Scalability bombing missions

[Paul Everitt]

Hi Jimmie and Patrick.

Jimmie wrote:
> It was never my intention to imply that throwing $$$dollars at DC was
> required to improve scalability. What I said was based on 
> perception of
> such requirement. I'll admit "currently" Zope does not scale 
> as well or
> as elegantly as I would like. A statement I was making was that "if" I
> (or anyone) "believed" that scalability == $$$dollars, I would rather
> give to DC than Oracle. I believe Larry is doing all right on his own.
> :)

I want a jet fighter too (sniff). :^)

> I find Oracle pricing obscene. For the kind of $dollars it 

Architectures like Oracle with Vignette running atop it are very
capital-intensive.  Something that startup dot coms don't have a lot of.
When you've spent all your budget on licenses, and _then_ you start
writing your application...

> would take to
> license Oracle I would rather see what DC could do with the additional
> employees on beefing up Zope/ZODBs concurrency/scalabilities. 
> There is a
> thread on the WebDB forum about a group of people talking 
> about putting
> together a co-op to buy an Oracle license based on Oracles "unit"
> pricing scheme for a moderately powered single processor Intel machine
> it was going to run about $20,000. You would have to 
> renegotiate if your
> site got busy and you added a processor. Yuck!

Yes, it's a shame, because this model introduces yet another way where
you have to partition your "cloud".  It shouldn't matter what processor
or network gets added to your website "cloud".  Just keep throwing
capacity into the cloud.

Existing scalability architectures have such a schism along the
three-tier boundaries that it takes wizards to implement and operate
them.  Making them, obviously, brittle and prone to problems.

> I believe Zope's scalability issues will be solved over time 
> regardless
> of external $dollars. I believe more $dollars could speed up the
> process, however it will still take some time. Currently ZEO is an
> option but I don't know how well understood it is by the community.

It will certainly take some time.  But obvious things can be solved as a
higher priority.  Since scalability means different things to different
people, though, there isn't One Answer.

> Scalability Now can still be a sticky wicket.
> 
> Future things on Zope's side:
> 1. Moore's law.
> 2. Advances in Python and it's performance.
>    a. Python becomes fully multi-threaded, multi-processorable.
>    b. Stackless Python gets adopted and performance improves.
>    c. Viper 

I don't feel that advances in Python are absolutely required.  Apache
doesn't have a threaded model (before v2), and if Zope could dish out
requests like Apache 1.x, everybody would be awfully happy. :^)

This begs the question: how fast is fast enough?  These stupid PCWeek
tests, that give MTS a higher performance score than WebObjects because
it claims to do 4,000 requests/sec vs. 2,000 requests/sec, are just
ridiculous.  At 150 requests/sec, you're talking about the top one
percent of websites on the Web.

The other worry is that these super-freaky Python advances will require
cooperation from the app developers to keep the train on the tracks.

> 3. Advances in Zope, it's architecture and performance.
>    a. Sam Rushing works miracles on Medusa/Zserver using Stackless
> Python and Continuations.

Again, I'm not so sure this is the big answer.  I don't think there's a
lot of evidence that Medusa/ZServer is the primary bottleneck.

>    b. Zope does process pooling with each process on a different
> processor. ZEO?

Yep, this is ultimately an attractive architecture.  Particularly when
you pour all three tiers (presentation, logic, data) into it, you get a
single facility for scaling and distributing all three.

>    c. ZODB does multiple data.fs's based on user design based on url
> path directories

Right, though I'd encourage you to instead speak in terms of the goal,
rather than the solution.  Is the goal here to solve the 2Gb problem on
Linux/x86, to "limit corruption", or is it to do resource accounting on
a per user basis?  As Jim has pointed out, if your goal is the latter,
you won't get there -- user data can still appear in global things like
catalogs.

Jim's counterproposal, which he claims is easier than doing accounting
across separate Data.fs files, is to finish putting accounting hooks
into the object database.  Whenever a user exceeded their capacity (no
matter where the object), an exception would be raised.  This would
probably require that people be able to do their own packing.

I gotta admit, I'm awfully intrigued by this.  It is the kind of thing
that is _seriously_ compelling vs. other architectures, where you have
to keep on-disk quotas for presentation and logic, and in-RDBMS quotas
for data.

Still, the point is, make sure you speak in terms of goals first.

>         allowing different Zope processes on different processors to
> handle their respective dir.
> 
> It would be nice if some C/Python programming guru/genius would solve
> Python's global threadlock issues where Python would scale on
> multiprocessor machines. This would potentially render scalability a

Doubt it.  There's still so much more, such as data scalability (vs. raw
performance).  As for the latter, we at DC _still_ need to make raw
performance go up.

Personally, I think a very smart caching system (where subtemplates are
pre-rendered in ZServer and know when the objects they depend on have
changed) is a huge win for a significant portion of high-load
situations.

> moot point. The geniuses at AOLserver rewrote the threading of the
> built-in Tcl interpreter so that it would be multi-threaded and work
> properly with their nice multi-threaded webserver. Tcl's standard
> threading I don't think (I could be wrong) is as nice as Python's.

Right.  Does this, though, limit them on the platforms you run on
because of assumptions in the threading model?

> If I had Oracle kind of money to throw around I would like to see
> Python's thread issue resolved where Python becomes fully
> multi-processorable. That would transparently scale Zope and benefit
> other Python user's as well. It would enable Python/Zope to embed well
> and play nice in multi-threaded webservers like AOLserver and others.
> IIRC Apache 2.0 will be multi-threaded. If Python scaled on
> multi-processors even a little guy could put together a dual-celery
> machine a serve a nice site.

Right.  But again, this begs the question: what is a "nice site"?  What
if the goal of a "nice site" could be met w/out the disruption of
rearchitecting Python?

Is a "nice site" 100K hits/day?  A million hits/day?  Ten million
hits/day?

This steps into the next part of the problem.  To date, few in the Zope
community has been willing to publish any numbers for highly-traffic'd
sites, despite pleas even from our board chairman.  Perhaps that means
there aren't really any sites where Zope's limit has been reached?
Probably not.  But why are people so reticent to go on the record with
their hit logs?

But it _does_ mean that, until folks are willing to come forward and
provide numbers and test conditions, that we'll be working without a
map.

> I agree wholeheartedly with you that there should be a concerted
> coordinated effort between DC and the community to improve scalability
> by allowing people to throw hardware at it. That was the primary issue
> causing me to wander the websphere looking at options. I've 
> learned some
> things in the search, but I always looked back at Zope, even tho' my
> wife liked Enhydra's logo, she thinks the otter is cute. 
> Zope's current
> logo is much nicer than the Purple and Gold. :)
> 
> If ZEO currently scales well, then DC needs some testimonials on their
> website.

Right.

> There definitely needs to be some documentation best ways to...
> As you said, there is a consistent recurring thread of is 
> ZServer alone
> or ZServer behind Apache better.
> Should I use a RDBMS or ZODB.

Right.

> Some of us have no legacy issues. From scratch what's the best way to
> design...

Ditto.

> I agree the solving of Performance/Scalability and 
> Documentation issues
> would definitely help convert lookers into believers.

That's a good way to phrase it.

--Paul