[Zope] Large Portal requirements (was: Other web servers?)

[Michel Pelletier]

> -----Original Message-----
> From: Jimmie Houchin [mailto:jhouchin@texoma.net]
> Sent: Monday, June 14, 1999 2:41 PM
> To: zope@zope.org
> Subject: [Zope] Large Portal requirements (was: Other web servers?)
> 
> 
nto.
> 
> Howabout for database intensive situations like one 
> encounters in some of
> the larger sites like Altavista, Goto, Yahoo, Amazon, etc.
> 
> Here are very, very large databases, probably very large indexes,
> multi-millions of objects, multi-multi-gigabytes to search, index.
> My concern is because Python does not scale with multiple processor
> machines, this limits ones ability to throw hardware at the 
> situation since
> you are limited to the fastest single processor.
> 

In the future, we will be have the client server model of ZODB3 (which
will be a commercial product) that lets you have multiple Zope frontend
processes (possibly on different machines) accessing multiple database
backend processes (possibly on different machines).  Also, ZServer is
currently multi-threaded, but there is no theoretical limitation to
making it multi-process, in which each Zope instance will have it's own
Python and thus it's own global Python lock, thus giving the OS the
ability to schedule multiple processes among multiple processors, a
little heavier than threads on multi-processor, but *vastly* easier to
impliment and far more stable.

> Do I misunderstand anything.
> 
> Would Zope be appropriate for these situations? How well 
> would Zope handle it?
> 
> For example in an article I read concerning the delay in 
> Intel bringing out
> it's chipset for 8way Pentium III servers. Barnes and Noble 
> was talking
> about beta-testing an 8way server to speed up their searches.

'Searching' is a whole nother question.  If n Zope *processes* are all
doing a search through the index at the same time and you have n
processors, then each process will complete the operation without
blocking each other, if multiple threads do it, then they will block
each other.

However, multiple processors will not speed up the time it takes to do
*one* search, it will just allow multiple searches to be done
concurently (which in high load situations makes each individual search
look much faster).  Getting into the ability break a searching algorithm
down into multiple processors is complex and probably only pays off when
your talking about searching through *massive* quantities of data.  I
mean big.  If you had a Zope index with one million entries in it, you
would, worst case, have to go through 20 comparison operations to find
the element your looking for (Zope indexing uses a very efficient binary
tree structure to store it's indexes).

> 
> Now I understand that different development models yield 
> different results
> and I don't know how Barnes and Noble's website is built. I 
> do know that
> B&N has a very large database with multi-millions of objects 
> with many,
> many gigs of data.
> 
> Would Zope be appropriate?
> 

Sure, I can see Zope easily scaling to this level.

> I know that my website will have similar and greater requirements
> concerning database size and searching. Hopefully 
> hits/traffics also will
> be high.
> 
> Zope and Python are my first choice.

Cool!  You won't be disapointed.

-Michel

> 
> Thanks for any help.
> 
> Jimmie Houchin
> 
> [snip]
> 
> 
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://www.zope.org/mailman/listinfo/zope
> 
> (For developer-specific issues, use the companion list,
> zope-dev@zope.org - http://www.zope.org/mailman/listinfo/zope-dev )
>