[ZODB-Dev] Very Large Application Use Case

[Nick Pavlica]

Sean,
  You have asked some very good questions and I agree with most of what y=
ou=20
have to say.  I would love to help support Z.C. if given the opportunity.=
 =20
However, some clarification of the question may better address it.  I was=
=20
hoping to get information based on actual implementations with these type=
s of=20
generic statistics, and not necessarily possible solutions or problems in=
=20
trying to achieve that type of scalability.  Obviously, the implementatio=
n=20
details will greatly affect this, and those are some of the details I was=
=20
looking for.

------------------------
Example:
  Yes we have a site with an average of these stats:  =20
    - 8,000,000 potential users
    - 400,000 concurrent users
    - 12,000,000 + transactions in 15 a hour period.
 =20
 To achieve this type of scalability we are doing the following:
 -  We are running Zope x.x.x, ZE0 x.x.x, and the Oracle storage adapter =
x.x.x
 -  Zope/ZEO is configured on 20 Alpha DS20's with 4GB of ram running lin=
ux=20
kernel 2.4.18 and the XFS file system.
-  Our storage servers is an Oracle9i cluster deployed on 3 Alpha ES40s w=
ith=20
8GB of ram and True 64 4.0g.
- Because of the load we had to write allot of c/cpp modules to solve key=
=20
performance problems x and y.
- Zope corp was a great asset and did allot of optimization work for us.
- etc, etc.  =20
------------------------

Thanks Again!

Nick

On Wednesday 26 February 2003 1:53 pm, sean.upton@uniontrib.com wrote:
> This is likely too open-ended of a question.  Perhaps you could be more
> specific as to what types of transactions?  Financial transactions, con=
tent
> changes?  Will you be using something like ZCatalog and updating it?  C=
ould
> your changes to certain pieces of data be queued for batch writes later
> (For example, the QueueCatalog product in zope.org cvs uses this
> technique)?  How many perisitent objects would be changed in one of the=
se
> transactions?  How much data can you stuff into a RDB? Are you able to
> approach this problem using a hybrid storage technique with some data
> stored in the ZODB and some data stored in a relational database?  How =
big
> do you anticipate your object database would be, both in terms of numbe=
r of
> objects and MB?
>
> When you say 8Mill users, do you mean registered users?  If so, I don't
> think Zope/ZODB is the place where scalability concerns would be an iss=
ue -
> rather, this would be a question for your directory services infrastruc=
ture
> (i.e. asking the question, can OpenLDAP handle 8 Million users).  If th=
ese
> are not registered users, is there a reason that the number of users is
> otherwise relevant that might affect performance?
>
> 400,000 concurrent users means what?  Total number of HTTP sessions ope=
n at
> any given time?  Total number of people browsing the site at any given
> point in time?  Total number of people cookied with login sessions?
>
> 12Million transactions in 15 hours during peak part of day?  Are these =
all
> writes? Does this mean writes to a database, and if so, what kind of
> writes? Session data? Data inputted by users?  Financial transactions?=20
> What writes can be deferred for later?  With this quantity, I would thi=
nk a
> relational database might be the best fit: can you utilize both ODB/RDB
> where appropriate?
>
> I think with an application with the write pressure you are looking at,=
 and
> with the load you are expecting, I would largely think it would be well
> within your interests to do your own benchmarks and experimentation wit=
h
> real hardware under simulated loads (at this scale, you would likely ne=
ed
> to scale both up and out and figure out strategic answers to the questi=
ons
> of how to partition storage, etc).  It would likely be worth your while=
 to
> hire someone familiar with the technology under high-stress to provide
> consulting (i.e. Zope Corp) for a strategic evaluation to determine if =
the
> technology was right for the project, and how to best utilize it.  I wo=
uld
> expect that a project with these kind of demands would need to have a
> budget for this sort of anticipatory testing and evaluation, whether do=
ne
> in-house or by a vendor intimately familiar with the technology.
>
> Canned numbers likely will not do much for you here, since you likely h=
ave
> very specific needs.  Generic benchmarks are, unfortunately, often a li=
e in
> specific cases (I get 40,000 Pystones out of a User-Mode-Linux virtual
> machine, which is obviously bogus).
>
> Sean
>
> -----Original Message-----
> From: Nick Pavlica [mailto:nick.pavlica@echostar.com]
> Sent: Wednesday, February 26, 2003 7:50 AM
> To: zodb-dev@zope.org
> Subject: [ZODB-Dev] Very Large Application Use Case
>
>
> All,
>   I know there have been many zope/zodb performance questions in the pa=
st,
> however I haven't found any statistical information regarding their use=
 in
> very large applications.  I'm developing the requirements for a new
> open-source project, and estimate the following utilization for a singl=
e
> installation:
>
> - 8,000,000 potential users
> - 400,000 concurrent users
> - 12,000,000 + transactions in 15 a hour period.
>
> I would love to have some feedback from those that have, or are using z=
ope
> in
> a "Real" 24x7 production facility under these or larger work loads.
>
> Thanks for your feedback!

--=20
Nick Pavlica
EchoStar Communications
CAS-Engineering
(307)633-5237