[ZODB-Dev] Very Large Application Use Case

[sean.upton@uniontrib.com]

This is likely too open-ended of a question.  Perhaps you could be more
specific as to what types of transactions?  Financial transactions, content
changes?  Will you be using something like ZCatalog and updating it?  Could
your changes to certain pieces of data be queued for batch writes later (For
example, the QueueCatalog product in zope.org cvs uses this technique)?  How
many perisitent objects would be changed in one of these transactions?  How
much data can you stuff into a RDB? Are you able to approach this problem
using a hybrid storage technique with some data stored in the ZODB and some
data stored in a relational database?  How big do you anticipate your object
database would be, both in terms of number of objects and MB?  

When you say 8Mill users, do you mean registered users?  If so, I don't
think Zope/ZODB is the place where scalability concerns would be an issue -
rather, this would be a question for your directory services infrastructure
(i.e. asking the question, can OpenLDAP handle 8 Million users).  If these
are not registered users, is there a reason that the number of users is
otherwise relevant that might affect performance?

400,000 concurrent users means what?  Total number of HTTP sessions open at
any given time?  Total number of people browsing the site at any given point
in time?  Total number of people cookied with login sessions?

12Million transactions in 15 hours during peak part of day?  Are these all
writes? Does this mean writes to a database, and if so, what kind of writes?
Session data? Data inputted by users?  Financial transactions?  What writes
can be deferred for later?  With this quantity, I would think a relational
database might be the best fit: can you utilize both ODB/RDB where
appropriate?

I think with an application with the write pressure you are looking at, and
with the load you are expecting, I would largely think it would be well
within your interests to do your own benchmarks and experimentation with
real hardware under simulated loads (at this scale, you would likely need to
scale both up and out and figure out strategic answers to the questions of
how to partition storage, etc).  It would likely be worth your while to hire
someone familiar with the technology under high-stress to provide consulting
(i.e. Zope Corp) for a strategic evaluation to determine if the technology
was right for the project, and how to best utilize it.  I would expect that
a project with these kind of demands would need to have a budget for this
sort of anticipatory testing and evaluation, whether done in-house or by a
vendor intimately familiar with the technology.

Canned numbers likely will not do much for you here, since you likely have
very specific needs.  Generic benchmarks are, unfortunately, often a lie in
specific cases (I get 40,000 Pystones out of a User-Mode-Linux virtual
machine, which is obviously bogus). 

Sean

-----Original Message-----
From: Nick Pavlica [mailto:nick.pavlica@echostar.com]
Sent: Wednesday, February 26, 2003 7:50 AM
To: zodb-dev@zope.org
Subject: [ZODB-Dev] Very Large Application Use Case


All,
  I know there have been many zope/zodb performance questions in the past, 
however I haven't found any statistical information regarding their use in 
very large applications.  I'm developing the requirements for a new 
open-source project, and estimate the following utilization for a single 
installation:

- 8,000,000 potential users
- 400,000 concurrent users
- 12,000,000 + transactions in 15 a hour period.  

I would love to have some feedback from those that have, or are using zope
in 
a "Real" 24x7 production facility under these or larger work loads.  

Thanks for your feedback!
-- 
Nick Pavlica
EchoStar Communications
CAS-Engineering
(307)633-5237

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev