[Zope-dev] Massive scalability

[Michael Bernstein]

RDM wrote:
> 
> On Mon, 15 Jan 2001, Michael Bernstein wrote:
> > as Squishdot). Adding a dependency on an RDBMS or requiring
> > additional setup on the server's FS seems a step in the
> > wrong direction.
> [...]
> > So the question remains: Will either approach (within the
> > ZODB) allow me to scale the application to hundreds of
> > thousands (or even millions) of objects indexed in a
> > ZCatalog?
> [...]
> > I know that the ZCatalog/ObjectManager approach used by
> > Squishdot will scale to over 9,000 objects (the number of
> > postings to date at technocrat.net), So I'm reasonably
> > certain that my proposed ZCatalog/BTree Folder approach will
> > be at least as scalable. I'm slightly less confident about
> > the Specialist/Rack approach, because I don't know of any
> > sites that have used them to store that many objects in the
> > ZODB, but only slightly.
> 
> My understanding is that the point of ZPatterns is to hide
> the data storage implementation from the application.[snip]
> The point being that you can *change your
> mind* later, with minimal disruption to your application.  Not
> only that, but people who *use* your product can make their
> own decision about where to store the data.  So by using ZPatterns
> you [...] let the users
> of your product use an RDBMs if that works better for them.

Very good points, and ones that I will keep in mind. Thanks.

> In addition, it seems to me that your comments about ZCatalog+BTree
> apply equally well to ZPatterns, since you can use the Catalog
> to index stuff stored in a rack through the use of appropriate
> triggers, and it is my understanding that the default in-ZODB
> rack storage uses BTree internally.

I do not know if BTree folders and Racks share the same
B-Tree implementation, which is why I qualified my statement
as 'slightly less confident'.
 
> Unfortunately I don't have much input on your question about
> real-life scalability...the most I've done is stored 60000 small objects
> in a hierarchy of zope folders, indexed by the catalog, with
> no perceptable slowdown in search or retrieval speed.

Hmm. John Eikenberry mentioned a slowdown with about 50,000
objects on partial-match searches, but I don't know how
simple/complex the objects were, or how many atributes were
being indexed. How many indexes of various types was your
ZCatalog maintaining on your objects?

Thanks,

Michael Bernstein.