[Zope-dev] Massive scalability

[Michael Bernstein]

John Eikenberry wrote:
> 
> Michael Bernstein wrote:
> 
> > So, again: Has anyone run up against any performance or
> > other limitations regarding large numbers (hundreds of
> > thousands or more) of objects stored within the ZODB either
> > in a BTree Folder or a Rack?
> 
> I was looking into the same issues recently, but for a much smaller set of
> data (50000ish). In my tests ZPatterns/binary-trees scaled well for storage
> and retrieval. But ZCatalog did not. It was basically useless for partial
> matching searches (taking many minutes for searches that retrieved more
> than 100 matches)

Was this true even for cases where the batch size was
smaller than 100? For example, if a search returns over 100
results but the batch size is only 20 (so that only 20
results at a time are displayed), do you still get the
performance hit?

> [snip]
> I ended up deciding to go with a RDBMS backend for data storage with a
> ZPatterns interface. SkinScripts work so well for this that I'm actually
> glad I switched. It simplified my design and implementation immensely.

So you're saying that you are doing all searching using SQL
statements, and not just object retreival and storage,
correct? How are you handling full text searches?

Cheers,

Michael Bernstein.