[Zope-dev] Specialist/Rack scalability

[Steve Alexander]

Hi Michael,

Michael Bernstein wrote:

>
> It seems clear that indexing and searching are more of a
> botleneck than storage/retreival. Nevertheless, so far I
> have not heard of anyone trying to store more than 60,000
> objects in a rack. I need to know if there is any reason to
> suspect that storage (in the ZODB) or retreival performance
> would suffer if the number of objects was in the hundreds of
> thousands or even millions.

I can't answer your question; however, I may be able to help clarify the 
question.

The ZODB is really just a transaction manager, and an interface and 
contract of behaviour, for an object database.

You can plug a variety of Storages into the ZODB. The default storage 
the Zope comes with is FileStorage -- Data.fs.

There are also BerkeleyStorage, OracleStorage, DBMStorage, and others, 
in varying states of finishedness.

So, storing things in a Rack happens in a number of stages:

   Your application interacts with the Rack
   The Rack (perhaps) stores the object persistently in its BTree
   The BTree is a collection of persistent ZODB objects
   The ZODB objects are stored as Python Pickles in a FileStorage

We can consider what the effect of storing 60 000 objects is at each of 
these interfaces.

The Rack shouldn't have a problem with 60 000 objects.

I doubt a BTree would have a problem.

The ZODB might not like accessing many large objects during a single 
transaction, as all those objects need to be in memory at once.

A FileStorage should have no problem reading 60 000 stored objects. 
However, if these objects are changing much, your Data.fs will grow 
fast. In any case, you may find undo and history screens take a long 
time to appear.

However, if you are using a Rack, you have a lot of choice about where 
you put your data. You can put frequently changed aspects of your data 
on the filesystem, and the rest in FileStorage for example.

--
Steve Alexander
Software Engineer
Cat-Box limited
http://www.cat-box.net