[Zope] Some basic questions on Zope performance/scalability

[Jens Vagelpohl]

first of all, please do not crosspost. this admonishment is part of 
every message footer sent out on these lists. i restricted the answer 
to zope@zope.org, where this question belongs.

> 1. I made a custom user folder using MySQL as a storage backend. But I 
> always feel wary about that User Folder method getUsers(). I have 
> about 300,000 users in a MySQL db table. Is this method really meant 
> to return that many users? Or the fundamental design of a Zope User 
> Folder is intended for a small group of users?

no one forces you to return all users. as a data point, the 
LDAPUserFolder product will only return those user objects marked as 
"active" in the built-in user object cache at that moment. it will not 
try to retrieve all records, which would possibly bring the site to a 
complete halt if the LDAP database had thousands of users.


> 2. Again, I have a 300,000-strong user base. Would it be wise to let 
> them create a home folder and start adding things of their own? Could 
> ZODB's FileStorage model take that much data without any problems or 
> without severe perfomance penalty? I'm trying to pull out huge data 
> objects as external files (as I did with NeoBoard's attachments), but 
> I guess there's a limit to that approach too (and it's seemingly 
> stupid not to use ZODB for all the benefits it offers).

FileStorage is more robust than most people think. my personal opinion 
is that if you stay away from the current versioning implementation it 
will be able to take most abuse you can throw at it.


> Simply put, say you already have 300,000 users with tons of data for 
> each of them and you want to put them and their data somewhere in 
> and/or outside of Zope. What would you do?

if you want large folders you need to look at BTree Folders, which are 
better suited to hold many objects. the BTreeFolder2 product (-> 
cvs.zope.org underneath "Products) is probably the best at this point.

also, since boundless objects in a folder will make your zope 
management interface access to those folders slow (and possibly 
impossibele) you should devise a more intelligent path scheme for the 
member folder locations. just putting everyone under "Members" is a bad 
idea. a suitable scheme (just an example, there are countless others) 
would be like this:

- user "joeblow" is under /members/j/jo/joeblow
- user "fredjones" is under /Members/f/fr/janedoe

jens