[Grok-dev] horizontal scaling
cmulcahy at avesi.com
Wed Apr 30 16:13:28 EDT 2008
I am currently using my own home-brewed python web-application framework,
but I would like to migrate to a better option. I have always admired Zope
from afar ( very far ) but I am very intrigued by Zope3 and ( by extension )
Grok. I bought Philipp and Stephan's books and I have been trying to
determine if Zope3 technologies are what I am looking for.
I have discarded the other python "Rails-alike" frameworks because as far as
I can tell, they ( and Rails ) suffer from the same problem. Every
web-application ( no matter how small or large ) requires its own
long-running process or processes. The larger an application becomes, the
more functionality it acquires, the larger its resource foot-print becomes
even if it is not handling any requests. This does not scale.
I wrote my own python framework because although I loathe PHP, I recognized
that it had some undeniable strengths. I own a small web-hosting company,
and I buy, install, and administer the machines my applications run on, so
I have a very SysAdmin ( as opposed to Web-Developer ) way of looking at
PHP's strength? Horizontal Scaling ( for lack of a better word ).
A web-server might have 200-odd virtual-hosts serving 200 different crappy
PHP bulletin-boards, dodgy shopping carts, and bug-ridden CMS applications.
The run-time costs of hosting these 200 different applications in 200
different Apache virtual hosts at 2:30 AM when the requests dribble in at
1/second or so? Next to nothing. The resource bloat of mod_php in all
thoses httpds is neglible ( the dodgy code bloat is something else. )
Under load, the "cost" of hosting all these disparate apps with 3 to 300 php
scripts each, is simply the runtime cost of each script request. It
Note, I am not advocating for PHP. Its weaknesses are too numerous to
mention, and as a matter of fact I don't host hundreds of crappy PHP
bulletin-boards, blogs, and CMS apps but it is for reasons of security, the
prospect of painful unpaid code-maintainence, and general aesthetics, not
resource utilization. All I am saying is that it is possible to host
hundreds of websites each with hundreds of possible dynamic URLs with
processor and memory to spare on 4 year-old hardware because the general
"machinery" for processing all these requests is ready and running ( I
don't want to say "in-process" because I don't believe in the fat buggy
httpds brought on by the whole mod_perl, mod_python thing and "external
process" techniques like SCGI work very well for me ) and GENERIC, each
mod_php enabled httpd, or php fast_cgi process capable of serving one
"application" as well as another from request to request.
Zope3 is very large and I haven't been able to get the 30,000 foot view yet.
Most simply I suppose:
Is it possible for 1 zope3 application server ( or a family of zope3
application servers sharing a ZEO or relstorage-like back-end ) to provide
multiple Grok application instances for many ( maybe not 200, but 30 or 40 )
different virtual hosts ( fronted by squid, apache et.al ) .
Can an application that has not been accessed in hours ( or days ) be
"passivated" and its runtime resource costs be reduced to disk-space and a
tiny entry in an in-memory hashtable?
I know that zope3 ( and thereby Grok ) is a lot heavier than my own
framework, but it brings a lot more to the table.
I just can't afford to run one for every site ( large or small ) that needs
I have a vision of one redundant zope3-cluster ( backed by relstorage on a
slony-replicated postgresql cluster ) that can provide all of the
web-application infrastructure for all of my development work for various
disparate customers going forward.
If this is feasible?
If this were possible it would be a slam-dunk for the adoption of grok/zope3
over rails, turbogears, django, et.al.
More information about the Grok-dev