[Grok-dev] Re: horizontal scaling
faassen at startifact.com
Wed Apr 30 16:53:33 EDT 2008
Christopher Mulcahy wrote:
[snip discussion of horizontal scaling]
> Zope3 is very large and I haven't been able to get the 30,000 foot view
> Most simply I suppose:
> Is it possible for 1 zope3 application server ( or a family of zope3
> application servers sharing a ZEO or relstorage-like back-end ) to
> provide multiple Grok application instances for many ( maybe not 200,
> but 30 or 40 ) different virtual hosts ( fronted by squid, apache et.al ).
Good question. I don't have experience with this amount of scaling. I
think the best chance to get people who do have such experience (if they
exist) is to ask this question on zope-dev as well. But there are some
of those people hanging out on grok-dev, so let's see whether someone
else replies. Christian Theune (who will be at the grokkerdam sprint
tomorrow) may be able to give interesting feedback, for instance.
> Can an application that has not been accessed in hours ( or days ) be
> "passivated" and its runtime resource costs be reduced to disk-space and
> a tiny entry in an in-memory hashtable?
ZODB (and ZEO) do keep a cache around of objects that have been
accessed, so I don't that this will happen on a Zope level. That said,
the operating system may swap out the application if it isn't used a
lot, but that's probably not what you ask for. I don't know what the
memory characteristics of RelStorage are - zope-dev or zodb-dev should
get you better answers.
There are ways emerging to run Zope without the ZODB at all (or just
with a very tiny ZODB), and store the data in a relational database, for
instance using Storm or SQLAlchemy. This may have better memory
characteristics. It's likely a typical relational object mapper still
has an object cache around, but it may be smaller, or at least it may be
easier to make this smaller. I imagine a typical PHP application would
still have a smaller amounts of objects around, as no relational object
mapper is used at all, and PHP also tends to have a "run the page and
then forget about whatever happened" nature.
> I know that zope3 ( and thereby Grok ) is a lot heavier than my own
> framework, but it brings a lot more to the table.
> I just can't afford to run one for every site ( large or small ) that
> needs dynamic content.
> I have a vision of one redundant zope3-cluster ( backed by relstorage
> on a slony-replicated postgresql cluster ) that can provide all of the
> web-application infrastructure for all of my development work for
> various disparate customers going forward.
> If this is feasible?
It's possible to run multiple applications in a single Zope instance.
There are drawbacks to this - if you upgrade Grok to a newer version (or
just a library), you would need to update all your underlying code, not
just for one application. Also a restart will be interesting - you could
restart each front-end separately until all are using the newer version
of your code.
I think it would also be possible to run different ZEO or RelStorage
frontends that share the same database backend. This way you could do
incremental updates while sharing the same database backend. I think
this would need some special approach though, as even though you'd use
one big database in the back, an individual frontend server would not be
able to access all the objects that are in that ZODB, only the ones that
make sense in the context of its application. I don't think there's
anything fundamentally stopping you from doing so, but I also suspect
there are few tools to support you there.
> If this were possible it would be a slam-dunk for the adoption of
> grok/zope3 over rails, turbogears, django, et.al.
Unfortunately I doubt that there's an easy "yes" or "no" to this answer.
It also would need to be measured whether such a setup would have
reliability and memory advantages in the real world.
The question would be most easily answered if someone already has done
things like this. I'm not really the right person to answer, but I still
wanted to provide you some feedback. :) zope-dev again would be a good
place to ask.
Thanks for your questions - it's interesting to see this kind of
question coming in; gives us new perspectives and news ideas, even if,
sadly, we may not end up being able to do what you're asking for.
More information about the Grok-dev