[Zope3-dev] Some thoughts on Zope 3, Zope 3 applications, and Zope 3 instances

[Jim Fulton]

Some thoughts on Zope 3, Zope 3 applications, and Zope 3 instances

I'd like to share some thoughts about what Zope 3 is (to me), what a  
Zope 3 application is, and what a Zope 3 instance is, or should be.   
I think this should influence our thinking about Zope 3 releases and  
tools we create, like zc.buildout and builout recipes, that help  
build and deploy applications.

First some background.

Zope 2 was first and foremost an application.  It was extensible and  
people could build applications on top of it, but that was through an  
extension mechanism.  Over time, greater and greater emphasis was  
placed on building applications on top of it.  I think it was and  
remains a pretty useful application.  I'd love to see a similar  
application built using Zope 3.  Perhaops that application will be  
Zope 2 someday. :)

Zope 3 was designed first for application developers.  It was always  
envisioned as a collection of components that could be assembled and  
combined with other components in a variety of ways.  In this way,  
Zope 3 is more like a library (actually a set of frameworks and  
libraries) than an application.

The current Zope 3 distribution also contains an application that  
resembles Zope 2 in many ways. There is an object tree and a Zope  
Management Interface.  At least up to now, when you install a Zope 3  
distribution, you get an application that you can run out of the box,  
for whatever that's worth.  In this document, I'll call this  
application the the Object Filing System (OFS).

I've found the OFS to be useful when developing Zope 3 itself.  The  
OFS was really the first application developed with Zope 3.  There  
are also certain kinds of applications that can use the OFS as a  
starting point.  We've built content-management systems that reuse  
much of it.  In the later case, we've used it less as an application  
than as a library.  It's not obvious to me that that the OFS has much  
value as an application in it's current form.

For the last year, I've been working on supporting some older  
applications in our hosting environment.  In doing that, I've come to  
appreciate some aspects of large scale application deployment:

- Deployments have to be highly automated. (We are moving toward  
deploying applications as RPMs in our hosting environment. These RPMs  
are created using zc.buildout and include the applications and tools  
for creating application instances.)

- Deployments like the ones we do, have many (tens of) instances of  
the same application.

- Applications need to be highly manageable.  Among other things,  
this means that application artifacts, like software, configuration  
files, log files, and so on should be in predictable, easy to find  
locations. (e.g. http://www.pathname.com/fhs/)

mkzopeinstance doesn't serve these needs very well.  It is a bit hard  
to automate, as it isn't as flexible as needed to define many  
application parameters.  When many instances are used, there tends to  
be a lot of duplication.  An extreme case of this is the package- 
includes directory (or the Zope 2 Products directory).  The instance  
directories distribute files in ways that Unix system administrators  
find hard to deal with.  When dealing with many instances, I too have  
found the traditional instance directory layout hard to deal with in  
production.

One of the problems with traditional zope instances is that they  
combine application definition with instance configuration.  For  
example, a Zope 3 instance created by mkzopeinstance includes lots of  
ZCML files, in the form of generated ZCML files such as site.zcml,  
and the contents of package-includes.  It also contains a number of  
generated scripts. This information is nearly identical accross  
instances of the same application.

My current thinking, FWIW:

It's not clear to me that the Zope 3 application we now distribute  
has much value.  I'd be interested to hear other people's thoughts on  
this.  I'd like to see some popular Zope 3 applicaton or applications  
that people download and use to get excited about Zope 3.  I don't  
know if anyone is developing something like that. I hope they are.  I  
don't think anybody is trying to turn the existing OFS into an  
application like that.

If Zope 3 is really primarily a library, then maybe we should  
distribute it that way, as eggs or as setuptools aware source  
distributions.

Zope Corporation is creating applications with Zope 3.  While these  
applications use many of the components from the OFS, they are not  
the OFS, nor are they extentions of the OFS.  We don't use package- 
includes. Rather we define an application package that has a ZCML  
file that includes ZCML files for the packages we use.  In the  
future, our application packages will have setuptools-style package  
dependencies on the packages they use.  The traditional Zope 3  
distribution won't be useful to us.

I'd like to separate application definition from instance  
definition.  For our needs, there will be a zc.buildout recipe for  
definiing a Zope 3 application.  This application will consist of a  
generated site.zcml file, a small number of scripts, and a bunch of  
eggs.  Unlike the scripts generated by mkzopeinstance, the scripts  
generated by the new application recipe will be indepencent of any  
instances and will not embed paths to configuration files.

There will be an instance recipe that can generate instances that use  
applications.  In a (Unix) production deployment, these instances  
will have their files spread around more or less according to http:// 
www.pathname.com/fhs/.  There won't be an instance directory.  For  
development, instances will probably still be directories, but will  
be much smaller and shallower than those produced by the existing  
mkzopeinstance.  They will probably have a zope.conf files and  
control scripts and their directories will also contain log and  
zdaemon socket files. (Perhaps Windows production instances would be  
similar.)  Instances won't contain ZCML files, because ZCML files are  
part of the application definition and multiple instances will share  
the same application definition.

Of course, I'll release these recipes when I have them.

In summary, I'm seeing Zope 3 applications as separate entities from  
the OFS.  The OFS application isn't something we'll use directlty.   
Instances will be instances of our applications, not of Zope 3.

I should note that there have been a number of sucessful Zope 3  
applications that were not the OFS.  In fact, some of the earliest  
production Zope 3 applications were not based on the OFS.

None of this should be taken as any sort of edict.  I'm also not  
trying to name anything. While I'd love to spur discussion, I hope  
not to start any arguments. :)

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org