[Zope3-dev] Through-the-web vs file-system development

[Jim Fulton]

Recently, we've had some ideas that could affect the plans
for through-the-web and file-system development.  I'd like
to share these ideas and see what other folks think.  But first, some
background.

Traditionally, Zope has supported two modes of development:

- Through-the-web (TTW) development

   In this mode, a web UI is used to create various kinds of
   software and configuration that are stored in the object database.

- File-system-based (FSB) development

   In this mode of development, software and configuration are managed
   as ordinary files on the system hosting Zope.

A third option, database-based (DBB) development is a variation on TTW
development. Software and configuration are stored in the object
database, but get into the object database through some sort of secure
means so that the software can be trusted,

TTW development has a number of advantages:

- Convenience

   o Changes can be made and (informally) tested interactively. There is
     no need to restart Zope when a change is made.

   o Changes can be made from anywhere. Direct access to the server's
     host system is unnecessary.

- ZEO cluster update

   Updating FSB software in a ZEO cluster is problematic because it's
   not possible to update all nodes in a cluster to the same version of
   software at the same time without bringing the cluster down.  If
   nodes are updated individually, then different versions of software
   will be running on different nodes.  Data produced by the new
   software won't be handled correctly by the old software running on
   the nodes that haven't been updated yet.

   TTW (DBB) development supports orderly update of a ZEO cluster.
   Because software is stored in the object database, changes are
   applied transactionally to the entire cluster.

- Local configuration

   Software and configuration can be loaded into specific folders in
   the database.

TTW-development has advantages too:

- It's more familiar to Python programmers

- You can use file-system based tools

   (Note however, that this is possible with DBB software too using
    file-system synchronization.)

- File-system code is trusted.

   (Note-however, that Zope 3 will allow DBB and/or TTW software to be
    optionally trusted.

    Also, the difference between trusted and untrusted code is a bit
    smaller in Zope 3.
    )

Supporting orderly update of FSB software in ZEO clusters

   The ability to update a ZEO cluster in an orderly manner has been a
   significant driver of the development of TTW tools, such as persistent
   modules, in Zope 3, however, creating a TTW development environment
   rich and robust enough to support TTW software "package" (previously
   known as "product") development remains a significant challenge.

   It is common to write software that supports multiple versions of data
   structures.  This is typically done by writing __setstate__ methods
   that accept multiple state structures, converting them to whatever
   data structure is currently expected. This is normally done to accept
   saved states written by past software versions.  These same technique,
   however, can be used to support *future* software versions.

   We can support orderly ZEO cluster updates by updating a cluster in
   two steps:

   - In the first step, an intermediate software update is installed that
     creates new objects with old data structures, but that understands
     new data structures.

   - In the second step, the new software version is installed that
     creates new objects with the new data structures.  This is possible
     because nodes that haven't been updated yet have software from the
     intermediate release that can work with the new data structures.

   To facilitate this sort of approach, we can provide some frameworks
   that make managing multiple state versions easier.  (Details of that
   framework will be provided in a separate proposal.)

   An alternative approach is to provide an intermediate release that
   doesn't allow creation of new objects. This represents a temporary
   reduction in service that might be acceptable during an upgrade cycle.

   With these approaches, we can provide update support for ZEO clusters
   without resorting to database-hosted code. This is very good news.

Creating local configuration with FSB software

   Recently, Steve and I have been discussing ways that we could use ZCML
   to define multiple site configurations. This could be used to support
   local configuration in ZCML.  This could work a number of ways.
   For example, having defined multiple named site configurations in
   ZCML, one could then configure a local site by simply selecting one of
   the named configurations.

   An added complication is that locally-configured components often need
   to store persistent state.  For example, catalogs and indexes are
   updated in response to object-modification events.  If such a
   component is defined in ZCML, there will need to be a way to
   let it store some of its state in the database. This is rather
   difficult.  A component cannot simply load it's data from the
   database on startup.  Each application thread needs to use a
   separate database connection.  Components with persistent state need
   to be loaded from whatever database connection(s) an application
   thread is using.

   While there are challenges, I'm pretty sure that ZCML-based
   configurations can be made to work locally with a moderate effort.

So what does all of this mean?

I think that we can arrange to use FSB development for both local and
global configuration and to support update of FSB software in ZEO
clusters. I believe that this will require far less effort and has far
less risk than providing a full TTW development and software packaging
environment.

TTW development still has the advantage of convenience.  I suggest that
TTW-development techniques should be used for scripting or "active
content" development.  I also think that it still makes sense to use
TTW-development techniques to support rapid development and
prototyping. After software is initially prototypes, file-system
synchronization will be used to convert TTW-developed software to FSB
software. The fact that Zope 3 uses persistent modules that closely
resemble regular FSB modules should make the transition of software
from TTW to FSB straightforward.

I suggest than that, for the foreseeable future, we treat TTW
development as a technique to be used solely for scripting and
prototyping and that we make sure it is straightforward to create
local configurations in ZCML and to update FSB software in ZEO
clusters.

This will have little or no impact on the Zope X3.0 release.
I propose to make the support for local ZCML configuration a focus of
the X3.1 release.  This will have an impact on documentation and
philosophy of Zope development.

Thoughts?

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org