[ZODB-Dev] How to implement a production / test environment using zope ?

[Anthony Baxter]

We take a more extreme approach - we copy _everything_ out. We have 
4 stages - (dev) development, (beta) internal test, (test) external 
test, (prod) production.
There's load balancers in front of each of the 4 systems, and a second
smaller set of zeo clients/server for each called foo_upgrade that are
used during upgrades.

All changes only happen on the dev system, and then they're tested 
with both internal and external testing.

The upgrade process (automated) does the following:

To upgrade server foo:
  . Check that the server foo_upgrade is not running (should be fine)

  . Update the CVS (on disk code) for foo_upgrade from the server before
    it in the pipeline.  (so for test_upgrade, use the current beta code). 
    Rebuild the code. It does this for both the zope code and the 
    instance_home, for zeo client and servers. This can be on multiple 
    machines - e.g the production system is 8 zeo clients on 5 machines, as 
    well as the zeo server.

  . Update foo_upgrade's Data.fs from the server before it in the pipeline,
    and pack it. 

  . Start the foo_upgrade zeo server. Wait for it to come up, test it's sane,
    then start the foo_upgrade zeo clients. Check they're running and sane. 

  . Tell the load balancer to direct requests to the foo_upgrade clients, not
    the foo clients.

  . Shut down the zeo clients and servers for server foo.

  . Upgrade the CVS and Data.fs for server foo (zeo clients and servers)
    from foo_upgrade (as above).

  . Start foo back up, check it's sane.

  . Switch the loadbalancer back to foo.

  . Shut down foo_upgrade.

This sounds all awfully complex, but it's actually now just a matter of
running 'updateserver beta' as a command line. The script I've developed
for controlling it all is about 1300 lines of python, plus the config 
file that controls all this (plus the other various zope servers here 
(there's a lot of them) is another few hundred lines). The script also 
does all the work of checking the systems are working, keeping the configs 
up to date on the various machines automagically, and a host of other 
annoying maintenance tasks that I wanted automated away.

There's also a bunch of stuff for pushing out smaller chunks - for instance
a single partner branding can be pushed out by a web developer when it's
ready from a simple zope interface. (our web app is heavily customisable
for our different partners)

I'm not sure if this is useful to people, but I thought it might at least 
be interesting.

Anthony