[Zope3-dev] Continuous integration, ZopeInstall
Tres Seaver
tseaver@zope.com
19 May 2003 09:28:55 -0400
--=-Q1pfU8ZVR6o9/sJvAlzW
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
On Sun, 2003-05-18 at 08:47, PieterB wrote:
> Hi,
>
> I've been working on a script to generate a Zope instance from scratch
> (based on Sidnea's scripts), see thread on Zope-Dev:
> http://mail.zope.org/pipermail/zope-dev/2003-May/019627.html
>
> I think a script like that will also be usefull for future installers
> (e.g. FreeBSD ports, RPM's, etc.) and for elaborating continious
> integration for Zope/Python.
>
> It would be great to have something similar to the Java developers, such
> as:
> - Gump (http://jakarta.apache.org/gump/)
> The primary goal of Gump is to get diverse projects to communicate
> early and often about integration, dependencies, and versioning
> management.
>
> - AntHill (http://www.urbancode.com/projects/anthill/default.jsp)
> tool that ensures a controlled build process (makes nightly builds, runs
> unit tests, generates source code metrics)
>
> - Maven (http://maven.apache.org/)
> Maven is a Java project management and project comprehension tool
> (metrics, checkstyle, changelog, file activity, docs, tests, tasks, etc.)
>
> The first step would be to able to generate SandBoxes using the
> same method on every platform. I think the building of Python and
> Python modules (e.g. PyXML) can be best done using a shellscript
> and/or Makefile. I think it would be the best to write some Python
> script that is able to download/get the necesarry files from websites
> or CVS (using wget and a CVS-client) and check the dependencies.
> The scripts will be controlable using environment variables. For
> example to get the most bleeding edge Zope 2, use something like:
>
> 8<-----------------------------------------------------
>
> #!/bin/sh
> PYTHON_VERSION=2.3b1 # 2.1.3, 2.2-stable, 2.2.3, 2.3b1, 2.3-dev, dev
> ZOPE_VERSION=2.7-dev # 2.6.1, 2.6-stable, 2.6-dev, 2.7-dev, 3-dev, dev
> PLONE_VERSION=1.0.2-dev # Empty (don't install plone), 1.0.1, 1.0-stable,
> # 1.0.2-dev, dev
> ZWIKI_VERSION=0.18-dev # 0.18, 0.18-dev, dev
>
> make build_python
> make test_python
> make build_zope_standbox
> make test_zope
> make test_zope_products
> make start # run the instance
> make publish_results # e.g. send e-mail to some mailinglist
>
> 8<-----------------------------------------------------
>
> I think the best name for such a product/pacakage will be ZopeInstaller.
>
> Please let me know what you think of this idea.
/me borrows the time machine.
Attached is the most recent version of a script for building out the
Zope sandbox we will be using at OSCOM Zope interop workshop. Note a
couple of things in particular about it:
- It builds its own Python, which might seem controversial. However,
relying on the "system" Python is insupportable:
o There is no guarantee that one can install needed modules (PyXML,
for instance) without being root.
o There is no guarantee that the system version is usable
(threading, large-file support, and specific Python versions
are all needed by Zope).
o There is not guarantee that the system version won't change
without warning (e.g., during an OS upgrade), and break the
site.
- Except for projects whose CVS is hosted at SourceForge (Python and
PyXML), everything is built from anonymous CVS checkouts. SF
is unreliable for anonymous CVS, so the script fetches tarballs for
those bits using wget.
- The script goes out of its way *not* to write into the source tree
at all: where possible, it uses 'configure' from a foreign
directory; where not, it creates symlink trees in the foreign
directory. The overall goal is to be able to mount the 'src'
tree via read-only NFS, or from a CD-ROM.
- The script builds an INSTANCE_HOME + ZEO sandbox (the *only* way to
fly :).
- Everything is built / installed "locally", and should run without
any dependencies on the rest of the system beyond 'libc' and
friends.
- The specific versions being built are in "version-qualified"
directories, which are then symlinked to more "well-known" names.
I can therefore just look at the output of 'ls -l opt' to tell which
versions of software are used, and can change the installed version
just by moving a symlink.
I am not suggesting that this particular script is the best way to
achieve these goals (I use a more complicated framework for consulting
projects, for instance); I do think that the goals are important for
any such effort, however.
Tres.
--
===============================================================
Tres Seaver tseaver@zope.com
Zope Corporation "Zope Dealers" http://www.zope.com
--=-Q1pfU8ZVR6o9/sJvAlzW
Content-Disposition: attachment; filename=buildout_zope_sandbox
Content-Transfer-Encoding: quoted-printable
Content-Type: text/x-sh; name=buildout_zope_sandbox; charset=ISO-8859-1
#!/bin/sh
#=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
# OSCOM 2003: Zope CMS Interoperability Workshop
#
# Build out a working Zope sandbox from sources.
#
# - Python and PyXML build from tarballs, because SourceForge won't
# do :pserver: CVS reliably.
#
# - Zope, CMF, and other products build from CVS checkouts.
#
# Usage:
#
# Run this script from within an empty "sandbox" directory.
#
# $ mkdir /tmp/OSCOM
# $ cd /tmp/OSCOM
# $ /tmp/build_sandbox
#
# Theory of Operation
#
# This script creates all the software needed to start a Zope applicati=
on
# server (with the corresponding ZEO storage server); in particular,
# it *builds its own Python*, to avoid clashing with the system version=
,
# and to ease installation of packages which Zope depends on.
#
# It creates the following top-level directories:
#
# 'src' -- "pristine" sources (from tarballs / CVS checkouts), in
# "version-qualified" directories
#
# 'opt' -- "built" sources (compilation artifacts go here) are in
# "version-qualified" directories. The versions in actual
# use are symlinked to "standard" names (e.g., 'opt/Python2'
# is symlinked to 'opt/Python-2.1.3').
#
# 'bin' -- aexecutables are linked here from the 'opt' tree.
#
# 'etc' -- configuration files for the appserver / storage server.
#
# 'var' -- log, data, and pid files for running servers. The Zope
# INSTANCE_HOME is in 'var/zope'.
#
# 'tmp' -- fetch directory for tarballs.
#
# $Id: buildout_zope_sandbox,v 1.9 2003/05/14 03:01:23 tseaver Exp $
#=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
PLATFORM=3D`uname`
#--------------------------------------------------------------------------=
----
# Tool macros
#--------------------------------------------------------------------------=
----
mkdir=3D"mkdir -p"
fetch_tarball=3D"wget" # OSX: curl?
cvs=3D"cvs -q"
chmodx=3D"chmod +x"
patch=3D"patch"
untar=3D"tar xzf"
make=3D"make"
lnsf=3D"ln -sf"
cprs=3D"cp -rs" # copy files as symlinks; OSX: 'cp -a'?
cat=3D"cat"
echo=3D"echo"
$echo Building OSCOM Zope sandbox: PLATFORM is $PLATFORM
#--------------------------------------------------------------------------=
----
# Version / repository selectors
#--------------------------------------------------------------------------=
----
PYTHON_VERSION=3D${PYTHON_VERSION:-2.1.3} # XXX 2.2.3?
PYTHON_ORG=3D"http://www.python.org/ftp/python"
PYTHON_SOURCE_URL=3D$PYTHON_ORG/$PYTHON_VERSION/Python-$PYTHON_VERSION.tgz
PYXML_VERSION=3D${PYXML_VERSION:-0.8.1} # 0.8.2?
PYXML_SF_DOWNLOAD=3D"http://aleron.dl.sourceforge.net/sourceforge/pyxml"
PYXML_URL=3D$PYXML_SF_DOWNLOAD/PyXML-$PYXML_VERSION.tar.gz
ZOPE_CVSROOT=3D:pserver:anonymous@cvs.zope.org:/cvs-repository
ZOPE_VERSION=3D${ZOPE_VERSION:-2.6.1} # XXX 2.6.2, 2.7?
ZOPE_MODULE=3D"Zope"
if [ "$ZOPE_VERSION" =3D=3D "2.6.1" ]; then
ZOPE_TAG=3D"-r Zope-2-6-1-src"
elif [ "$ZOPE_VERSION" =3D=3D "head" ]; then
ZOPE_TAG=3D
fi
if [ "$ZOPE_VERSION" =3D=3D "2.6.1" -o "$ZOPE_VERSION" =3D=3D "2.6.2" ]; th=
en
ZODB3_VERSION=3D${ZODB3_VERSION:-3.1.1} # XXX 3.1.2?
ZODB3_MODULE=3D"ZODB3"
if [ "$ZODB3_VERSION" =3D=3D "3.1.1" ]; then
ZODB3_TAG=3D"-r ZODB3-3-1-1-final"
elif [ "$ZODB3_VERSION" =3D=3D "3.1.2" ]; then
ZODB3_TAG=3D"-r ZODB3-3-1-2-final"
fi
fi
CMF_VERSION=3D${CMF_VERSION:-1.3.1} # XXX 1.4?
CMF_MODULE=3D"CMF"
if [ "$CMF_VERSION" =3D=3D "1.3.1" ]; then
CMF_TAG=3D"-r CMF-1_3_1-release"
elif [ "$CMF_VERSION" =3D=3D "1.4" ]; then
CMF_TAG=3D"-r CMF-1_4-release"
elif [ "$CMF_VERSION" =3D=3D "1.4beta1" ]; then
CMF_TAG=3D"-r CMF-1_4beta1"
elif [ "$CMF_VERSION" =3D=3D "head" ]; then
CMF_TAG=3D
fi
if [ "$CMF_VERSION" =3D=3D "1.3.1" ]; then
DCWORKFLOW_VERSION=3D${DCWORKFLOW_VERSION:-0.5} # not needed for CMF =
>=3D 1.4
DCWORKFLOW_MODULE=3D"CMF/DCWorkflow"
DCWORKFLOW_TAG=3D"-r DCWorkflow-0_5"
fi
if [ "$ZOPE_VERSION" =3D=3D "2.6.1" -o "$ZOPE_VERSION" =3D=3D "2.6.2" ]; th=
en
# DBTab makes managing ZEO ClientStorage simpler under 2.6.*
DBTAB_VERSION=3D${DBTAB_VERSION:-1.2}
DBTAB_MODULE=3D"Products/DBTab"
if [ "$DBTAB_VERSION" =3D=3D "1.2" ]; then
DBTAB_TAG=3D"-r DBTab-1_2"
elif [ "$DBTAB_VERSION" =3D=3D "head" ]; then
DBTAB_TAG=3D
fi
fi
NOTSITE_VERSION=3D${NOTSITE_VERSION:-head}
NOTSITE_MODULE=3D"Products/OSCOM/NOTSite"
NOTSITE_TAG=3D
#--------------------------------------------------------------------------=
----
# Build top-level sandbox dirs
#--------------------------------------------------------------------------=
----
sandbox=3D`pwd` # XXX
srcdir=3D$sandbox/src
optdir=3D$sandbox/opt
bindir=3D$sandbox/bin
vardir=3D$sandbox/var
etcdir=3D$sandbox/etc
tmpdir=3D$sandbox/tmp
$mkdir $srcdir
$mkdir $optdir
$mkdir $bindir
$mkdir $vardir
$mkdir $etcdir
$mkdir $tmpdir
#--------------------------------------------------------------------------=
----
# Build Python
#--------------------------------------------------------------------------=
----
PYTHON_CC=3D"gcc"
PYTHON_CONFIGS=3D"--prefix=3D$sandbox/opt/Python-$PYTHON_VERSION --with-thr=
ead"
if [ "$PLATFORM" =3D=3D "Linux" ]; then
PYTHON_CC=3D"gcc -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=3D64"
elif [ "$PLATFORM" =3D=3D "SunOS" ]; then
PYTHON_CC=3D"gcc `getconf LFS_CFLAGS`"
elif [ "$PLATFORM" =3D=3D "Darwin" ]; then
PYTHON_CC=3D"cc"
PYTHON_CONFIGS=3D"$PYTHON_CONFIGS --with-dyld --with-suffix"
fi
cd $tmpdir
$fetch_tarball --continue $PYTHON_SOURCE_URL
cd $srcdir
$untar $tmpdir/Python-$PYTHON_VERSION.tgz
#
# Work around glitch in remote configure for 2.2.2
#
if [ "$PYTHON_VERSION" =3D=3D "2.2.2" ]; then
cd $srcdir/Python-$PYTHON_VERSION
$patch Lib/distutils/sysconfig.py << PYTHON_222_DISTUTILS_PATCH_EOF
--- python/python/dist/src/Lib/distutils/sysconfig.py 2002/10/08 14:59:43 1=
.44.6.2
+++ python/python/dist/src/Lib/distutils/sysconfig.py 2002/11/26 09:42:57 1=
.44.6.3
@@ -29,13 +29,9 @@
=20
argv0_path =3D os.path.dirname(os.path.abspath(sys.executable))
landmark =3D os.path.join(argv0_path, "Modules", "Setup")
-if not os.path.isfile(landmark):
- python_build =3D 0
-elif os.path.isfile(os.path.join(argv0_path, "Lib", "os.py")):
- python_build =3D 1
-else:
- python_build =3D os.path.isfile(os.path.join(os.path.dirname(argv0_pat=
h),
- "Lib", "os.py"))
+
+python_build =3D os.path.isfile(landmark)
+
del argv0_path, landmark
=20
# set_python_build() was present in 2.2 and 2.2.1; it's not needed
PYTHON_222_DISTUTILS_PATCH_EOF
fi
cd $optdir
$mkdir -p Python-$PYTHON_VERSION/build
cd Python-$PYTHON_VERSION/build
CC=3D$PYTHON_CC $srcdir/Python-$PYTHON_VERSION/configure $PYTHON_CONFIGS
$make
$make install
cd $optdir
$lnsf Python-$PYTHON_VERSION Python2
cd $bindir
$lnsf $optdir/Python2/bin/python .
#--------------------------------------------------------------------------=
----
# Install PyXML
#--------------------------------------------------------------------------=
----
cd $tmpdir
$fetch_tarball --continue $PYXML_URL
cd $srcdir
$untar $tmpdir/PyXML-$PYXML_VERSION.tar.gz
#
# Work around a bug with Unicode DTD URLs.
#
cd $srcdir/PyXML-$PYXML_VERSION
$patch xml/sax/xmlreader.py << PYXML_081_PATCH_EOF
@@ -221,7 +221,7 @@
=20
def getSystemId(self):
"Returns the system identifier of this InputSource."
- return self.__system_id
+ return str(self.__system_id)
=20
def setEncoding(self, encoding):
"""Sets the character encoding of this InputSource.
PYXML_081_PATCH_EOF
cd $optdir
$cprs $srcdir/PyXML-$PYXML_VERSION .
$lnsf PyXML-$PYXML_VERSION PyXML
cd PyXML
$bindir/python setup.py install
#--------------------------------------------------------------------------=
----
# Build Zope SOFTWARE_HOME
#--------------------------------------------------------------------------=
----
cd $srcdir
$cvs -d $ZOPE_CVSROOT co -d Zope-$ZOPE_VERSION $ZOPE_TAG $ZOPE_MODULE
cd $optdir
if [ "$ZOPE_VERSION" =3D=3D "2.6.1" -o "$ZOPE_VERSION" =3D=3D "2.6.2" ]; th=
en
$cprs $srcdir/Zope-$ZOPE_VERSION .
cd Zope-$ZOPE_VERSION
$bindir/python wo_pcgi.py
rm inituser
#
# Install ZODB (only needed for Zope 2.6.*).
#
cd $srcdir
$cvs -d $ZOPE_CVSROOT co -d ZODB3-$ZODB3_VERSION $ZODB3_TAG $ZODB3_MODU=
LE
cd $optdir
$cprs $srcdir/ZODB3-$ZODB3_VERSION .
$lnsf ZODB3-$ZODB3_VERSION ZODB3
cd ZODB3
$bindir/python setup.py install --home=3D$optdir/Zope-$ZOPE_VERSION
else
$mkdir Zope-$ZOPE_VERSION/build
cd Zope-$ZOPE_VERSION/build
$srcdir/Zope-$ZOPE_VERSION/configure \
--with-python=3D$bindir/python --prefix=3D$optdir/Zope-$ZOPE_VERSION
$make
$make install
fi
cd $optdir
$lnsf Zope-$ZOPE_VERSION Zope
#
# Build ZEO INSTANCE_HOME
#
if [ "$ZOPE_VERSION" =3D=3D "2.6.1" -o "$ZOPE_VERSION" =3D=3D "2.6.2" ]; th=
en
cd $vardir
$mkdir -p storage/var
else
#cd $optdir/Zope-$ZOPE_VERSION
$optdir/Zope-$ZOPE_VERSION/bin/mkzeoinstance $vardir/storage 8001
cd $etcdir
$lnsf $vardir/storage/etc/zeo.conf .
$lnsf $vardir/storage/etc/zeoctl.conf .
cd $bindir
$lnsf $vardir/storage/bin/zeoctl .
fi
#
# Build Zope INSTANCE_HOME
#
if [ "$ZOPE_VERSION" =3D=3D "2.6.1" -o "$ZOPE_VERSION" =3D=3D "2.6.2" ]; th=
en
cd $vardir
$mkdir -p zope/var zope/Products zope/import zope/Extensions
$echo Creating initial "manager" account.
$bindir/python $optdir/Zope/zpasswd.py $vardir/zope/inituser
else
#cd $optdir/Zope-$ZOPE_VERSION
$optdir/Zope-$ZOPE_VERSION/bin/mkzopeinstance \
--zeo localhost:8001 $vardir/zope
cd $etcdir
$lnsf $vardir/storage/etc/zope.conf .
cd $bindir
$lnsf $vardir/zope/bin/zopectl .
fi
#--------------------------------------------------------------------------=
----
# Fetch products.
#--------------------------------------------------------------------------=
----
cd $srcdir
$cvs -d $ZOPE_CVSROOT co -d CMF-$CMF_VERSION $CMF_TAG $CMF_MODULE
cd $optdir
$cprs $srcdir/CMF-$CMF_VERSION .
$lnsf CMF-$CMF_VERSION CMF
if [ "$CMF_VERSION" =3D=3D "1.3.1" ]; then
cd $srcdir
$cvs -d $ZOPE_CVSROOT co -d DCWorkflow-$DCWORKFLOW_VERSION \
$DCWORKFLOW_TAG $DCWORKFLOW_MODULE
cd $optdir
$cprs $srcdir/DCWorkflow-$DCWORKFLOW_VERSION .
$lnsf DCWorkflow-$DCWORKFLOW_VERSION DCWorkflow
fi
if [ "$ZOPE_VERSION" =3D=3D "2.6.1" -o "$ZOPE_VERSION" =3D=3D "2.6.2" ]; th=
en
cd $srcdir
$cvs -d $ZOPE_CVSROOT co -d DBTab-$DBTAB_VERSION $DBTAB_TAG $DBTAB_MODU=
LE
cd $optdir
$cprs $srcdir/DBTab-$DBTAB_VERSION .
$lnsf DBTab-$DBTAB_VERSION DBTab
fi
cd $srcdir
$cvs -d $ZOPE_CVSROOT co -d NOTSite-$NOTSITE_VERSION $NOTSITE_TAG $NOTSITE_=
MODULE
cd $optdir
$lnsf $srcdir/NOTSite-$NOTSITE_VERSION NOTSite
#--------------------------------------------------------------------------=
----
# Install products
#--------------------------------------------------------------------------=
----
cd $vardir/zope/Products
$lnsf $optdir/CMF/CMFCore .
$lnsf $optdir/CMF/CMFDefault .
$lnsf $optdir/CMF/CMFTopic .
$lnsf $optdir/CMF/CMFCalendar .
$lnsf $optdir/DCWorkflow .
if [ "$ZOPE_VERSION" =3D=3D "2.6.1" -o "$ZOPE_VERSION" =3D=3D "2.6.2" ]; th=
en
$lnsf $optdir/DBTab .
fi
$lnsf $optdir/NOTSite .
#--------------------------------------------------------------------------=
----
# Set up DBTab
#--------------------------------------------------------------------------=
----
if [ "$ZOPE_VERSION" =3D=3D "2.6.1" -o "$ZOPE_VERSION" =3D=3D "2.6.2" ]; th=
en
cd $vardir/zope
$lnsf Products/DBTab/custom_zodb.py .
$cat > $etcdir/dbtab.conf << DBTAB_CONF_EOF
[Storage: Main]
type=3DClientStorage
addr=3Dzeo://localhost:8001
storage=3D1
wait=3D1
cache_size=3D20000000
[Database: Main]
cache_size=3D4000
mount_paths=3D/
open_at_startup=3DTrue
DBTAB_CONF_EOF
$lnsf $etcdir/dbtab.conf .
fi
#--------------------------------------------------------------------------=
----
# Set up start / stop scripts
#--------------------------------------------------------------------------=
----
if [ "$ZOPE_VERSION" =3D=3D "2.6.1" -o "$ZOPE_VERSION" =3D=3D "2.6.2" ]; th=
en
$cat > $bindir/zeoctl << ZEOCTL_EOF
#!/bin/sh
#--------------------------------------------------------------------------=
----
# Control the ZEO storage server
#--------------------------------------------------------------------------=
----
zeo_port=3D8001
python=3D$bindir/python
zeo_start=3D$optdir/Zope/lib/python/ZEO/start.py
INSTANCE_HOME=3D$vardir/storage
export INSTANCE_HOME
pidfile=3D\$INSTANCE_HOME/var/ZEO_SERVER.pid
start_zeo () {
echo Starting ZEO storage server
echo \$python \$zeo_start -p \$zeo_port "\$@"
\$python \$zeo_start -p \$zeo_port "\$@"
}
stop_zeo () {
echo Stopping ZEO storage server
pid=3D\`cat \$pidfile\`
echo kill "\$@" \$pid
kill "\$@" \$pid
}
case "\$1" in
start)
shift
start_zeo $*
;;
stop)
shift
stop_zeo $*
;;
restart)
stop_zeo
start_zeo
;;
*)
echo "zeoctl [ start | stop | restart ]"
esac
ZEOCTL_EOF
$chmodx $bindir/zeoctl
$cat > $bindir/zopectl << ZOPECTL_EOF
#!/bin/sh
#--------------------------------------------------------------------------=
----
# Control Zope application storage server
#--------------------------------------------------------------------------=
----
www_port=3D8080
python=3D$bindir/python
zope_home=3D$optdir/Zope
zope_start=3D\$zope_home/z2.py
INSTANCE_HOME=3D$vardir/zope
SOFTWARE_HOME=3D\$zope_home/lib/python
PYTHONPATH=3D\$SOFTWARE_HOME:\$PYTHONPATH
export INSTANCE_HOME SOFTWARE_HOME PYTHONPATH
pidfile=3D\$INSTANCE_HOME/var/Z2.pid
start_zope () {
echo Starting Zope application server
echo \$python \$zope_start -X -w \$www_port "\$@"
\$python \$zope_start -X -w \$www_port "\$@"
}
stop_zope () {
echo Stopping Zope application server
pid=3D\`cat \$pidfile\`
echo kill "\$@" \$pid
kill "\$@" \$pid
}
case "\$1" in
start)
shift
start_zope $*
;;
stop)
shift
stop_zope $*
;;
restart)
stop_zope
start_zope
;;
*)
echo "zopectl [ start | stop | restart ]"
esac
ZOPECTL_EOF
$chmodx $bindir/zopectl
fi
#--------------------------------------------------------------------------=
----
# Start the storage and the appserver
#--------------------------------------------------------------------------=
----
cd $sandbox
$bindir/zeoctl start
$bindir/zopectl start
--=-Q1pfU8ZVR6o9/sJvAlzW--