[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like

[Ian Bicking]

(sorry if this ends up going out twice, I'm trying to figure out this 
gmane thing and apparently I hadn't figured this out when I first sent it)

Whit pointed me to this thread.  I won't reply to specifics here, but 
maybe just describe what we're doing (and planning to do), and how 
workingenv differs from zc.buildout.

So... recently we (The Open Planning Project -- more specifically Rob 
Miller) decided we need a better way to deploy our specific stack.  It 
was hard, we couldn't tell other people how to do it, and we spent a 
significant amount of time debugging each other's installations, or 
chasing down bugs that we ultimately realized were just because of 
slightly different installations.  I'm sure you all understand the 
motivation; it's similar to what buildout is for, though we were more 
focused on repeatability as a development tool than as a deployment tool.

I actually tried to do this once before with zc.buildout, but I didn't 
get far -- probably a result of lack of effort and lack of familiarity 
with the overall stack.  But I also recognize lots of the questions 
about stuff like the zope.conf file and Data.fs that still seem 
unresolved.  The thing that frustrated me with zc.buildout is that I 
knew how to do what I wanted to do, but I still felt like I was a long 
way from being able to do it.  And little things, like unhelpful error 
messages and frustratingly slow performance really killed my motivation. 
  After setting that project aside someone else at TOPP (Luke Tucker) 
did a buildout for Deliverance because we needed to build some 
non-Python libraries and that was a feature of buildout; that did end up 
working eventually (after considerable effort), but it was not a very 
satisfying experience, and *using* the buildout was itself a real 
challenge.  Since Deliverance is just a library, to do anything useful I 
also had to install another package that *uses* that library, and that 
was surprisingly difficult.  Actually developing those libraries was 
even more frustrating.

So then Rob decides to devote some time to deployment.  And because Rob 
just wants to get this finished, he wrote the whole things from scratch. 
  You can see the result of that here: 
https://svn.openplans.org/svn/topp.deploy/trunk -- it's not a generic 
setup tool, just what works for us.  The end result is something that 
does everything we want, that we understand, and that we'll understand 
how to extend.  An important difference from zc.buildout is that all the 
logic and work is in *our* script, not in a framework driven by a config 
file.  In the future Rob plans to use buildit 
(http://cvs.plope.com/viewcvs/Packages/buildit/), and one of the 
motivations there is that it has handy routines to do what topp.deploy 
already does, but better.  It's more like a library and less like a 
framework.  I could make some generalization here but I'll restrain myself.

The deployment script also uses workingenv, somewhat similarly because 
it is more library-like.  Well, also because I work at TOPP and can 
easily support their use, which is certainly a nontrivial reason for the 
choice.  Workingenv in this case is basically a tool that provides the 
isolation that we need to create something repeatable.  This is the main 
feature overlap with buildout.  Workingenv is not the framework in which 
topp.deploy is written, and workingenv is not intended as a framework.

Note also that topp.deploy does not have the full set of features we'll 
ultimately need.  You can't tell it to install another egg, or setup 
another script, or whatever.  And we don't *have* to add those features, 
because workingenv is compatible with all the other tools.  Where "all 
the other" is mostly easy_install.  But someday there will be more, even 
if the progress is slow.  buildout is basically incompatible with 
easy_install (the script).  And frankly I like easy_install.  It's 
probably 10x faster than buildout.  easy_install is what people use in 
documentation, and its conventions are the ones people know (why does 
buildout not use "Pkg==version", for instance?).

As for the technical reasons they don't work together:

* workingenv allows and leaves it to setuptools to maintain the package 
installation database (basically easy-install.pth).  This is not a very 
good database, but eh.  buildout doesn't really have a database, but 
instead just enforces what buildout.cfg indicates.

* workingenv relies on that database to give default versions and to 
setup the Python path.  The fixup it does of installed scripts is fairly 
minimal, just setting up sys.path enough to force its site.py to get 
called.  buildout enumerates all the activated packages, and ignores 
easy-install.pth.  This is basically what makes it 
easy_install-incompatible.  Plus buildout's desire to own everything and 
destroy everything it does not own ;)

* As a result buildout supports multiple things in the same buildout 
that have conflicting version requirements, but where the packages 
themselves don't realize this (but the deployer does).  If the packages 
know their requirements then setuptools' native machinery allows things 
to work fine.  The solution with workingenv is to create multiple 
environments.  Since the actual building happens higher up (e.g., 
topp.deploy), there's nothing stopping you from creating multiple 
environments from one deployment script.  Anyway, in summary the way 
scripts are generated is one of the major incompatibilities between 
buildout and workingenv.  In effect Buildout's jail is too strong for 
workingenv to penetrate, and buildout doesn't tell anyone else about 
what it is doing.

* workingenv allows you to change versions without informing or using 
workingenv.  Once you've created the environment you mostly stop using 
workingenv directly.

* Some see bin/activate as a jail.  Both workingenv and buildout are 
deliberately jail-like.  Both Jim and I loathe the non-repeatability of 
system-wide installations (at least I think I can speak for him on that 
one point ;).  bin/activate lets you into that jail, and lets you work 
there.  There is no way into a buildout.  Frankly this weirds me out, 
and is a big part of my past frustration with it.  Maybe that's because 
I'm in the relatively uncommon situation that I actually know what's 
going on under the hood of Python imports and packaging, and so it 
bothers me that I can't debug things directly.  Anyway, neither requires 
activation when using scripts generated in the environment.  And 
bin/activate is really just something that sets PYTHONPATH and then does 
other non-essential things like changing the prompt and $PATH -- I 
should probably document that more clearly.  Neither can be entirely 
compatible with a system-wide Python installation, because Python's 
standard site.py f**ks up the environment really early in the process, 
and avoiding that isn't all that easy.  In some ways virtual-python is 
the more complete solution to all of this, and sometimes I think I 
should just use that technique (of a separate Python interpreter, and 
hence separate prefix) with some of the improvements I could take from 
workingenv.

Anyway, this is my very long summary of why we aren't using buildout, 
and are using workingenv.

Cheers,
   Ian