[Zope3-dev] RFC: Known working sets

[Philipp von Weitershausen]

Problem
-------

We recently had our first Grok release (0.10). Due to various versions 
of certain packages, we had to pin down the versions of those packages 
to ensure you'd get the version we had tested the release with, and to 
prevent buildout from downloading the versions of packages that had no 
Windows binaries. The pinning had to be done in setup.py to ensure all 
Grok installations out there would be on the same page.

Pinning versions in setup.py has major disadvantages, though:

* It's easy to get a version conflict (e.g. you depend on X==1.0 and Y, 
but Y depends on X==1.0.1)

* It's pretty much impossible to opt for a newer version of a certain 
package (e.g. in order to fix a security problem).


Fortunately, zc.buildout allows us to keep setup.py free of versions and 
pin the versions in buildout.cfg instead (see [1] for an extensive 
example). This works for us experienced developers. This doesn't work 
for the majority of the consumers of our packages, though. For instance, 
we can't expect everybody out there to copy and paste 100 lines of stuff 
into their buildout.cfg when we release a new Grok version. And I'm not 
even talking about a major or minor release. It would just as well need 
to happen on every bugfix release.

Grok is just one example where an efficient way of defining, managing 
and distributing the information on a "known working set" is needed. As 
we're phasing out the Zope 3 tarball, I'm sure Zope 3 developers who 
don't keep a close watch on the individual eggs would appreciate if they 
had a known working set to depend on. The Zope 2 project, for example, 
is a large consumer of the Zope 3 packages. Other projects such as Plone 
have the same need for defining known working sets as they're embracing 
eggs more and more.


Goals
-----

Here's what I'd expect from a known working set:

* It pins packages to certain versions. This information is made 
available in a way that buildout and/or setuptools can work with it. 
This has to be easy. Copy and pasting large sections of buildout.cfg is 
not what I'd consider easy.

* If nothing else is provided, buildout and/or setuptools will install 
the versions specified in the known working set. It should be possible, 
however, to deviate from this locally and install different versions of 
certain packages.

* However the known working set may be defined, it needs to be 
discoverable and manageable in subversion. It should be possible to give 
it a version tag.


Proposal
--------

I think zc.buildout's version mechanism in buildout.cfg is a good 
technical basis. It's simple, light-weight and easy to version-control. 
For example, the Grok project could create a grok-0.11.cfg defining the 
known working set for the Grok 0.11 release, with the following contents::

   [grok-0.11]
   grok = 0.11
   ZODB = 3.8.0
   zope.component = 3.4.0
   ...

Then, the Grok folks could distribute this file and everybody who wanted 
to use Grok 0.11 would simply have to put the following two lines into 
their buildout.cfg::

   [buildout]
   ...
   extends = grok-0.11.cfg
   versions = grok-0.11

With this, it'll be easily possible to switch to a newer version of e.g. 
ZODB3 by overriding the setting in buildout.cfg::

   [buildout]
   ...
   extends = grok-0.11.cfg
   versions = grok-0.11

   [grok-0.11]
   ZODB = 3.9.0  # this overrides whatever is set in grok-0.11.cfg


The only problem is that distributing grok-0.11.cfg is a bit tedious. 
How about if buildout could get it from the web?

   [buildout]
   ...
   extends = http://grok.zope.org/releases/grok-0.11.cfg
   versions = grok-0.11

(alternatively, you could point it to 
http://svn.zope.org/*checkout*/grok/releases/grok-0.11.cfg to get it 
directly from SVN.)

So now if you wanted to switch to a newer version of Grok, all you would 
have to do is point to another known working set in your buildout.cfg 
(no tedious download and redistribution of a new file involved), e.g.::

   [buildout]
   ...
   extends = http://grok.zope.org/releases/grok-0.12.cfg
   versions = grok-0.12


Deliverables
------------

* zc.buildout's extends mechanism would have to be enhanced to be able 
to load files from HTTP locations.

* A short and light-weight process would have to be written down to 
document how we define known good sets, where they're put in subversion 
and how they're managed.


Thoughts?


[1] http://svn.zope.org/*checkout*/zope.testbrowser/trunk/buildout.cfg

-- 
http://worldcookery.com -- Professional Zope documentation and training