[Zope3-dev] More thoughts on packaging

[Phillip J. Eby]

At 12:36 AM 2/18/04 +1100, Anthony Baxter wrote:
>Jim Fulton wrote:
>>A major problem I have with distutils is that I have to write rather
>>complex scripts to do somewhat simple things.  I don't
>>necessarily have a problem with capturing this meta data in Python.
>
>Ok, then, as a step forward - perhaps someone who's familiar with the
>current Z3 setup.py could do a first cut of what their new setup.py
>would look like, assuming the backend magically appears. This at least 
>allows those of us with some knowledge of distutils innards (my eyes, my 
>eyes!) to get an idea of what the solution might entail.
>
>I'm not saying "come up with the perfect answer" but instead saying "if 
>distutils was implemented in a way that your current needs were met, what 
>would the setup.py say?"

Speaking as another large package maintainer, I'd like setup.py to be 
either nonexistent, or automatically generated from something else that's 
modular.


>I know, for instance, that in my perfect world, there wouldn't be little
>dinky text files (or xml crud) required - this stuff _can_ be expressed in 
>Python, and I'd prefer that it is.

I'm certainly -1 on using XML for this, and I don't have a problem with 
using Python syntax.  But the single file approach has got to go.

The problem with putting everything in a setup.py is that it's not 
reusable.  Modularity is what's needed for large systems like Zope and 
PEAK.  For example, right now I bundle PyProtocols with the PEAK 
distribution.  This means that I have to maintain two setup.py files that 
cover some of the same stuff: one for the independent PyProtocols, and one 
for PEAK that contains PyProtocols setup info.  That's not too big a deal 
right now, but mainly what it does is discourage me from making other parts 
of PEAK separately distributable, because all the other parts are much 
harder to break out, and the dependencies between the parts change over time.

Yes, I know you can create a convention for separating the information into 
multiple files, using execfile or import.  But that convention's not a 
standard, so you can't really bundle somebody else's setup into your own to 
make a larger distribution.

So, if the solution here is to create such a convention, it needs to be 
PEPed so that folks will adopt it.  The existing 'setup.py' mechanism 
creates a tension that usually resolves in favor of distributing one 
mega-package instead of lots of little packages.  So communities that 
develop a lot of software (e.g. Zope, PEAK, Twisted, and others) tend to 
release those systems as humongous "sumo distributions", instead of 
thinking more modularly.  And they tend to duplicate effort because it's 
too bloody much trouble to distribute something from somebody else.

Please understand that I'm not bitching about distutils.  I *love* the 
distutils, because it's so much better than not having the distutils.  I'm 
in this conversation to find solutions to take things to the next 
level.  First, the modular assembly of packages so that people doing "sumo 
distributions" can break them up a little, and encourage people writing 
smaller packages to use an approach that will let their packages be bundled 
in larger packages.  (It'd be nice, for example, if you could run something 
like 'setup.py generate' and have an existing 'setup.py' spit out data for 
incorporation into a larger package, to make redistribution of "legacy" 
packages easier, and to make migration to the modular approach 
easier.)  Then, dependency information to automate assembly of 
already-present packages (e.g. slices of Zope or PEAK).

Anyway, I guess what I'm saying here is that the convention of using a 
monolithic setup.py encourages monolithic packaging, and discourages 
CPAN-style packaging.

It might seem I'm contradicting myself here, in that I'm advocating making 
*more* distributions, incorporating packages from multiple sources, which 
isn't the CPAN way either.  But it's a stepping stone.  Until we have a 
reliable CPAN-like mechanism to download and install dependencies, the gap 
will be filled by motivated packagers (like me and the Zope community) who 
want to provide their users with a single download for some targeted set of 
functionality.  And if it's *easy* to do, then other people will create 
bundles of the stuff they need in their environment, and maybe offer them 
for others to download.  So, it's a stopgap measure, but we have to evolve 
our way to CPAN, which means having survivable options at each point along 
the way, that are more functional than the previous step.