[Zope3-dev] Re: More thoughts on packaging

[Tres Seaver]

Phillip J. Eby wrote:
> At 12:36 AM 2/18/04 +1100, Anthony Baxter wrote:
> 
>> Jim Fulton wrote:
>>
>>> A major problem I have with distutils is that I have to write rather
>>> complex scripts to do somewhat simple things.  I don't
>>> necessarily have a problem with capturing this meta data in Python.
>>
>>
>> Ok, then, as a step forward - perhaps someone who's familiar with the
>> current Z3 setup.py could do a first cut of what their new setup.py
>> would look like, assuming the backend magically appears. This at least 
>> allows those of us with some knowledge of distutils innards (my eyes, 
>> my eyes!) to get an idea of what the solution might entail.
>>
>> I'm not saying "come up with the perfect answer" but instead saying 
>> "if distutils was implemented in a way that your current needs were 
>> met, what would the setup.py say?"
> 
> 
> Speaking as another large package maintainer, I'd like setup.py to be 
> either nonexistent, or automatically generated from something else 
> that's modular.

+1.  The configuration metadata in a typical setup.py can *never* be 
introspected, for intsance, short of writing bytecode hacks to pick 
apart the arguments passed to setup().

>> I know, for instance, that in my perfect world, there wouldn't be little
>> dinky text files (or xml crud) required - this stuff _can_ be 
>> expressed in Python, and I'd prefer that it is.
> 
> I'm certainly -1 on using XML for this, and I don't have a problem with 
> using Python syntax.  But the single file approach has got to go.

-1 for Python syntax;  Turing-completeness is a *disadvantage* in a 
configuration language (e.g., why is ZCML not in Python?  why ZConfig?)


WRT the format, I am agnostic about the spelling, but anti-NIH suggests 
that we should adopt a widely-supported parseable format:

   - ConfigParser style INI files (order can't matter, some restrictions
     on spelling section and key names).

   - An XML dialect of some sort (we should be looking to how Debian and
     RPM are spelling the XML variants of their package specification
     files).

> The problem with putting everything in a setup.py is that it's not 
> reusable.  Modularity is what's needed for large systems like Zope and 
> PEAK.  For example, right now I bundle PyProtocols with the PEAK 
> distribution.  This means that I have to maintain two setup.py files 
> that cover some of the same stuff: one for the independent PyProtocols, 
> and one for PEAK that contains PyProtocols setup info.  That's not too 
> big a deal right now, but mainly what it does is discourage me from 
> making other parts of PEAK separately distributable, because all the 
> other parts are much harder to break out, and the dependencies between 
> the parts change over time.
> 
> Yes, I know you can create a convention for separating the information 
> into multiple files, using execfile or import.  But that convention's 
> not a standard, so you can't really bundle somebody else's setup into 
> your own to make a larger distribution.
> 
> So, if the solution here is to create such a convention, it needs to be 
> PEPed so that folks will adopt it.

Or not.  The Python community has not always been amenable to adopting 
stuff which large-system folks find invaluable.


>  The existing 'setup.py' mechanism 
> creates a tension that usually resolves in favor of distributing one 
> mega-package instead of lots of little packages.  So communities that 
> develop a lot of software (e.g. Zope, PEAK, Twisted, and others) tend to 
> release those systems as humongous "sumo distributions", instead of 
> thinking more modularly.  And they tend to duplicate effort because it's 
> too bloody much trouble to distribute something from somebody else.
> 
> Please understand that I'm not bitching about distutils.  I *love* the 
> distutils, because it's so much better than not having the distutils.  
> I'm in this conversation to find solutions to take things to the next 
> level.  First, the modular assembly of packages so that people doing 
> "sumo distributions" can break them up a little, and encourage people 
> writing smaller packages to use an approach that will let their packages 
> be bundled in larger packages.  (It'd be nice, for example, if you could 
> run something like 'setup.py generate' and have an existing 'setup.py' 
> spit out data for incorporation into a larger package, to make 
> redistribution of "legacy" packages easier, and to make migration to the 
> modular approach easier.)  Then, dependency information to automate 
> assembly of already-present packages (e.g. slices of Zope or PEAK).
> 
> Anyway, I guess what I'm saying here is that the convention of using a 
> monolithic setup.py encourages monolithic packaging, and discourages 
> CPAN-style packaging.

Exactly.

> It might seem I'm contradicting myself here, in that I'm advocating 
> making *more* distributions, incorporating packages from multiple 
> sources, which isn't the CPAN way either.  But it's a stepping stone.  
> Until we have a reliable CPAN-like mechanism to download and install 
> dependencies, the gap will be filled by motivated packagers (like me and 
> the Zope community) who want to provide their users with a single 
> download for some targeted set of functionality.  And if it's *easy* to 
> do, then other people will create bundles of the stuff they need in 
> their environment, and maybe offer them for others to download.  So, 
> it's a stopgap measure, but we have to evolve our way to CPAN, which 
> means having survivable options at each point along the way, that are 
> more functional than the previous step.

Agreed.

Tres.
-- 
===============================================================
Tres Seaver                                tseaver at zope.com
Zope Corporation      "Zope Dealers"       http://www.zope.com