[Zope3-dev] More thoughts on packaging

[Phillip J. Eby]

At 05:18 PM 2/13/04 -0500, Jim Fulton wrote:
>Phillip J. Eby wrote:
>>At 01:30 PM 2/13/04 -0500, Jim Fulton wrote:
>>
>>>There seems to be a possible consensus emerging that we should
>>>use a packaging system.  I suggest that we should use an existing
>>>system (or systems) such as RPM or the debian system and
>>>provide the ability to generate course-grained distributions
>>>from these systems for those systems (e.g. windows) that either
>>>don't have package systems or don't have package systems that
>>>we support.
>>
>>I haven't been following the packaging discussion that closely, so I may 
>>be projecting the following from a different set of requirements in mind 
>>than what you have.  So, apologies in advance if this is completely 
>>off-base from where you're heading...
>
>I'd say that you are on target.
>
>>Suppose that we put dependency metadata in package directories as an 
>>extra file, and then we wrote a 'package.py' script that generates a 
>>custom 'setup.py' from the metadata, based on an intended target.  E.g. 
>>'package.py sometarget'.
>>Advantages:
>>* Leverages distutils packaging system support, since you can build 
>>Windows .exes, RPMs, source tarballs, etc.
>>* Takes advantage of any future distutils support for additional 
>>packaging systems
>>* Keeps the metadata lightweight and focused on Zope-specific needs
>>* Doesn't require anyone to learn another "full" packaging system
>
>Yup.
>
>I thnink that dependencies are one central issue that is going
>to be critical if we do fine-grained  packaging.
>
>>Disadvantages:
>>* Has to be written (but is relatively lightweight)
>>
>>The metadata could actually be provided in the form of a Python module 
>>(e.g. '__setup__.py')  that defines any extensions, datafiles, etc., as 
>>well as "provides" and "requires" keys.
>
>"provides" and "requires"?  I assume that "requires" expresses the
>dependencies.  What is "provides" for?

That's if we say that a bundle may offer one or more "features" that other 
bundles require.  For example, the 'zope.interface' package might "provide" 
the "feature" 'zope.interface v1.03'.  Another package might specify that 
it requires 'zope.interface v1.05', and thus won't work with the other 
package.

It might be more useful to have make versions more explicit, instead.  My 
draft of the concept was borrowed from certain 'init' implementations that 
do dependency resolution.  A given script might say that it depends on the 
'http' service, but the 'apache' script says that it "provides" 'http' and 
'https'.

Anyway, feel free to simplify/discard/whatever; I just was throwing this 
out as an idea of a possible general direction to take.  I'm not really 
particular about the details at the moment.  :)


>>The 'package.py' script would write a 'setup.py' script that would look 
>>something like:
>>execfile('setupstuff/prologue.py')
>>gatherSetup('src/some/pkg1/__setup__.py')
>>gatherSetup('src/other/something/__setup__.py')
>
>why 2 gather setups? Is this for a distribution package that
>includes 2 Python packages?

Or maybe 2 "bundles", where there is some mapping between bundles and 
packages.  I'm being somewhat vague and handwavy here.  Also, Fred 
suggested replacing all that with one function call and a list of 
packages/bundles/directories/whatever, which certainly works too.



>># ... etc, etc.
>>execfile('setupstuff/epilogue.py')
>>Based on sorting out the dependencies.  'epilogue' would then call the 
>>distutils 'setup' function with the accumulated data.
>
>How does it sort out the dependencies?

By looking at the metadata, and creating a list of all the setup metadata 
snippets that would constitute a distribution with all the dependencies 
satisfied.


>   Does it retrieve other packages?
>Does it assume that they are around and just need to be installed?

The mental model I have in mind is that you have a "complete" source 
distribution from which you are "packaging" things.  So, let's say I wanted 
to make a "mini-Me" Zope distro that had only what was needed to support 
'zope.publisher'.  I would check-out the entire Zope X3 source tree, and 
run something like 'package.py zope.publisher' (?) and then run setup.py to 
create source and binary distributions.

This is the aspect I thought might be off-base from where you're 
headed.  My presumption here was not to make it possible for an end-user to 
wave a magic wand and download/install dependencies, as that seems like a 
future distutils function.  Instead, I was targeting the ability to *make* 
distribution packages, if one had the source of the things one wanted to 
package.

Of course, it *would* be pretty cool if it could download dependencies.  If 
package metadata files (and their dependencies) were accessed/referenced by 
URLs, and there were standardized formats/methods/locations for 
downloading...  Hm.  But then there are issues with things like the running 
of untrusted code, etc. etc.  I think it'd be better to require the user to 
have everything needed already unpacked on the local machine, in a 
known/standard directory structure.  The 'package.py' program will have to 
tell you about missing dependencies anyway, so you'll at least get a list 
of things you need to download.  Anyway, this keeps the scope of the effort 
from exploding into "rewrite distutils" territory.