[Zope3-dev] Re: Packaging pre-proposal/notes

[Philipp von Weitershausen]

Jim,

I agree with most of the goals and assumptions you have listed. So 
assume a +1 from me for everything I've *not* commented below.

 >       - Support fine-grained software distribution packaging.
 >
 >         Software distribution packages will often correspond directly
 >         to Python packages that are rather small. For example, a small
 >         software distribution package might include a single Python
 >         package with only a few source files.

I think we should think of this as being the usual case.

 >       - Work with distutils.  We don't want to reinvent the wheel.
 >         OTOH, we may have to work with the rest of the Python
 >         community to improve the wheel we have to provide or support
 >         package management. The most imporatnt need being dependency
 >         management.  See PEP 262.

Distutils so far have been things magical thing to me, most probably 
because package metadata is provided within the setup.py script. When 
we, err, you guys invented ZCML, you didn't take Python as the 
configuration language for a good reason. I think the same reason 
applies to package installation through distutils. I would really rather 
like to see package metadata being provided in a parseable text file...

I don't think it's too late now. Distutils lacks the ability to express 
package dependencies, so we need to come up with a solution for that 
anyway. PEP 262 suggests to use the following in a setup.py file::

   setup(..., requires=['xml.utils.iso8601', ...)

I don't think we should have a setup.py file in every source code 
package just to express package dependency... Like I've said before and 
was agreed by some people, I would prefer a human- and machine-readable 
text format. The distribution packaging process would then receive a 
list of packages that a certain distribution should include. It would 
resolve all the dependencies and create a distribution containing the 
desired packages including their dependencies. While being at it, that 
process could also read a list of binary modules to be compiled, a list 
of to-be-installed complementary data files, etc. from that text-file.

In the end, I imagine the process to look like this (pseudo code)::

   to_be_installed = []  # command line argument or similar
   all_packages = resolveDependencies(to_be_installed)
   setup(... all_packages ...)

 >     Non Goal
 >
 >       - An important non-goal is Zope instance creation.  The software
 >         packaging mechanism will be responsible for installing
 >         software. It will generally not be responsible for setting up
 >         Zope instances. Rather, among the software installed will be
 >         software for creating Zope instances.

I agree with that being a non-goal. It should be easy for distribution 
makers to create custom instances after the software itself has been 
installed. I think the Plone people are already doing it with their 
Plone Controller... Eventually, we might provide a few hooks, though. 
But we indeed should not worry about that right now.

 >         We should strive to have needed distutils improvements ready
 >         for inclusion in Python 2.4.
...
 >       We use the term "package" to refer *both* to Python packages and
 >       to units of software distribution.  There will often, but not
 >       always be a one-to-one correspondence between software
 >       distribution packages and Python packages.   Distutils attempts
 >       to solve this by refering to software distributions as "module
 >       distributions". I (Jim) find this rather unsatisfying.  This is
 >       an issue that will, ultimately, need to be resolved by the
 >       Python community. We will follow thier lead.
 >
 >     Possible vision
 >
 >       - Work with the Python community at large to extend distutils
 >         to manage information necessary for packaging and create a
 >         simple package-management system for python.

I see why you would like the Python community to help solve this 
problem. I hope you do realize though that we will depend on them in 
this issue. Now, it might be an advantage for you having the majority of 
PythonLabs right next door to your office; but that doesn't change the 
fact that we're going to involve a much bigger community with different 
ideas, etc.

The inevitable result is that this process will take a long time until 
the ideas are set, the infrastructure is in place and stable software is 
produces. We might make it for Python 2.4, but only if we're lucky.

My focus of interest has been the Zope3 package hierarchy. The 
discussion on package and dependency management was kind of started by 
this issue in the first place. So they are connected. But I think the 
links between them are limited, and we should keep it that way. 
Obviously, the package and dependency management needs certain 
requirements to be fullfilled by the Zope3 package hierarchy, which you 
have mentioned:

 >       - Liberate the Zope repository from having to reflect
 >         distribution or packages status (e.g. core or not core.)

 >       - Provide a simpler repository layout to make Zope 3 software
 >         easier to understand and navigate.

 >         Note that source and binary distributions may not look
 >         anything like the CVS repository.

The fact that the packaging/distutils issue will take long time, 
shouldn't stop us from rearranging the Zope3 package hierarchy, though. 
I think we *should* come up with a proposal for that soon, and I think 
we should implement it a.s.a.p. I see the packaging/distutils issue 
decoupled from the package hierarchy issue in the sense that we should 
still go ahead with the latter without being stopped by the former, and 
we should only keep in mind the points mentioned by you above.

We also will have to face the fact that we might not get the 
packaging/dependency management system until after Zope X3.0 has been 
released.

 >       We are aware of a number of issues using distutils to package
 >       Zope software. These are collected here so we can seek to
 >       address them in this project:
 >
 >       - Lack of dependency data needed to support a packaging
 >         system. See PEP 262.

I don't find PEP 262 useful at all for this issue. Other people have 
pointed their fingers towards Linux distribution package managers and 
their format of package dependency description. I think a package file 
format similar to RedHat's dst files, the output of Debian's apt-cache 
show or the listings provided by Gentoo users are good templates to work 
with.

Philipp