[Zope3-dev] Spaghetti-hunting

[Barry A. Warsaw]

It's nearly 2am so what better thing to do than clear out old email?
:)  This is an old thread, but one I didn't want to just let die
without further comment.

    [Barry A. Warsaw wrote:]
    >> I think this approach opens the possibility for much more code
    >> reuse of Python applications, by integrating into the standard
    >> Python extension mechanisms.  For the Mailman situation, I
    >> could easily see people writing their own web gui's or other
    >> applications on top of the Mailman package (it's been talked
    >> about in the MM discussion lists).

    >> For Zope-derived software, I can easily see how other Python
    >> applications might like to use TAL, PageTemplate, etc.  These
    >> should probably be top-level package names, and not be in the
    >> Zope.* namespace.  Then, you could distutils install them and
    >> any Python app could just use "import TAL".  Thoughts?

>>>>> "SH" == Shane Hathaway <shane@zope.com> writes:

    SH> Well, that's what Zope 2 did.  You can "import TAL" and it
    SH> works.  The problem is that there are so many packages that
    SH> you really don't want to install them all in
    SH> site-packages--they would cause too much clutter. And you
    SH> wouldn't want to install only some of them in site-packages,
    SH> since you want to upgrade all Zope packages together.  For
    SH> Zope 2, it's usually easier to just change PYTHONPATH or
    SH> sys.path if you want to use Zope packages outside Zope.

I have a slightly different take on this, but I'm not sure how
relevant it is to Zope3.  Let's start by agreeing that a Zope install
should /not/ install a ton of packages into site-packages.  I'll
explain the Mailman approach since that's what I'm intimately familar
with. ;)

In Mailman, I have three distutils'd packages that have an independent
development cycle: two codecs and the email package.  Now, all three
can be installed by Joe I. Notamailman using normal distutils
procedures, and they do indeed go in site-packages.  This is actually
what you want because when distributing those packages as separate
pieces, usable outside of any Mailman context, you want them
accessible to all Python applications.

Now, I distribute the .tar.gz files for those packages created with
"python setup.py sdist", and I use "tar zxf ; python setup.py install"
to install them.  However I use the --install-lib option[1] to put the
packages, not in site-packages, but in some Mailman-specific
subdirectory (I call it $prefix/pythonlib where $prefix is the
standard configure script install location).  The key is that I
arrange for all Mailman entry points, be it cgi, mail program, cron,
or command line script, to put $prefix/pythonlib first on sys.path.

That way, even if you've installed email package 1.1 in site-packages
for use in all your other Python 2.1 programs, I am guaranteed to get
email 2.0.2 when I do an "import email".

This is (IMO) an elegant way to allow those packages to have their own
independent release cycle, be usable in non-related Python
applications (promoting re-use), but still allow my application to be
assured it will get the right version of the package.

The one tricky bit is making sure $prefix/pythonlib is always first on
sys.path.  For my cgi's it's easy because I have a C wrapper that
manipulates the environment before calling the Python cgi script (so I
can twiddle $PYTHONPATH to my heart's content).  Similarly for mail
programs.

For cron and command line scripts, I have a file called paths.py which
contains the sys.path twiddling logic.  As part of the configure
process, I actually take a single paths.py.in template, fill in the
right variables, and copy the resulting paths.py file to my bin/ and
cron/ directories.  The first thing any cron or command line script
must do before it imports a non stdlib library is "import paths" which
twiddles sys.path as a side-effect.  Because there's always a paths.py
file in the directory of the script, I know that it will always get
properly imported, regardless of the path used to reach the script.

The other issues are that I do have separate configure and "make
install" steps, and that Mailman cannot be run out of its source
directory.  The first shouldn't be foreign to Zope; what Mailman
spells "configure", Zope2 spells "python wo_pcgi.py" <wink>.  The
second is surmountable I believe -- I think if I had as many
co-developers as Zope had, I could make Mailman runnable out of its
source directory, which I agree is a very convenient thing.  For now,
I just deal with it.

All-in-all the whole thing hangs together very nicely.

sleepi-ly y'rs,
-Barry

[1] To worm around a bug, I also have to add the --install-purelib
option, otherwise the japanese.pth file will get erroneously installed
to site-packages, ignoring the --install-lib option.