[Zope-dev] Re: ploneout - Or how using zc.buildout for a common
Zope2 project might look like
jim at zope.com
Fri Jan 26 08:28:09 EST 2007
Ian Bicking wrote:
> Jim Fulton wrote:
>>> I actually tried to do this once before with zc.buildout, but I didn't
>>> get far -- probably a result of lack of effort and lack of familiarity
>>> with the overall stack. But I also recognize lots of the questions
>>> about stuff like the zope.conf file and Data.fs that still seem
>> Certainly when you tried this, buildout was very young and we hadn't
>> written recipes to deal with these issues. We've made a lot of
>> progress since then.
> Well, the last time I really used it was early December, and it still
> felt slow and awkward to me at the time, with several funny quirks.
Hm, It's a bit hard to respond to "awkward" and "quirks". I'll
respond to performance issues a bit below.
>>> And frankly I like easy_install. It's
>>> probably 10x faster than buildout.
>> I doubt that that is true now. Although that probably depends on what
>> you are doing. Early versions of buildout did a lot of things
>> inefficiently as I was still learning setuptools. Because of the way
>> that buildout caches index information, I expect that creating a
>> buildout from scratch that used a lot of eggs would be much faster
>> than using easy_install. One difference though is that buildout
>> checks for the most recent compatible versions of all of the eggs it's
>> using every time you run it, whereas, as I understand it, with
>> workingenv, you'd just run easy_install manually when you want a new
> Correct. The basic process with workingenv is:
> 1. Set it up.
> 2. Start installing stuff.
> 3. Try running stuff.
> 4. Realize you got it wrong, missed something, want to do more
> development, return to 2.
> I actually find myself doing the 2-4 loop pretty often, both in
> development and when first deploying something. Just the amount of time
> to do "bin/buildout -h" was substantial (though I don't really
> understand why, except that buildout seemed to be working way too hard
> to update itself).
Ah yes. This is a good point. By default, buildout checks for newer versions
of distributions for which there are open-ended requirements. This can take
frustratingly long -- especially because pypi is so darn slow.
One advantage of buildout over easy_install (and I assume workingenv)
is that the eggs you get are deterministic by default. They are always
the newest versions that satisfy your requirements. With easy_install,
you get the most recently installed eggs that satisfy your requirements.
This means that the eggs you have depend a lot on when you installed them.
To achieve this, buildout looks for newer distributions when a requirement
doesn't have an upper bound or when the upper bound isn't satisfied by an
I really should add a quick mode that skips looking for newer versions
of requirements are met by what's already installed. This would make
the iterative style you describe go much faster. I would certainly
appreciate this myself. I will do this soon.
>> You can bypass the checks by running in offline mode. Then buildout
>> runs very fast. Because of the ability to share eggs accross
>> buildouts, it is often possible to run a buidout using lots of eggs in
>> offline mode.
>> It has been suggested that there should be a mode for buildout that
>> only talks to the network when there isn't a local egg that satisfied
>> a requirement. This would make buildout work more like workingenv
>> when few if any eggs are actually needed.
> Yes; more like easy_install does as well, actually. Though the way
> easy_install works is hardly intuitive; I find myself frequently saying
> "yes, you installed it, but did you -U install it?"
In particular, upgrading a distribution doesn't upgrade it's dependencies.
This makes it harder to control which distributions are used in an environment.
With easy_install, even through distributions are automatically included
by virtue of being dependencies, they aren't automatically updated.
There's no way to say "I want the most recent version of everything".
I wanted to make it easier to get the most recent version of the distributions
used, which is why buildout has a different policy for looking up distributions.
>>> Plus buildout's desire to own everything and
>>> destroy everything it does not own ;)
>> I'm not aware that it destroys anything. Could you be more specific?
> Well, it owns parts, and the recipes control that. Doesn't it also
> delete and reinstall there?
Yes. Buildout tried to make a buildout reflect it's specification.
This is an important feature. It uninstalls as well as installs.
But it isn't controlling anything it wasn't asked to control.
> How it treats each area of the buildout I'm
I can't help that. I've documented how this works in great detail.
> Simply making the file layout a bit more conventional, and
> describing anything non-obvious, would make buildout feel a lot more
> comfortable to the new user.
What is conventional? Python uses different layouts on different systems.
The Unix layout and Windows layout are quite different. When I came up
with the layout for Zope installations, I tried to mimic the layout
that Python used on Unix systems at the time, and then that layout
changed. We were stuck with lib/python even though we never had
anything else in lib.
I chose a shallow layout in buildout following "flat is better
>>> * As a result buildout supports multiple things in the same buildout
>>> that have conflicting version requirements, but where the packages
>>> themselves don't realize this (but the deployer does). If the packages
>>> know their requirements then setuptools' native machinery allows things
>>> to work fine.
>> Yes. I expect that usually, packages won't be very specific. The
>> buildout configuration file provides a place to be specific.
> workingenv allows this, insofar as you can be specific while installing
> things, and with the requirements file. But it doesn't make the
> individual scripts very specific, if for instance appfoo requires
> libX>1.0, and appbar requires libX>1.1, but you actually want appfoo to
> use libX==1.0 and appbar to use libX==1.1 and install them in the same
> buildout. That's the only case where buildout seems to be able to
> express something workingenv can't.
In practice, this can be very important. At least for us at ZC.
>>> * Some see bin/activate as a jail. Both workingenv and buildout are
>>> deliberately jail-like. Both Jim and I loathe the non-repeatability of
>>> system-wide installations (at least I think I can speak for him on that
>>> one point ;). bin/activate lets you into that jail, and lets you work
>>> there. There is no way into a buildout.
>> I'm not familiar with bin/activate, but it sounds like an interpreter
>> script created with buildout.
> It's created by workingenv, and you have to source it because basically
> its only function is to add the workingenv/lib/pythonX.Y to $PYTHONPATH.
> Adding that path to $PYTHONPATH is the only thing that really
> "activates" a workingenv.
Ah, so it modifies the user's environment. I think that's a reasonable
approach, although not one that I care for myself. To each his own.
The important things here, IMO, is that both activate and buildout interpreter
scripts let you get a Python interactive session or run scripts with a controlled
>>> Neither can be entirely
>>> compatible with a system-wide Python installation, because Python's
>>> standard site.py f**ks up the environment really early in the process,
>>> and avoiding that isn't all that easy.
>> This reminds me of a place where buildout is looser than workenv.
>> buildout doesn;t try to disable anything in the system python. It
>> just augments it. I always use a clean python, so avoiding
>> customizations in the Python I use isn't a problem. If I wanted to
>> take advantage of something in a system Python, as I occasionally do,
>> I can do that with buildout.
> I find the isolation useful when testing things for release;
Yup. I do the same things by always using clean Python installs.
I *never* use a system Python for development. I always use a
Python that I build myself from sources.
Your approach is certainly a valid alternative.
> I can be
> sure that I haven't been using any packages that I don't explicitly
> include in the egg requirements or instructions.
> But it can be annoying
> in other cases, like when there's a library that doesn't install cleanly
> (of which there's still quite a few).
> Anyway, if you do want to include
> the global packages, --site-packages will change your workingenv to do so.
> It could be argued that workingenv's default should be to include
> site-packages. Another option would be to have a tool that allows you
> to easily include something from the system Python (probably just a tool
> to manage a custom .pth file, which works even when setuptools' fairly
> heroic attempts to fix broken setup.py's doesn't work).
<shrug> My remark was not meant to criticize or to suggest a different
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
More information about the Zope-Dev