[Checkins] SVN: zc.buildout/trunk/ merge lp:~gary/zc.buildout/python-support (also available as svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/gary-launchpad). This adds support for system Pythons, as well as other changes, such as improved Distribute support and some cleanups.
Gary Poster
gary.poster at canonical.com
Thu Apr 29 14:11:59 EDT 2010
Log message for revision 111588:
merge lp:~gary/zc.buildout/python-support (also available as svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/gary-launchpad). This adds support for system Pythons, as well as other changes, such as improved Distribute support and some cleanups.
Changed:
_U zc.buildout/trunk/
A zc.buildout/trunk/.bzrignore
U zc.buildout/trunk/CHANGES.txt
U zc.buildout/trunk/README.txt
U zc.buildout/trunk/bootstrap/bootstrap.py
U zc.buildout/trunk/buildout.cfg
U zc.buildout/trunk/dev.py
U zc.buildout/trunk/setup.py
U zc.buildout/trunk/src/zc/buildout/bootstrap.txt
U zc.buildout/trunk/src/zc/buildout/buildout.py
U zc.buildout/trunk/src/zc/buildout/buildout.txt
U zc.buildout/trunk/src/zc/buildout/easy_install.py
U zc.buildout/trunk/src/zc/buildout/easy_install.txt
U zc.buildout/trunk/src/zc/buildout/testing.py
U zc.buildout/trunk/src/zc/buildout/tests.py
U zc.buildout/trunk/src/zc/buildout/testselectingpython.py
U zc.buildout/trunk/src/zc/buildout/update.txt
A zc.buildout/trunk/z3c.recipe.scripts_/
U zc.buildout/trunk/z3c.recipe.scripts_/CHANGES.txt
U zc.buildout/trunk/z3c.recipe.scripts_/README.txt
U zc.buildout/trunk/z3c.recipe.scripts_/setup.py
U zc.buildout/trunk/z3c.recipe.scripts_/src/z3c/__init__.py
U zc.buildout/trunk/z3c.recipe.scripts_/src/z3c/recipe/__init__.py
U zc.buildout/trunk/z3c.recipe.scripts_/src/z3c/recipe/scripts/README.txt
U zc.buildout/trunk/z3c.recipe.scripts_/src/z3c/recipe/scripts/__init__.py
U zc.buildout/trunk/z3c.recipe.scripts_/src/z3c/recipe/scripts/scripts.py
U zc.buildout/trunk/z3c.recipe.scripts_/src/z3c/recipe/scripts/tests.py
U zc.buildout/trunk/zc.recipe.egg_/setup.py
U zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/README.txt
U zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/api.txt
U zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/custom.txt
U zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/egg.py
U zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/selecting-python.txt
U zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/tests.py
-=-
Property changes on: zc.buildout/trunk
___________________________________________________________________
Modified: svn:mergeinfo
- /zc.buildout/trunk:108946
+ /zc.buildout/branches/gary-launchpad:111587
/zc.buildout/trunk:108946
Modified: svk:merge
- 62d5b8a3-27da-0310-9561-8e5933582275:/zc.buildout/trunk:108946
+ 62d5b8a3-27da-0310-9561-8e5933582275:/zc.buildout/branches/gary-launchpad:111587
62d5b8a3-27da-0310-9561-8e5933582275:/zc.buildout/trunk:108946
Copied: zc.buildout/trunk/.bzrignore (from rev 111587, zc.buildout/branches/gary-launchpad/.bzrignore)
===================================================================
--- zc.buildout/trunk/.bzrignore (rev 0)
+++ zc.buildout/trunk/.bzrignore 2010-04-29 18:11:58 UTC (rev 111588)
@@ -0,0 +1,9 @@
+.installed.cfg
+bin
+build
+develop-eggs
+eggs
+parts
+src/zc.buildout.egg-info
+z3c.recipe.scripts_/src/z3c.recipe.scripts.egg-info
+zc.recipe.egg_/src/zc.recipe.egg.egg-info
Modified: zc.buildout/trunk/CHANGES.txt
===================================================================
--- zc.buildout/trunk/CHANGES.txt 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/CHANGES.txt 2010-04-29 18:11:58 UTC (rev 111588)
@@ -1,10 +1,76 @@
Change History
**************
-1.4.4 (?)
-=========
+1.5.0 (201?-??-??)
+==================
+New Features:
+- Added buildout:socket-timout option so that socket timeout can be configured
+ both from command line and from config files. (gotcha)
+
+- Buildout can be safely used with a system Python (or any Python with code
+ in site-packages), as long as you use the new z3c.recipe.scripts
+ recipe to generate scripts and interpreters, rather than zc.recipe.egg.
+
+ zc.recipe.egg is still a fully supported, and simpler, way of
+ generating scripts and interpreters if you are using a "clean" Python,
+ without code installed in site-packages. It keeps its previous behavior in
+ order to provide backwards compatibility.
+
+ The z3c.recipe.scripts recipe allows you to control how you use the
+ code in site-packages. You can exclude it entirely; allow eggs in it
+ to fulfill package dependencies declared in setup.py and buildout
+ configuration; allow it to be available but not used to fulfill
+ dependencies declared in setup.py or buildout configuration; or only
+ allow certain eggs in site-packages to fulfill dependencies.
+
+- Added new function, ``zc.buildout.easy_install.sitepackage_safe_scripts``,
+ to generate scripts and interpreter. It produces a full-featured
+ interpreter (all command-line options supported) and the ability to
+ safely let scripts include site packages, such as with a system
+ Python. The ``z3c.recipe.scripts`` recipe uses this new function.
+
+- Improve bootstrap.
+
+ * New options let you specify where to find ez_setup.py and where to find
+ a download cache. These options can keep bootstrap from going over the
+ network.
+
+ * Another new option lets you specify where to put generated eggs.
+
+ * The buildout script generated by bootstrap honors more of the settings
+ in the designated configuration file (e.g., buildout.cfg).
+
+- You can develop zc.buildout using Distribute instead of Setuptools. Use
+ the --distribute option on the dev.py script. (Releases should be tested
+ with both Distribute and Setuptools.)
+
+- The ``distribute-version`` now works in the [buildout] section, mirroring
+ the ``setuptools-version`` option (this is for consistency; using the
+ general-purpose ``versions`` option is preferred).
+
+Bugs fixed:
+
+- Using Distribute with the ``allow-picked-versions = false`` buildout
+ option no longer causes an error.
+
+- The handling and documenting of default buildout options was normalized.
+ This means, among other things, that ``bin/buildout -vv`` and
+ ``bin/buildout annotate`` correctly list more of the options.
+
+- Installing a namespace package using a Python that already has a package
+ in the same namespace (e.g., in the Python's site-packages) failed in
+ some cases. It is now handled correctly.
+
+- Another variation of this error showed itself when at least two
+ dependencies were in a shared location like site-packages, and the
+ first one met the "versions" setting. The first dependency would be
+ added, but subsequent dependencies from the same location (e.g.,
+ site-packages) would use the version of the package found in the
+ shared location, ignoring the version setting. This is also now
+ handled correctly.
+
1.4.3 (2009-12-10)
==================
Modified: zc.buildout/trunk/README.txt
===================================================================
--- zc.buildout/trunk/README.txt 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/README.txt 2010-04-29 18:11:58 UTC (rev 111588)
@@ -35,8 +35,16 @@
`zc.recipe.egg <http://pypi.python.org/pypi/zc.recipe.egg>`_
The egg recipe installes one or more eggs, with their
dependencies. It installs their console-script entry points with
- the needed eggs included in their paths.
+ the needed eggs included in their paths. It is suitable for use with
+ a "clean" Python: one without packages installed in site-packages.
+`z3c.recipe.scripts <http://pypi.python.org/pypi/z3c.recipe.scripts>`_
+ Like zc.recipe.egg, this recipe builds interpreter scripts and entry
+ point scripts based on eggs. It can be used with a Python that has
+ packages installed in site-packages, such as a system Python. The
+ interpreter also has more features than the one offered by
+ zc.recipe.egg.
+
`zc.recipe.testrunner <http://pypi.python.org/pypi/zc.recipe.testrunner>`_
The testrunner egg creates a test runner script for one or
more eggs.
Modified: zc.buildout/trunk/bootstrap/bootstrap.py
===================================================================
--- zc.buildout/trunk/bootstrap/bootstrap.py 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/bootstrap/bootstrap.py 2010-04-29 18:11:58 UTC (rev 111588)
@@ -20,102 +20,187 @@
$Id$
"""
-import os, shutil, sys, tempfile, urllib2
+import os, shutil, sys, tempfile, textwrap, urllib, urllib2
from optparse import OptionParser
-tmpeggs = tempfile.mkdtemp()
+if sys.platform == 'win32':
+ def quote(c):
+ if ' ' in c:
+ return '"%s"' % c # work around spawn lamosity on windows
+ else:
+ return c
+else:
+ quote = str
+# In order to be more robust in the face of system Pythons, we want to
+# run without site-packages loaded. This is somewhat tricky, in
+# particular because Python 2.6's distutils imports site, so starting
+# with the -S flag is not sufficient. However, we'll start with that:
+if 'site' in sys.modules:
+ # We will restart with python -S.
+ args = sys.argv[:]
+ args[0:0] = [sys.executable, '-S']
+ args = map(quote, args)
+ os.execv(sys.executable, args)
+# Now we are running with -S. We'll get the clean sys.path, import site
+# because distutils will do it later, and then reset the path and clean
+# out any namespace packages from site-packages that might have been
+# loaded by .pth files.
+clean_path = sys.path[:]
+import site
+sys.path[:] = clean_path
+for k, v in sys.modules.items():
+ if (hasattr(v, '__path__') and
+ len(v.__path__)==1 and
+ not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))):
+ # This is a namespace package. Remove it.
+ sys.modules.pop(k)
+
is_jython = sys.platform.startswith('java')
+setuptools_source = 'http://peak.telecommunity.com/dist/ez_setup.py'
+distribute_source = 'http://python-distribute.org/distribute_setup.py'
+
# parsing arguments
-parser = OptionParser()
+def normalize_to_url(option, opt_str, value, parser):
+ if value:
+ if '://' not in value: # It doesn't smell like a URL.
+ value = 'file://%s' % (
+ urllib.pathname2url(
+ os.path.abspath(os.path.expanduser(value))),)
+ if opt_str == '--download-base' and not value.endswith('/'):
+ # Download base needs a trailing slash to make the world happy.
+ value += '/'
+ else:
+ value = None
+ name = opt_str[2:].replace('-', '_')
+ setattr(parser.values, name, value)
+
+usage = '''\
+[DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]
+
+Bootstraps a buildout-based project.
+
+Simply run this script in a directory containing a buildout.cfg, using the
+Python that you want bin/buildout to use.
+
+Note that by using --setup-source and --download-base to point to
+local resources, you can keep this script from going over the network.
+'''
+
+parser = OptionParser(usage=usage)
parser.add_option("-v", "--version", dest="version",
help="use a specific zc.buildout version")
parser.add_option("-d", "--distribute",
- action="store_true", dest="distribute", default=False,
+ action="store_true", dest="use_distribute", default=False,
help="Use Distribute rather than Setuptools.")
-
+parser.add_option("--setup-source", action="callback", dest="setup_source",
+ callback=normalize_to_url, nargs=1, type="string",
+ help=("Specify a URL or file location for the setup file. "
+ "If you use Setuptools, this will default to " +
+ setuptools_source + "; if you use Distribute, this "
+ "will default to " + distribute_source +"."))
+parser.add_option("--download-base", action="callback", dest="download_base",
+ callback=normalize_to_url, nargs=1, type="string",
+ help=("Specify a URL or directory for downloading "
+ "zc.buildout and either Setuptools or Distribute. "
+ "Defaults to PyPI."))
+parser.add_option("--eggs",
+ help=("Specify a directory for storing eggs. Defaults to "
+ "a temporary directory that is deleted when the "
+ "bootstrap script completes."))
parser.add_option("-c", None, action="store", dest="config_file",
help=("Specify the path to the buildout configuration "
"file to be used."))
options, args = parser.parse_args()
-# if -c was provided, we push it back into args for buildout' main function
+# if -c was provided, we push it back into args for buildout's main function
if options.config_file is not None:
args += ['-c', options.config_file]
-if options.version is not None:
- VERSION = '==%s' % options.version
+if options.eggs:
+ eggs_dir = os.path.abspath(os.path.expanduser(options.eggs))
else:
- VERSION = ''
+ eggs_dir = tempfile.mkdtemp()
-USE_DISTRIBUTE = options.distribute
+if options.setup_source is None:
+ if options.use_distribute:
+ options.setup_source = distribute_source
+ else:
+ options.setup_source = setuptools_source
+
args = args + ['bootstrap']
-to_reload = False
+
try:
+ to_reload = False
import pkg_resources
+ to_reload = True
if not hasattr(pkg_resources, '_distribute'):
- to_reload = True
raise ImportError
+ import setuptools # A flag. Sometimes pkg_resources is installed alone.
except ImportError:
+ ez_code = urllib2.urlopen(
+ options.setup_source).read().replace('\r\n', '\n')
ez = {}
- if USE_DISTRIBUTE:
- exec urllib2.urlopen('http://python-distribute.org/distribute_setup.py'
- ).read() in ez
- ez['use_setuptools'](to_dir=tmpeggs, download_delay=0, no_fake=True)
- else:
- exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py'
- ).read() in ez
- ez['use_setuptools'](to_dir=tmpeggs, download_delay=0)
-
+ exec ez_code in ez
+ setup_args = dict(to_dir=eggs_dir, download_delay=0)
+ if options.download_base:
+ setup_args['download_base'] = options.download_base
+ if options.use_distribute:
+ setup_args['no_fake'] = True
+ ez['use_setuptools'](**setup_args)
if to_reload:
reload(pkg_resources)
else:
import pkg_resources
+ # This does not (always?) update the default working set. We will
+ # do it.
+ for path in sys.path:
+ if path not in pkg_resources.working_set.entries:
+ pkg_resources.working_set.add_entry(path)
-if sys.platform == 'win32':
- def quote(c):
- if ' ' in c:
- return '"%s"' % c # work around spawn lamosity on windows
- else:
- return c
-else:
- def quote (c):
- return c
+cmd = [quote(sys.executable),
+ '-c',
+ quote('from setuptools.command.easy_install import main; main()'),
+ '-mqNxd',
+ quote(eggs_dir)]
-cmd = 'from setuptools.command.easy_install import main; main()'
-ws = pkg_resources.working_set
+if options.download_base:
+ cmd.extend(['-f', quote(options.download_base)])
-if USE_DISTRIBUTE:
- requirement = 'distribute'
+requirement = 'zc.buildout'
+if options.version:
+ requirement = '=='.join((requirement, options.version))
+cmd.append(requirement)
+
+if options.use_distribute:
+ setup_requirement = 'distribute'
else:
- requirement = 'setuptools'
+ setup_requirement = 'setuptools'
+ws = pkg_resources.working_set
+env = dict(
+ os.environ,
+ PYTHONPATH=ws.find(
+ pkg_resources.Requirement.parse(setup_requirement)).location)
if is_jython:
import subprocess
+ exitcode = subprocess.Popen(cmd, env=env).wait()
+else: # Windows prefers this, apparently; otherwise we would prefer subprocess
+ exitcode = os.spawnle(*([os.P_WAIT, sys.executable] + cmd + [env]))
+if exitcode != 0:
+ sys.stdout.flush()
+ sys.stderr.flush()
+ print ("An error occured when trying to install zc.buildout. "
+ "Look above this message for any errors that "
+ "were output by easy_install.")
+ sys.exit(exitcode)
- assert subprocess.Popen([sys.executable] + ['-c', quote(cmd), '-mqNxd',
- quote(tmpeggs), 'zc.buildout' + VERSION],
- env=dict(os.environ,
- PYTHONPATH=
- ws.find(pkg_resources.Requirement.parse(requirement)).location
- ),
- ).wait() == 0
-
-else:
- assert os.spawnle(
- os.P_WAIT, sys.executable, quote (sys.executable),
- '-c', quote (cmd), '-mqNxd', quote (tmpeggs), 'zc.buildout' + VERSION,
- dict(os.environ,
- PYTHONPATH=
- ws.find(pkg_resources.Requirement.parse(requirement)).location
- ),
- ) == 0
-
-ws.add_entry(tmpeggs)
-ws.require('zc.buildout' + VERSION)
+ws.add_entry(eggs_dir)
+ws.require(requirement)
import zc.buildout.buildout
zc.buildout.buildout.main(args)
-shutil.rmtree(tmpeggs)
+if not options.eggs: # clean up temporary egg directory
+ shutil.rmtree(eggs_dir)
Modified: zc.buildout/trunk/buildout.cfg
===================================================================
--- zc.buildout/trunk/buildout.cfg 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/buildout.cfg 2010-04-29 18:11:58 UTC (rev 111588)
@@ -1,25 +1,27 @@
[buildout]
-develop = zc.recipe.egg_ .
+develop = zc.recipe.egg_ z3c.recipe.scripts_ .
parts = test oltest py
[py]
-recipe = zc.recipe.egg
+recipe = z3c.recipe.scripts
eggs = zc.buildout
zope.testing
interpreter = py
[test]
recipe = zc.recipe.testrunner
-eggs =
+eggs =
zc.buildout
zc.recipe.egg
+ z3c.recipe.scripts
# Tests that can be run wo a network
[oltest]
recipe = zc.recipe.testrunner
-eggs =
+eggs =
zc.buildout
zc.recipe.egg
+ z3c.recipe.scripts
defaults =
[
'-t',
Modified: zc.buildout/trunk/dev.py
===================================================================
--- zc.buildout/trunk/dev.py 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/dev.py 2010-04-29 18:11:58 UTC (rev 111588)
@@ -13,43 +13,126 @@
##############################################################################
"""Bootstrap the buildout project itself.
-This is different from a normal boostrapping process because the
+This is different from a normal bootstrapping process because the
buildout egg itself is installed as a develop egg.
$Id$
"""
import os, shutil, sys, subprocess, urllib2
+from optparse import OptionParser
+if sys.platform == 'win32':
+ def quote(c):
+ if ' ' in c:
+ return '"%s"' % c # work around spawn lamosity on windows
+ else:
+ return c
+else:
+ quote = str
+
+# In order to be more robust in the face of system Pythons, we want to
+# run without site-packages loaded. This is somewhat tricky, in
+# particular because Python 2.6's distutils imports site, so starting
+# with the -S flag is not sufficient. However, we'll start with that:
+if 'site' in sys.modules:
+ # We will restart with python -S.
+ args = sys.argv[:]
+ args[0:0] = [sys.executable, '-S']
+ args = map(quote, args)
+ os.execv(sys.executable, args)
+# Now we are running with -S. We'll get the clean sys.path, import site
+# because distutils will do it later, and then reset the path and clean
+# out any namespace packages from site-packages that might have been
+# loaded by .pth files.
+clean_path = sys.path[:]
+import site
+sys.path[:] = clean_path
+for k, v in sys.modules.items():
+ if (hasattr(v, '__path__') and
+ len(v.__path__)==1 and
+ not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))):
+ # This is a namespace package. Remove it.
+ sys.modules.pop(k)
+
is_jython = sys.platform.startswith('java')
+setuptools_source = 'http://peak.telecommunity.com/dist/ez_setup.py'
+distribute_source = 'http://python-distribute.org/distribute_setup.py'
+
+usage = '''\
+[DESIRED PYTHON FOR DEVELOPING BUILDOUT] dev.py [options]
+
+Bootstraps buildout itself for development.
+
+This is different from a normal bootstrapping process because the
+buildout egg itself is installed as a develop egg.
+'''
+
+parser = OptionParser(usage=usage)
+parser.add_option("-d", "--distribute",
+ action="store_true", dest="use_distribute", default=False,
+ help="Use Distribute rather than Setuptools.")
+
+options, args = parser.parse_args()
+
+if args:
+ parser.error('This script accepts no arguments other than its options.')
+
+if options.use_distribute:
+ setup_source = distribute_source
+else:
+ setup_source = setuptools_source
+
for d in 'eggs', 'develop-eggs', 'bin':
if not os.path.exists(d):
os.mkdir(d)
-
if os.path.isdir('build'):
shutil.rmtree('build')
try:
+ to_reload = False
import pkg_resources
+ to_reload = True
+ if not hasattr(pkg_resources, '_distribute'):
+ raise ImportError
+ import setuptools # A flag. Sometimes pkg_resources is installed alone.
except ImportError:
+ ez_code = urllib2.urlopen(setup_source).read().replace('\r\n', '\n')
ez = {}
- exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py'
- ).read() in ez
- ez['use_setuptools'](to_dir='eggs', download_delay=0)
+ exec ez_code in ez
+ setup_args = dict(to_dir='eggs', download_delay=0)
+ if options.use_distribute:
+ setup_args['no_fake'] = True
+ ez['use_setuptools'](**setup_args)
+ if to_reload:
+ reload(pkg_resources)
+ else:
+ import pkg_resources
+ # This does not (always?) update the default working set. We will
+ # do it.
+ for path in sys.path:
+ if path not in pkg_resources.working_set.entries:
+ pkg_resources.working_set.add_entry(path)
- import pkg_resources
-
+env = os.environ.copy() # Windows needs yet-to-be-determined values from this.
+env['PYTHONPATH'] = os.path.dirname(pkg_resources.__file__)
subprocess.Popen(
[sys.executable] +
['setup.py', '-q', 'develop', '-m', '-x', '-d', 'develop-eggs'],
- env = {'PYTHONPATH': os.path.dirname(pkg_resources.__file__)}).wait()
+ env=env).wait()
pkg_resources.working_set.add_entry('src')
import zc.buildout.easy_install
-zc.buildout.easy_install.scripts(
- ['zc.buildout'], pkg_resources.working_set , sys.executable, 'bin')
+if not os.path.exists('parts'):
+ os.mkdir('parts')
+partsdir = os.path.join('parts', 'buildout')
+if not os.path.exists(partsdir):
+ os.mkdir(partsdir)
+zc.buildout.easy_install.sitepackage_safe_scripts(
+ 'bin', pkg_resources.working_set, sys.executable, partsdir,
+ reqs=['zc.buildout'])
bin_buildout = os.path.join('bin', 'buildout')
@@ -57,4 +140,5 @@
# Jython needs the script to be called twice via sys.executable
assert subprocess.Popen([sys.executable] + [bin_buildout]).wait() == 0
+
sys.exit(subprocess.Popen(bin_buildout).wait())
Modified: zc.buildout/trunk/setup.py
===================================================================
--- zc.buildout/trunk/setup.py 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/setup.py 2010-04-29 18:11:58 UTC (rev 111588)
@@ -12,7 +12,7 @@
#
##############################################################################
name = "zc.buildout"
-version = "1.4.4dev"
+version = "1.5.0dev"
import os
from setuptools import setup
Modified: zc.buildout/trunk/src/zc/buildout/bootstrap.txt
===================================================================
--- zc.buildout/trunk/src/zc/buildout/bootstrap.txt 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/src/zc/buildout/bootstrap.txt 2010-04-29 18:11:58 UTC (rev 111588)
@@ -47,7 +47,7 @@
X...
d zc.buildout-...egg
-Now trying the `--version` option, that let you define a version for
+Now we will try the `--version` option, which lets you define a version for
`zc.buildout`. If not provided, bootstrap will look for the latest one.
Let's try with an unknown version::
@@ -57,7 +57,7 @@
... 'bootstrap.py --version UNKNOWN'); print 'X' # doctest: +ELLIPSIS
...
X
- No local packages or download links found for zc.buildout==UNKNOWN
+ No local packages or download links found for zc.buildout==UNKNOWN...
...
Now let's try with `1.1.2`, which happens to exist::
@@ -119,9 +119,9 @@
zc.buildout.buildout.main()
<BLANKLINE>
-`zc.buildout` now can also run with `Distribute` with the `--distribute` option::
+`zc.buildout` now can also run with `Distribute` with the `--distribute`
+option::
-
>>> print 'X'; print system(
... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
... 'bootstrap.py --distribute'); print 'X' # doctest: +ELLIPSIS
@@ -153,7 +153,8 @@
>>> print 'X'; print system(
... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
- ... 'bootstrap.py --distribute --version 1.2.1'); print 'X' # doctest: +ELLIPSIS
+ ... 'bootstrap.py --distribute --version 1.2.1'); print 'X'
+ ... # doctest: +ELLIPSIS
...
X
...
@@ -161,7 +162,8 @@
<BLANKLINE>
X
-Let's make sure the generated `buildout` script uses ``Distribute`` *and* ``zc.buildout-1.2.1``::
+Let's make sure the generated `buildout` script uses ``Distribute`` *and*
+``zc.buildout-1.2.1``::
>>> print open(buildout_script).read() # doctest: +ELLIPSIS
#...
@@ -194,4 +196,70 @@
<BLANKLINE>
X
+You can specify a location of ez_setup.py or distribute_setup, so you
+can rely on a local or remote location. We'll write our own ez_setup.py
+that we will also use to test some other bootstrap options.
+ >>> write('ez_setup.py', '''\
+ ... def use_setuptools(**kwargs):
+ ... import sys, pprint
+ ... pprint.pprint(kwargs, width=40)
+ ... sys.exit()
+ ... ''')
+ >>> print system(
+ ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
+ ... 'bootstrap.py --setup-source=./ez_setup.py')
+ ... # doctest: +ELLIPSIS
+ {'download_delay': 0,
+ 'to_dir': '...'}
+ <BLANKLINE>
+
+You can also pass a download-cache, and a place in which eggs should be stored
+(they are normally stored in a temporary directory).
+
+ >>> print system(
+ ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
+ ... 'bootstrap.py --setup-source=./ez_setup.py '+
+ ... '--download-base=./download-cache --eggs=eggs')
+ ... # doctest: +ELLIPSIS
+ {'download_base': '/sample/download-cache/',
+ 'download_delay': 0,
+ 'to_dir': '/sample/eggs'}
+ <BLANKLINE>
+
+Here's the entire help text.
+
+ >>> print system(
+ ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
+ ... 'bootstrap.py --help'),
+ ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
+ Usage: [DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]
+ <BLANKLINE>
+ Bootstraps a buildout-based project.
+ <BLANKLINE>
+ Simply run this script in a directory containing a buildout.cfg, using the
+ Python that you want bin/buildout to use.
+ <BLANKLINE>
+ Note that by using --setup-source and --download-base to point to
+ local resources, you can keep this script from going over the network.
+ <BLANKLINE>
+ <BLANKLINE>
+ Options:
+ -h, --help show this help message and exit
+ -v VERSION, --version=VERSION
+ use a specific zc.buildout version
+ -d, --distribute Use Distribute rather than Setuptools.
+ --setup-source=SETUP_SOURCE
+ Specify a URL or file location for the setup file. If
+ you use Setuptools, this will default to
+ http://peak.telecommunity.com/dist/ez_setup.py; if you
+ use Distribute, this will default to http://python-
+ distribute.org/distribute_setup.py.
+ --download-base=DOWNLOAD_BASE
+ Specify a URL or directory for downloading zc.buildout
+ and either Setuptools or Distribute. Defaults to PyPI.
+ --eggs=EGGS Specify a directory for storing eggs. Defaults to a
+ temporary directory that is deleted when the bootstrap
+ script completes.
+ -c CONFIG_FILE Specify the path to the buildout configuration file to
+ be used.
Modified: zc.buildout/trunk/src/zc/buildout/buildout.py
===================================================================
--- zc.buildout/trunk/src/zc/buildout/buildout.py 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/src/zc/buildout/buildout.py 2010-04-29 18:11:58 UTC (rev 111588)
@@ -112,15 +112,26 @@
return data
_buildout_default_options = _annotate_section({
+ 'allow-hosts': '*',
+ 'allow-picked-versions': 'true',
+ 'bin-directory': 'bin',
+ 'develop-eggs-directory': 'develop-eggs',
'eggs-directory': 'eggs',
- 'develop-eggs-directory': 'develop-eggs',
- 'bin-directory': 'bin',
+ 'executable': sys.executable,
+ 'find-links': '',
+ 'install-from-cache': 'false',
+ 'installed': '.installed.cfg',
+ 'log-format': '',
+ 'log-level': 'INFO',
+ 'newest': 'true',
+ 'offline': 'false',
'parts-directory': 'parts',
- 'installed': '.installed.cfg',
+ 'prefer-final': 'false',
'python': 'buildout',
- 'executable': sys.executable,
- 'log-level': 'INFO',
- 'log-format': '',
+ 'relative-paths': 'false',
+ 'socket-timeout': '',
+ 'unzip': 'false',
+ 'use-dependency-links': 'true',
}, 'DEFAULT_VALUE')
@@ -191,7 +202,7 @@
# provide some defaults before options are parsed
# because while parsing options those attributes might be
# used already (Gottfried Ganssauge)
- buildout_section = data.get('buildout')
+ buildout_section = data['buildout']
# Try to make sure we have absolute paths for standard
# directories. We do this before doing substitutions, in case
@@ -204,22 +215,28 @@
d = self._buildout_path(buildout_section[name+'-directory'])
buildout_section[name+'-directory'] = d
- links = buildout_section and buildout_section.get('find-links', '')
+ # Attributes on this buildout object shouldn't be used by
+ # recipes in their __init__. It can cause bugs, because the
+ # recipes will be instantiated below (``options = self['buildout']``)
+ # before this has completed initializing. These attributes are
+ # left behind for legacy support but recipe authors should
+ # beware of using them. A better practice is for a recipe to
+ # use the buildout['buildout'] options.
+ links = buildout_section['find-links']
self._links = links and links.split() or ()
-
- allow_hosts = buildout_section and buildout_section.get(
- 'allow-hosts', '*').split('\n')
+ allow_hosts = buildout_section['allow-hosts'].split('\n')
self._allow_hosts = tuple([host.strip() for host in allow_hosts
if host.strip() != ''])
-
self._logger = logging.getLogger('zc.buildout')
- self.offline = False
- self.newest = True
+ self.offline = (buildout_section['offline'] == 'true')
+ self.newest = (buildout_section['newest'] == 'true')
##################################################################
## WARNING!!!
## ALL ATTRIBUTES MUST HAVE REASONABLE DEFAULTS AT THIS POINT
- ## OTHERWISE ATTRIBUTEERRORS MIGHT HAPPEN ANY TIME
+ ## OTHERWISE ATTRIBUTEERRORS MIGHT HAPPEN ANY TIME FROM RECIPES.
+ ## RECIPES SHOULD GENERALLY USE buildout['buildout'] OPTIONS, NOT
+ ## BUILDOUT ATTRIBUTES.
##################################################################
# initialize some attrs and buildout directories.
options = self['buildout']
@@ -228,7 +245,7 @@
links = options.get('find-links', '')
self._links = links and links.split() or ()
- allow_hosts = options.get('allow-hosts', '*').split('\n')
+ allow_hosts = options['allow-hosts'].split('\n')
self._allow_hosts = tuple([host.strip() for host in allow_hosts
if host.strip() != ''])
@@ -246,44 +263,42 @@
self._setup_logging()
- offline = options.get('offline', 'false')
+ offline = options['offline']
if offline not in ('true', 'false'):
self._error('Invalid value for offline option: %s', offline)
- options['offline'] = offline
- self.offline = offline == 'true'
+ self.offline = (offline == 'true')
if self.offline:
newest = options['newest'] = 'false'
else:
- newest = options.get('newest', 'true')
+ newest = options['newest']
if newest not in ('true', 'false'):
self._error('Invalid value for newest option: %s', newest)
- options['newest'] = newest
- self.newest = newest == 'true'
+ self.newest = (newest == 'true')
versions = options.get('versions')
if versions:
zc.buildout.easy_install.default_versions(dict(self[versions]))
- prefer_final = options.get('prefer-final', 'false')
+ prefer_final = options['prefer-final']
if prefer_final not in ('true', 'false'):
self._error('Invalid value for prefer-final option: %s',
prefer_final)
zc.buildout.easy_install.prefer_final(prefer_final=='true')
- use_dependency_links = options.get('use-dependency-links', 'true')
+ use_dependency_links = options['use-dependency-links']
if use_dependency_links not in ('true', 'false'):
self._error('Invalid value for use-dependency-links option: %s',
use_dependency_links)
zc.buildout.easy_install.use_dependency_links(
use_dependency_links == 'true')
- allow_picked_versions = options.get('allow-picked-versions', 'true')
+ allow_picked_versions = options['allow-picked-versions']
if allow_picked_versions not in ('true', 'false'):
self._error('Invalid value for allow-picked-versions option: %s',
allow_picked_versions)
zc.buildout.easy_install.allow_picked_versions(
- allow_picked_versions=='true')
+ allow_picked_versions == 'true')
download_cache = options.get('download-cache')
if download_cache:
@@ -300,23 +315,19 @@
zc.buildout.easy_install.download_cache(download_cache)
- install_from_cache = options.get('install-from-cache')
- if install_from_cache:
- if install_from_cache not in ('true', 'false'):
- self._error('Invalid value for install-from-cache option: %s',
- install_from_cache)
- if install_from_cache == 'true':
- zc.buildout.easy_install.install_from_cache(True)
+ install_from_cache = options['install-from-cache']
+ if install_from_cache not in ('true', 'false'):
+ self._error('Invalid value for install-from-cache option: %s',
+ install_from_cache)
+ zc.buildout.easy_install.install_from_cache(
+ install_from_cache=='true')
+ always_unzip = options['unzip']
+ if always_unzip not in ('true', 'false'):
+ self._error('Invalid value for unzip option: %s',
+ always_unzip)
+ zc.buildout.easy_install.always_unzip(always_unzip=='true')
- always_unzip = options.get('unzip')
- if always_unzip:
- if always_unzip not in ('true', 'false'):
- self._error('Invalid value for unzip option: %s',
- always_unzip)
- if always_unzip == 'true':
- zc.buildout.easy_install.always_unzip(True)
-
# "Use" each of the defaults so they aren't reported as unused options.
for name in _buildout_default_options:
options[name]
@@ -338,11 +349,35 @@
self._setup_directories()
+ options = self['buildout']
+
+ # Get a base working set for our distributions that corresponds to the
+ # stated desires in the configuration.
+ distributions = ['setuptools', 'zc.buildout']
+ if options.get('offline') == 'true':
+ ws = zc.buildout.easy_install.working_set(
+ distributions, options['executable'],
+ [options['develop-eggs-directory'],
+ options['eggs-directory']],
+ include_site_packages=False,
+ )
+ else:
+ ws = zc.buildout.easy_install.install(
+ distributions, options['eggs-directory'],
+ links=self._links,
+ index=options.get('index'),
+ executable=options['executable'],
+ path=[options['develop-eggs-directory']],
+ newest=self.newest,
+ allow_hosts=self._allow_hosts,
+ include_site_packages=False,
+ )
+
# Now copy buildout and setuptools eggs, and record destination eggs:
entries = []
for name in 'setuptools', 'zc.buildout':
r = pkg_resources.Requirement.parse(name)
- dist = pkg_resources.working_set.find(r)
+ dist = ws.find(r)
if dist.precedence == pkg_resources.DEVELOP_DIST:
dest = os.path.join(self['buildout']['develop-eggs-directory'],
name+'.egg-link')
@@ -361,9 +396,19 @@
# Create buildout script
ws = pkg_resources.WorkingSet(entries)
ws.require('zc.buildout')
- zc.buildout.easy_install.scripts(
- ['zc.buildout'], ws, sys.executable,
- self['buildout']['bin-directory'])
+ partsdir = os.path.join(options['parts-directory'], 'buildout')
+ if not os.path.exists(partsdir):
+ os.mkdir(partsdir)
+ # (Honor the relative-paths option.)
+ relative_paths = options.get('relative-paths', 'false')
+ if relative_paths == 'true':
+ relative_paths = options['directory']
+ else:
+ assert relative_paths == 'false'
+ relative_paths = ''
+ zc.buildout.easy_install.sitepackage_safe_scripts(
+ options['bin-directory'], ws, options['executable'], partsdir,
+ reqs=['zc.buildout'], relative_paths=relative_paths)
init = bootstrap
@@ -533,7 +578,7 @@
if installed_files is None:
self._logger.warning(
"The %s install returned None. A path or "
- "iterable os paths should be returned.",
+ "iterable of paths should be returned.",
part)
installed_files = ()
elif isinstance(installed_files, str):
@@ -792,16 +837,24 @@
if not self.newest:
return
+ options = self['buildout']
+
+ specs = ['zc.buildout']
+ if zc.buildout.easy_install.is_distribute:
+ specs.append('distribute')
+ else:
+ specs.append('setuptools')
ws = zc.buildout.easy_install.install(
[
- (spec + ' ' + self['buildout'].get(spec+'-version', '')).strip()
- for spec in ('zc.buildout', 'setuptools')
+ (spec + ' ' + options.get(spec+'-version', '')).strip()
+ for spec in specs
],
- self['buildout']['eggs-directory'],
- links = self['buildout'].get('find-links', '').split(),
- index = self['buildout'].get('index'),
- path = [self['buildout']['develop-eggs-directory']],
- allow_hosts = self._allow_hosts
+ options['eggs-directory'],
+ links = options.get('find-links', '').split(),
+ index = options.get('index'),
+ path = [options['develop-eggs-directory']],
+ allow_hosts = self._allow_hosts,
+ include_site_packages=False
)
upgraded = []
@@ -817,7 +870,7 @@
__doing__ = 'Upgrading.'
should_run = realpath(
- os.path.join(os.path.abspath(self['buildout']['bin-directory']),
+ os.path.join(os.path.abspath(options['bin-directory']),
'buildout')
)
if sys.platform == 'win32':
@@ -849,21 +902,34 @@
# the new dist is different, so we've upgraded.
# Update the scripts and return True
- zc.buildout.easy_install.scripts(
- ['zc.buildout'], ws, sys.executable,
- self['buildout']['bin-directory'],
- )
+ partsdir = os.path.join(options['parts-directory'], 'buildout')
+ if os.path.exists(partsdir):
+ # This is primarily for unit tests, in which .py files change too
+ # fast for Python to know to regenerate the .pyc/.pyo files.
+ shutil.rmtree(partsdir)
+ os.mkdir(partsdir)
+ zc.buildout.easy_install.sitepackage_safe_scripts(
+ options['bin-directory'], ws, sys.executable, partsdir,
+ reqs=['zc.buildout'])
# Restart
args = map(zc.buildout.easy_install._safe_arg, sys.argv)
if not __debug__:
args.insert(0, '-O')
- args.insert(0, zc.buildout.easy_install._safe_arg (sys.executable))
-
+ args.insert(0, zc.buildout.easy_install._safe_arg(sys.executable))
+ # We want to make sure that our new site.py is used for rerunning
+ # buildout, so we put the partsdir in PYTHONPATH for our restart.
+ # This overrides any set PYTHONPATH, but since we generally are
+ # trying to run with a completely "clean" python (only the standard
+ # library) then that should be fine.
+ env = os.environ.copy()
+ env['PYTHONPATH'] = partsdir
if is_jython:
- sys.exit(subprocess.Popen([sys.executable] + list(args)).wait())
+ sys.exit(
+ subprocess.Popen(
+ [sys.executable] + list(args), env=env).wait())
else:
- sys.exit(os.spawnv(os.P_WAIT, sys.executable, args))
+ sys.exit(os.spawnve(os.P_WAIT, sys.executable, args, env))
def _load_extensions(self):
__doing__ = 'Loading extensions.'
@@ -884,7 +950,8 @@
working_set=pkg_resources.working_set,
links = self['buildout'].get('find-links', '').split(),
index = self['buildout'].get('index'),
- newest=self.newest, allow_hosts=self._allow_hosts)
+ newest=self.newest, allow_hosts=self._allow_hosts,
+ include_site_packages=False)
# Clear cache because extensions might now let us read pages we
# couldn't read before.
@@ -1001,7 +1068,8 @@
path=path,
working_set=pkg_resources.working_set,
newest=buildout.newest,
- allow_hosts=buildout._allow_hosts
+ allow_hosts=buildout._allow_hosts,
+ include_site_packages=False,
)
__doing__ = 'Loading %s recipe entry %s:%s.', group, spec, entry
Modified: zc.buildout/trunk/src/zc/buildout/buildout.txt
===================================================================
--- zc.buildout/trunk/src/zc/buildout/buildout.txt 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/src/zc/buildout/buildout.txt 2010-04-29 18:11:58 UTC (rev 111588)
@@ -56,10 +56,9 @@
- setuptools-0.6-py2.4.egg
- zc.buildout-1.0-py2.4.egg
-The develop-eggs and parts directories are initially empty:
+The develop-eggs directory is initially empty:
>>> ls(sample_buildout, 'develop-eggs')
- >>> ls(sample_buildout, 'parts')
The develop-eggs directory holds egg links for software being
developed in the buildout. We separate develop-eggs and other eggs to
@@ -69,6 +68,12 @@
allows larger buildouts to be set up much more quickly and saves disk
space.
+The parts directory just contains some helpers for the buildout script
+itself.
+
+ >>> ls(sample_buildout, 'parts')
+ d buildout
+
The parts directory provides an area where recipes can install
part data. For example, if we built a custom Python, we would
install it in the part directory. Part data is stored in a
@@ -576,7 +581,7 @@
.. Wait for the file to really disappear. My linux is weird.
>>> wait_until("foo goes away", lambda : not os.path.exists('foo'),
- ... timeout=100)
+ ... timeout=200)
we get the same error, but we don't get the directory left behind:
@@ -724,6 +729,10 @@
==================
<BLANKLINE>
[buildout]
+ allow-hosts= *
+ DEFAULT_VALUE
+ allow-picked-versions= true
+ DEFAULT_VALUE
bin-directory= bin
DEFAULT_VALUE
develop= recipes
@@ -736,18 +745,36 @@
DEFAULT_VALUE
executable= ...
DEFAULT_VALUE
+ find-links=
+ DEFAULT_VALUE
+ install-from-cache= false
+ DEFAULT_VALUE
installed= .installed.cfg
DEFAULT_VALUE
log-format=
DEFAULT_VALUE
log-level= INFO
DEFAULT_VALUE
+ newest= true
+ DEFAULT_VALUE
+ offline= false
+ DEFAULT_VALUE
parts= data-dir
/sample-buildout/buildout.cfg
parts-directory= parts
DEFAULT_VALUE
+ prefer-final= false
+ DEFAULT_VALUE
python= buildout
DEFAULT_VALUE
+ relative-paths= false
+ DEFAULT_VALUE
+ socket-timeout=
+ DEFAULT_VALUE
+ unzip= false
+ DEFAULT_VALUE
+ use-dependency-links= true
+ DEFAULT_VALUE
<BLANKLINE>
[data-dir]
path= foo bins
@@ -2194,17 +2221,21 @@
>>> print system(buildout+' -vv'), # doctest: +NORMALIZE_WHITESPACE
Installing 'zc.buildout', 'setuptools'.
- We have a develop egg: zc.buildout 1.0.0.
+ We have a develop egg: zc.buildout X.X.
We have the best distribution that satisfies 'setuptools'.
- Picked: setuptools = 0.6
+ Picked: setuptools = V.V
<BLANKLINE>
Configuration data:
[buildout]
+ allow-hosts = *
+ allow-picked-versions = true
bin-directory = /sample-buildout/bin
develop-eggs-directory = /sample-buildout/develop-eggs
directory = /sample-buildout
eggs-directory = /sample-buildout/eggs
- executable = /usr/local/bin/python2.3
+ executable = python
+ find-links =
+ install-from-cache = false
installed = /sample-buildout/.installed.cfg
log-format =
log-level = INFO
@@ -2212,7 +2243,12 @@
offline = false
parts =
parts-directory = /sample-buildout/parts
+ prefer-final = false
python = buildout
+ relative-paths = false
+ socket-timeout =
+ unzip = false
+ use-dependency-links = true
verbosity = 20
<BLANKLINE>
@@ -2273,6 +2309,33 @@
Python executable. By default, the buildout section defines the
default Python as the Python used to run the buildout.
+relative-paths
+ The paths generated by zc.buildout are absolute by default, and this
+ option is ``false``. However, if you set this value to be ``true``,
+ bin/buildout will be generated with code that makes the paths relative.
+ Some recipes, such as zc.recipe.egg and z3c.recipe.scripts, honor this
+ value as well.
+
+unzip
+ By default, zc.buildout doesn't unzip zip-safe eggs ("unzip = false").
+ This follows the policy followed by setuptools itself. Experience shows
+ this policy to to be inconvenient. Zipped eggs make debugging more
+ difficult and often import more slowly. You can include an unzip option in
+ the buildout section to change the default unzipping policy ("unzip =
+ true").
+
+use-dependency-links
+ By default buildout will obey the setuptools dependency_links metadata
+ when it looks for dependencies. This behavior can be controlled with
+ the use-dependency-links buildout option::
+
+ [buildout]
+ ...
+ use-dependency-links = false
+
+ The option defaults to true. If you set it to false, then dependency
+ links are only looked for in the locations specified by find-links.
+
verbosity
A log-level adjustment. Typically, this is set via the -q and -v
command-line options.
@@ -2319,9 +2382,57 @@
directory if the original buildout had develop eggs for either
buildout or setuptools.)
-Note that the buildout script was installed but not run. To run
-the buildout, we'd have to run the installed buildout script.
+If relative-paths is ``true``, the buildout script uses relative paths.
+ >>> write(sample_bootstrapped, 'setup.cfg',
+ ... '''
+ ... [buildout]
+ ... relative-paths = true
+ ... parts =
+ ... ''')
+
+ >>> print system(buildout
+ ... +' -c'+os.path.join(sample_bootstrapped, 'setup.cfg')
+ ... +' bootstrap'),
+ Generated script '/sample-bootstrapped/bin/buildout'.
+
+ >>> buildout_script = join(sample_bootstrapped, 'bin', 'buildout')
+ >>> import sys
+ >>> if sys.platform.startswith('win'):
+ ... buildout_script += '-script.py'
+ >>> print open(buildout_script).read() # doctest: +ELLIPSIS
+ #!... -S
+ <BLANKLINE>
+ import os
+ <BLANKLINE>
+ join = os.path.join
+ base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
+ base = os.path.dirname(base)
+ <BLANKLINE>
+ import sys
+ sys.path[0:0] = [
+ join(base, 'parts/buildout'),
+ ]
+ <BLANKLINE>
+ <BLANKLINE>
+ import os
+ path = sys.path[0]
+ if os.environ.get('PYTHONPATH'):
+ path = os.pathsep.join([path, os.environ['PYTHONPATH']])
+ os.environ['PYTHONPATH'] = path
+ import site # imports custom buildout-generated site.py
+ <BLANKLINE>
+ import zc.buildout.buildout
+ <BLANKLINE>
+ if __name__ == '__main__':
+ zc.buildout.buildout.main()
+ <BLANKLINE>
+
+
+Note that, in the above two examples, the buildout script was installed
+but not run. To run the buildout, we'd have to run the installed
+buildout script.
+
If we have an existing buildout that already has a buildout.cfg, we'll
normally use the bootstrap command instead of init. It will complain
if there isn't a configuration file:
Modified: zc.buildout/trunk/src/zc/buildout/easy_install.py
===================================================================
--- zc.buildout/trunk/src/zc/buildout/easy_install.py 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/src/zc/buildout/easy_install.py 2010-04-29 18:11:58 UTC (rev 111588)
@@ -19,6 +19,7 @@
"""
import distutils.errors
+import fnmatch
import glob
import logging
import os
@@ -50,6 +51,8 @@
is_win32 = sys.platform == 'win32'
is_jython = sys.platform.startswith('java')
+is_distribute = (
+ pkg_resources.Requirement.parse('setuptools').key=='distribute')
if is_jython:
import java.lang.System
@@ -60,14 +63,77 @@
pkg_resources.Requirement.parse('setuptools')
).location
-# Include buildout and setuptools eggs in paths
-buildout_and_setuptools_path = [
- setuptools_loc,
- pkg_resources.working_set.find(
- pkg_resources.Requirement.parse('zc.buildout')).location,
- ]
+# Include buildout and setuptools eggs in paths. We prevent dupes just to
+# keep from duplicating any log messages about them.
+buildout_loc = pkg_resources.working_set.find(
+ pkg_resources.Requirement.parse('zc.buildout')).location
+buildout_and_setuptools_path = [setuptools_loc]
+if os.path.normpath(setuptools_loc) != os.path.normpath(buildout_loc):
+ buildout_and_setuptools_path.append(buildout_loc)
+def _get_system_paths(executable):
+ """Return lists of standard lib and site paths for executable.
+ """
+ # We want to get a list of the site packages, which is not easy.
+ # The canonical way to do this is to use
+ # distutils.sysconfig.get_python_lib(), but that only returns a
+ # single path, which does not reflect reality for many system
+ # Pythons, which have multiple additions. Instead, we start Python
+ # with -S, which does not import site.py and set up the extra paths
+ # like site-packages or (Ubuntu/Debian) dist-packages and
+ # python-support. We then compare that sys.path with the normal one
+ # (minus user packages if this is Python 2.6, because we don't
+ # support those (yet?). The set of the normal one minus the set of
+ # the ones in ``python -S`` is the set of packages that are
+ # effectively site-packages.
+ #
+ # The given executable might not be the current executable, so it is
+ # appropriate to do another subprocess to figure out what the
+ # additional site-package paths are. Moreover, even if this
+ # executable *is* the current executable, this code might be run in
+ # the context of code that has manipulated the sys.path--for
+ # instance, to add local zc.buildout or setuptools eggs.
+ def get_sys_path(*args, **kwargs):
+ cmd = [executable]
+ cmd.extend(args)
+ cmd.extend([
+ "-c", "import sys, os;"
+ "print repr([os.path.normpath(p) for p in sys.path if p])"])
+ # Windows needs some (as yet to be determined) part of the real env.
+ env = os.environ.copy()
+ # We need to make sure that PYTHONPATH, which will often be set
+ # to include a custom buildout-generated site.py, is not set, or
+ # else we will not get an accurate sys.path for the executable.
+ env.pop('PYTHONPATH', None)
+ env.update(kwargs)
+ _proc = subprocess.Popen(
+ cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
+ stdout, stderr = _proc.communicate();
+ if _proc.returncode:
+ raise RuntimeError(
+ 'error trying to get system packages:\n%s' % (stderr,))
+ res = eval(stdout.strip())
+ try:
+ res.remove('.')
+ except ValueError:
+ pass
+ return res
+ stdlib = get_sys_path('-S') # stdlib only
+ no_user_paths = get_sys_path(PYTHONNOUSERSITE='x')
+ site_paths = [p for p in no_user_paths if p not in stdlib]
+ return (stdlib, site_paths)
+def _get_version_info(executable):
+ cmd = [executable, '-Sc', 'import sys; print repr(sys.version_info)']
+ _proc = subprocess.Popen(
+ cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ stdout, stderr = _proc.communicate();
+ if _proc.returncode:
+ raise RuntimeError(
+ 'error trying to get system packages:\n%s' % (stderr,))
+ return eval(stdout.strip())
+
+
class IncompatibleVersionError(zc.buildout.UserError):
"""A specified version is incompatible with a given requirement.
"""
@@ -109,7 +175,12 @@
_indexes = {}
-def _get_index(executable, index_url, find_links, allow_hosts=('*',)):
+def _get_index(executable, index_url, find_links, allow_hosts=('*',),
+ path=None):
+ # If path is None, the index will use sys.path. If you provide an empty
+ # path ([]), it will complain uselessly about missing index pages for
+ # packages found in the paths that you expect to use. Therefore, this path
+ # is always the same as the _env path in the Installer.
key = executable, index_url, tuple(find_links)
index = _indexes.get(key)
if index is not None:
@@ -118,7 +189,8 @@
if index_url is None:
index_url = default_index_url
index = AllowHostsPackageIndex(
- index_url, hosts=allow_hosts, python=_get_version(executable)
+ index_url, hosts=allow_hosts, search_path=path,
+ python=_get_version(executable)
)
if find_links:
@@ -131,17 +203,81 @@
if is_win32:
# work around spawn lamosity on windows
- # XXX need safe quoting (see the subproces.list2cmdline) and test
+ # XXX need safe quoting (see the subprocess.list2cmdline) and test
def _safe_arg(arg):
return '"%s"' % arg
else:
_safe_arg = str
-_easy_install_cmd = _safe_arg(
- 'from setuptools.command.easy_install import main; main()'
- )
+# The following string is used to run easy_install in
+# Installer._call_easy_install. It is started with python -S (that is,
+# don't import site at start). That flag, and all of the code in this
+# snippet above the last two lines, exist to work around a relatively rare
+# problem. If
+#
+# - your buildout configuration is trying to install a package that is within
+# a namespace package, and
+#
+# - you use a Python that has a different version of this package
+# installed in in its site-packages using
+# --single-version-externally-managed (that is, using the mechanism
+# sometimes used by system packagers:
+# http://peak.telecommunity.com/DevCenter/setuptools#install-command ), and
+#
+# - the new package tries to do sys.path tricks in the setup.py to get a
+# __version__,
+#
+# then the older package will be loaded first, making the setup version
+# the wrong number. While very arguably packages simply shouldn't do
+# the sys.path tricks, some do, and we don't want buildout to fall over
+# when they do.
+#
+# The namespace packages installed in site-packages with
+# --single-version-externally-managed use a mechanism that cause them to
+# be processed when site.py is imported (see
+# http://mail.python.org/pipermail/distutils-sig/2009-May/011730.html
+# for another description of the problem). Simply starting Python with
+# -S addresses the problem in Python 2.4 and 2.5, but Python 2.6's
+# distutils imports a value from the site module, so we unfortunately
+# have to do more drastic surgery in the _easy_install_cmd code below.
+#
+# Here's an example of the .pth files created by setuptools when using that
+# flag:
+#
+# import sys,new,os;
+# p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('<NAMESPACE>',));
+# ie = os.path.exists(os.path.join(p,'__init__.py'));
+# m = not ie and sys.modules.setdefault('<NAMESPACE>',new.module('<NAMESPACE>'));
+# mp = (m or []) and m.__dict__.setdefault('__path__',[]);
+# (p not in mp) and mp.append(p)
+#
+# The code, below, then, runs under -S, indicating that site.py should
+# not be loaded initially. It gets the initial sys.path under these
+# circumstances, and then imports site (because Python 2.6's distutils
+# will want it, as mentioned above). It then reinstates the old sys.path
+# value. Then it removes namespace packages (created by the setuptools
+# code above) from sys.modules. It identifies namespace packages by
+# iterating over every loaded module. It first looks if there is a
+# __path__, so it is a package; and then it sees if that __path__ does
+# not have an __init__.py. (Note that PEP 382,
+# http://www.python.org/dev/peps/pep-0382, makes it possible to have a
+# namespace package that has an __init__.py, but also should make it
+# unnecessary for site.py to preprocess these packages, so it should be
+# fine, as far as can be guessed as of this writing.) Finally, it
+# imports easy_install and runs it.
+_easy_install_cmd = _safe_arg('''\
+import sys,os;\
+p = sys.path[:];\
+import site;\
+sys.path[:] = p;\
+[sys.modules.pop(k) for k, v in sys.modules.items()\
+ if hasattr(v, '__path__') and len(v.__path__)==1 and\
+ not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))];\
+from setuptools.command.easy_install import main;\
+main()''')
+
class Installer:
_versions = {}
@@ -151,6 +287,8 @@
_use_dependency_links = True
_allow_picked_versions = True
_always_unzip = False
+ _include_site_packages = True
+ _allowed_eggs_from_site_packages = ('*',)
def __init__(self,
dest=None,
@@ -162,7 +300,9 @@
newest=True,
versions=None,
use_dependency_links=None,
- allow_hosts=('*',)
+ allow_hosts=('*',),
+ include_site_packages=None,
+ allowed_eggs_from_site_packages=None
):
self._dest = dest
self._allow_hosts = allow_hosts
@@ -184,7 +324,25 @@
self._executable = executable
if always_unzip is not None:
self._always_unzip = always_unzip
- path = (path and path[:] or []) + buildout_and_setuptools_path
+ path = (path and path[:] or [])
+ if include_site_packages is not None:
+ self._include_site_packages = include_site_packages
+ if allowed_eggs_from_site_packages is not None:
+ self._allowed_eggs_from_site_packages = tuple(
+ allowed_eggs_from_site_packages)
+ stdlib, self._site_packages = _get_system_paths(executable)
+ version_info = _get_version_info(executable)
+ if version_info == sys.version_info:
+ # Maybe we can add the buildout and setuptools path. If we
+ # are including site_packages, we only have to include the extra
+ # bits here, so we don't duplicate. On the other hand, if we
+ # are not including site_packages, we only want to include the
+ # parts that are not in site_packages, so the code is the same.
+ path.extend(
+ set(buildout_and_setuptools_path).difference(
+ self._site_packages))
+ if self._include_site_packages:
+ path.extend(self._site_packages)
if dest is not None and dest not in path:
path.insert(0, dest)
self._path = path
@@ -193,13 +351,42 @@
self._newest = newest
self._env = pkg_resources.Environment(path,
python=_get_version(executable))
- self._index = _get_index(executable, index, links, self._allow_hosts)
+ self._index = _get_index(executable, index, links, self._allow_hosts,
+ self._path)
if versions is not None:
self._versions = versions
+ _allowed_eggs_from_site_packages_regex = None
+ def allow_site_package_egg(self, name):
+ if (not self._include_site_packages or
+ not self._allowed_eggs_from_site_packages):
+ # If the answer is a blanket "no," perform a shortcut.
+ return False
+ if self._allowed_eggs_from_site_packages_regex is None:
+ pattern = '(%s)' % (
+ '|'.join(
+ fnmatch.translate(name)
+ for name in self._allowed_eggs_from_site_packages),
+ )
+ self._allowed_eggs_from_site_packages_regex = re.compile(pattern)
+ return bool(self._allowed_eggs_from_site_packages_regex.match(name))
+
def _satisfied(self, req, source=None):
- dists = [dist for dist in self._env[req.project_name] if dist in req]
+ # We get all distributions that match the given requirement. If we are
+ # not supposed to include site-packages for the given egg, we also
+ # filter those out. Even if include_site_packages is False and so we
+ # have excluded site packages from the _env's paths (see
+ # Installer.__init__), we need to do the filtering here because an
+ # .egg-link, such as one for setuptools or zc.buildout installed by
+ # zc.buildout.buildout.Buildout.bootstrap, can indirectly include a
+ # path in our _site_packages.
+ dists = [dist for dist in self._env[req.project_name] if (
+ dist in req and (
+ dist.location not in self._site_packages or
+ self.allow_site_package_egg(dist.project_name))
+ )
+ ]
if not dists:
logger.debug('We have no distributions for %s that satisfies %r.',
req.project_name, str(req))
@@ -301,7 +488,7 @@
try:
path = setuptools_loc
- args = ('-c', _easy_install_cmd, '-mUNxd', _safe_arg(tmp))
+ args = ('-Sc', _easy_install_cmd, '-mUNxd', _safe_arg(tmp))
if self._always_unzip:
args += ('-Z', )
level = logger.getEffectiveLevel()
@@ -400,14 +587,22 @@
# Nothing is available.
return None
- # Filter the available dists for the requirement and source flag
- dists = [dist for dist in index[requirement.project_name]
- if ((dist in requirement)
- and
- ((not source) or
- (dist.precedence == pkg_resources.SOURCE_DIST)
- )
- )
+ # Filter the available dists for the requirement and source flag. If
+ # we are not supposed to include site-packages for the given egg, we
+ # also filter those out. Even if include_site_packages is False and so
+ # we have excluded site packages from the _env's paths (see
+ # Installer.__init__), we need to do the filtering here because an
+ # .egg-link, such as one for setuptools or zc.buildout installed by
+ # zc.buildout.buildout.Buildout.bootstrap, can indirectly include a
+ # path in our _site_packages.
+ dists = [dist for dist in index[requirement.project_name] if (
+ dist in requirement and (
+ dist.location not in self._site_packages or
+ self.allow_site_package_egg(dist.project_name))
+ and (
+ (not source) or
+ (dist.precedence == pkg_resources.SOURCE_DIST))
+ )
]
# If we prefer final dists, filter for final and use the
@@ -567,7 +762,7 @@
self._links.append(link)
self._index = _get_index(self._executable,
self._index_url, self._links,
- self._allow_hosts)
+ self._allow_hosts, self._path)
for dist in dists:
# Check whether we picked a version and, if we did, report it:
@@ -609,6 +804,8 @@
def _constrain(self, requirement):
+ if is_distribute and requirement.key == 'setuptools':
+ requirement = pkg_resources.Requirement.parse('distribute')
version = self._versions.get(requirement.project_name)
if version:
if version not in requirement:
@@ -648,35 +845,52 @@
self._maybe_add_setuptools(ws, dist)
# OK, we have the requested distributions and they're in the working
- # set, but they may have unmet requirements. We'll simply keep
- # trying to resolve requirements, adding missing requirements as they
- # are reported.
- #
- # Note that we don't pass in the environment, because we want
+ # set, but they may have unmet requirements. We'll resolve these
+ # requirements. This is code modified from
+ # pkg_resources.WorkingSet.resolve. We can't reuse that code directly
+ # because we have to constrain our requirements (see
+ # versions_section_ignored_for_dependency_in_favor_of_site_packages in
+ # zc.buildout.tests).
+ requirements.reverse() # Set up the stack.
+ processed = {} # This is a set of processed requirements.
+ best = {} # This is a mapping of key -> dist.
+ # Note that we don't use the existing environment, because we want
# to look for new eggs unless what we have is the best that
# matches the requirement.
- while 1:
- try:
- ws.resolve(requirements)
- except pkg_resources.DistributionNotFound, err:
- [requirement] = err
- requirement = self._constrain(requirement)
- if destination:
- logger.debug('Getting required %r', str(requirement))
- else:
- logger.debug('Adding required %r', str(requirement))
- _log_requirement(ws, requirement)
-
- for dist in self._get_dist(requirement, ws, self._always_unzip
- ):
-
- ws.add(dist)
- self._maybe_add_setuptools(ws, dist)
- except pkg_resources.VersionConflict, err:
- raise VersionConflict(err, ws)
- else:
- break
-
+ env = pkg_resources.Environment(ws.entries)
+ while requirements:
+ # Process dependencies breadth-first.
+ req = self._constrain(requirements.pop(0))
+ if req in processed:
+ # Ignore cyclic or redundant dependencies.
+ continue
+ dist = best.get(req.key)
+ if dist is None:
+ # Find the best distribution and add it to the map.
+ dist = ws.by_key.get(req.key)
+ if dist is None:
+ try:
+ dist = best[req.key] = env.best_match(req, ws)
+ except pkg_resources.VersionConflict, err:
+ raise VersionConflict(err, ws)
+ if dist is None:
+ if destination:
+ logger.debug('Getting required %r', str(req))
+ else:
+ logger.debug('Adding required %r', str(req))
+ _log_requirement(ws, req)
+ for dist in self._get_dist(req,
+ ws, self._always_unzip):
+ ws.add(dist)
+ self._maybe_add_setuptools(ws, dist)
+ if dist not in req:
+ # Oops, the "best" so far conflicts with a dependency.
+ raise VersionConflict(
+ pkg_resources.VersionConflict(dist, req), ws)
+ requirements.extend(dist.requires(req.extras)[::-1])
+ processed[req] = True
+ if dist.location in self._site_packages:
+ logger.debug('Egg from site-packages: %s', dist)
return ws
def build(self, spec, build_ext):
@@ -771,6 +985,18 @@
Installer._prefer_final = bool(setting)
return old
+def include_site_packages(setting=None):
+ old = Installer._include_site_packages
+ if setting is not None:
+ Installer._include_site_packages = bool(setting)
+ return old
+
+def allowed_eggs_from_site_packages(setting=None):
+ old = Installer._allowed_eggs_from_site_packages
+ if setting is not None:
+ Installer._allowed_eggs_from_site_packages = tuple(setting)
+ return old
+
def use_dependency_links(setting=None):
old = Installer._use_dependency_links
if setting is not None:
@@ -793,19 +1019,27 @@
links=(), index=None,
executable=sys.executable, always_unzip=None,
path=None, working_set=None, newest=True, versions=None,
- use_dependency_links=None, allow_hosts=('*',)):
+ use_dependency_links=None, allow_hosts=('*',),
+ include_site_packages=None, allowed_eggs_from_site_packages=None):
installer = Installer(dest, links, index, executable, always_unzip, path,
newest, versions, use_dependency_links,
- allow_hosts=allow_hosts)
+ allow_hosts=allow_hosts,
+ include_site_packages=include_site_packages,
+ allowed_eggs_from_site_packages=
+ allowed_eggs_from_site_packages)
return installer.install(specs, working_set)
def build(spec, dest, build_ext,
links=(), index=None,
executable=sys.executable,
- path=None, newest=True, versions=None, allow_hosts=('*',)):
+ path=None, newest=True, versions=None, allow_hosts=('*',),
+ include_site_packages=None, allowed_eggs_from_site_packages=None):
installer = Installer(dest, links, index, executable, True, path, newest,
- versions, allow_hosts=allow_hosts)
+ versions, allow_hosts=allow_hosts,
+ include_site_packages=include_site_packages,
+ allowed_eggs_from_site_packages=
+ allowed_eggs_from_site_packages)
return installer.build(spec, build_ext)
@@ -900,9 +1134,15 @@
undo.reverse()
[f() for f in undo]
+def working_set(specs, executable, path, include_site_packages=None,
+ allowed_eggs_from_site_packages=None):
+ return install(
+ specs, None, executable=executable, path=path,
+ include_site_packages=include_site_packages,
+ allowed_eggs_from_site_packages=allowed_eggs_from_site_packages)
-def working_set(specs, executable, path):
- return install(specs, None, executable=executable, path=path)
+############################################################################
+# Script generation functions
def scripts(reqs, working_set, executable, dest,
scripts=None,
@@ -912,20 +1152,95 @@
initialization='',
relative_paths=False,
):
+ """Generate scripts and/or an interpreter.
+ See sitepackage_safe_scripts for a version that can be used with a Python
+ that has code installed in site-packages. It has more options and a
+ different approach.
+ """
+ path = _get_path(working_set, extra_paths)
+ if initialization:
+ initialization = '\n'+initialization+'\n'
+ generated = _generate_scripts(
+ reqs, working_set, dest, path, scripts, relative_paths,
+ initialization, executable, arguments)
+ if interpreter:
+ sname = os.path.join(dest, interpreter)
+ spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths)
+ generated.extend(
+ _pyscript(spath, sname, executable, rpsetup))
+ return generated
+
+def sitepackage_safe_scripts(
+ dest, working_set, executable, site_py_dest,
+ reqs=(), scripts=None, interpreter=None, extra_paths=(),
+ initialization='', include_site_packages=False, exec_sitecustomize=False,
+ relative_paths=False, script_arguments='', script_initialization=''):
+ """Generate scripts and/or an interpreter from a system Python.
+
+ This accomplishes the same job as the ``scripts`` function, above,
+ but it does so in an alternative way that allows safely including
+ Python site packages, if desired, and choosing to execute the Python's
+ sitecustomize.
+ """
+ generated = []
+ generated.append(_generate_sitecustomize(
+ site_py_dest, executable, initialization, exec_sitecustomize))
+ generated.append(_generate_site(
+ site_py_dest, working_set, executable, extra_paths,
+ include_site_packages, relative_paths))
+ script_initialization = _script_initialization_template % dict(
+ site_py_dest=site_py_dest,
+ script_initialization=script_initialization)
+ if not script_initialization.endswith('\n'):
+ script_initialization += '\n'
+ generated.extend(_generate_scripts(
+ reqs, working_set, dest, [site_py_dest], scripts, relative_paths,
+ script_initialization, executable, script_arguments, block_site=True))
+ if interpreter:
+ generated.extend(_generate_interpreter(
+ interpreter, dest, executable, site_py_dest, relative_paths))
+ return generated
+
+_script_initialization_template = '''
+import os
+path = sys.path[0]
+if os.environ.get('PYTHONPATH'):
+ path = os.pathsep.join([path, os.environ['PYTHONPATH']])
+os.environ['PYTHONPATH'] = path
+import site # imports custom buildout-generated site.py
+%(script_initialization)s'''
+
+# Utilities for the script generation functions.
+
+# These are shared by both ``scripts`` and ``sitepackage_safe_scripts``
+
+def _get_path(working_set, extra_paths=()):
+ """Given working set and extra paths, return a normalized path list."""
path = [dist.location for dist in working_set]
path.extend(extra_paths)
- path = map(realpath, path)
+ return map(realpath, path)
- generated = []
+def _generate_scripts(reqs, working_set, dest, path, scripts, relative_paths,
+ initialization, executable, arguments,
+ block_site=False):
+ """Generate scripts for the given requirements.
+ - reqs is an iterable of string requirements or entry points.
+ - The requirements must be findable in the given working_set.
+ - The dest is the directory in which the scripts should be created.
+ - The path is a list of paths that should be added to sys.path.
+ - The scripts is an optional dictionary. If included, the keys should be
+ the names of the scripts that should be created, as identified in their
+ entry points; and the values should be the name the script should
+ actually be created with.
+ - relative_paths, if given, should be the path that is the root of the
+ buildout (the common path that should be the root of what is relative).
+ """
if isinstance(reqs, str):
raise TypeError('Expected iterable of requirements or entry points,'
' got string.')
-
- if initialization:
- initialization = '\n'+initialization+'\n'
-
+ generated = []
entry_points = []
for req in reqs:
if isinstance(req, str):
@@ -939,7 +1254,6 @@
)
else:
entry_points.append(req)
-
for name, module_name, attrs in entry_points:
if scripts is not None:
sname = scripts.get(name)
@@ -947,40 +1261,51 @@
continue
else:
sname = name
-
sname = os.path.join(dest, sname)
spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths)
-
generated.extend(
- _script(module_name, attrs, spath, sname, executable, arguments,
- initialization, rpsetup)
- )
+ _script(sname, executable, rpsetup, spath, initialization,
+ module_name, attrs, arguments, block_site=block_site))
+ return generated
- if interpreter:
- sname = os.path.join(dest, interpreter)
- spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths)
- generated.extend(_pyscript(spath, sname, executable, rpsetup))
+def _relative_path_and_setup(sname, path,
+ relative_paths=False, indent_level=1,
+ omit_os_import=False):
+ """Return a string of code of paths and of setup if appropriate.
- return generated
-
-def _relative_path_and_setup(sname, path, relative_paths):
+ - sname is the full path to the script name to be created.
+ - path is the list of paths to be added to sys.path.
+ - relative_paths, if given, should be the path that is the root of the
+ buildout (the common path that should be the root of what is relative).
+ - indent_level is the number of four-space indents that the path should
+ insert before each element of the path.
+ """
if relative_paths:
relative_paths = os.path.normcase(relative_paths)
sname = os.path.normcase(os.path.abspath(sname))
- spath = ',\n '.join(
+ spath = _format_paths(
[_relativitize(os.path.normcase(path_item), sname, relative_paths)
- for path_item in path]
- )
+ for path_item in path], indent_level=indent_level)
rpsetup = relative_paths_setup
+ if not omit_os_import:
+ rpsetup = '\n\nimport os\n' + rpsetup
for i in range(_relative_depth(relative_paths, sname)):
- rpsetup += "base = os.path.dirname(base)\n"
+ rpsetup += "\nbase = os.path.dirname(base)"
else:
- spath = repr(path)[1:-1].replace(', ', ',\n ')
+ spath = _format_paths((repr(p) for p in path),
+ indent_level=indent_level)
rpsetup = ''
return spath, rpsetup
+def _relative_depth(common, path):
+ """Return number of dirs separating ``path`` from ancestor, ``common``.
-def _relative_depth(common, path):
+ For instance, if path is /foo/bar/baz/bing, and common is /foo, this will
+ return 2--in UNIX, the number of ".." to get from bing's directory
+ to foo.
+
+ This is a helper for _relative_path_and_setup.
+ """
n = 0
while 1:
dirname = os.path.dirname(path)
@@ -993,6 +1318,11 @@
return n
def _relative_path(common, path):
+ """Return the relative path from ``common`` to ``path``.
+
+ This is a helper for _relativitize, which is a helper to
+ _relative_path_and_setup.
+ """
r = []
while 1:
dirname, basename = os.path.split(path)
@@ -1006,6 +1336,11 @@
return os.path.join(*r)
def _relativitize(path, script, relative_paths):
+ """Return a code string for the given path.
+
+ Path is relative to the base path ``relative_paths``if the common prefix
+ between ``path`` and ``script`` starts with ``relative_paths``.
+ """
if path == script:
raise AssertionError("path == script")
common = os.path.dirname(os.path.commonprefix([path, script]))
@@ -1016,66 +1351,82 @@
else:
return repr(path)
-
relative_paths_setup = """
-import os
-
join = os.path.join
-base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
-"""
+base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))"""
-def _script(module_name, attrs, path, dest, executable, arguments,
- initialization, rsetup):
+def _write_script(full_name, contents, logged_type):
+ """Write contents of script in full_name, logging the action.
+
+ The only tricky bit in this function is that it supports Windows by
+ creating exe files using a pkg_resources helper.
+ """
generated = []
- script = dest
+ script_name = full_name
if is_win32:
- dest += '-script.py'
-
- contents = script_template % dict(
- python = _safe_arg(executable),
- path = path,
- module_name = module_name,
- attrs = attrs,
- arguments = arguments,
- initialization = initialization,
- relative_paths_setup = rsetup,
- )
- changed = not (os.path.exists(dest) and open(dest).read() == contents)
-
- if is_win32:
- # generate exe file and give the script a magic name:
- exe = script+'.exe'
+ script_name += '-script.py'
+ # Generate exe file and give the script a magic name.
+ exe = full_name + '.exe'
new_data = pkg_resources.resource_string('setuptools', 'cli.exe')
if not os.path.exists(exe) or (open(exe, 'rb').read() != new_data):
# Only write it if it's different.
open(exe, 'wb').write(new_data)
generated.append(exe)
-
+ changed = not (os.path.exists(script_name) and
+ open(script_name).read() == contents)
if changed:
- open(dest, 'w').write(contents)
- logger.info("Generated script %r.", script)
-
+ open(script_name, 'w').write(contents)
try:
- os.chmod(dest, 0755)
+ os.chmod(script_name, 0755)
except (AttributeError, os.error):
pass
-
- generated.append(dest)
+ logger.info("Generated %s %r.", logged_type, full_name)
+ generated.append(script_name)
return generated
+def _format_paths(paths, indent_level=1):
+ """Format paths for inclusion in a script."""
+ separator = ',\n' + indent_level * ' '
+ return separator.join(paths)
+
+def _script(dest, executable, relative_paths_setup, path, initialization,
+ module_name, attrs, arguments, block_site=False):
+ if block_site:
+ dash_S = ' -S'
+ else:
+ dash_S = ''
+ contents = script_template % dict(
+ python=_safe_arg(executable),
+ dash_S=dash_S,
+ path=path,
+ module_name=module_name,
+ attrs=attrs,
+ arguments=arguments,
+ initialization=initialization,
+ relative_paths_setup=relative_paths_setup,
+ )
+ return _write_script(dest, contents, 'script')
+
if is_jython and jython_os_name == 'linux':
- script_header = '#!/usr/bin/env %(python)s'
+ script_header = '#!/usr/bin/env %(python)s%(dash_S)s'
else:
- script_header = '#!%(python)s'
+ script_header = '#!%(python)s%(dash_S)s'
+sys_path_template = '''\
+import sys
+sys.path[0:0] = [
+ %s,
+ ]
+'''
script_template = script_header + '''\
+%(relative_paths_setup)s
-%(relative_paths_setup)s
import sys
sys.path[0:0] = [
- %(path)s,
- ]
+ %(path)s,
+ ]
+
%(initialization)s
import %(module_name)s
@@ -1083,47 +1434,25 @@
%(module_name)s.%(attrs)s(%(arguments)s)
'''
+# These are used only by the older ``scripts`` function.
def _pyscript(path, dest, executable, rsetup):
- generated = []
- script = dest
- if is_win32:
- dest += '-script.py'
-
contents = py_script_template % dict(
- python = _safe_arg(executable),
- path = path,
- relative_paths_setup = rsetup,
+ python=_safe_arg(executable),
+ dash_S='',
+ path=path,
+ relative_paths_setup=rsetup,
)
- changed = not (os.path.exists(dest) and open(dest).read() == contents)
+ return _write_script(dest, contents, 'interpreter')
- if is_win32:
- # generate exe file and give the script a magic name:
- exe = script + '.exe'
- open(exe, 'wb').write(
- pkg_resources.resource_string('setuptools', 'cli.exe')
- )
- generated.append(exe)
-
- if changed:
- open(dest, 'w').write(contents)
- try:
- os.chmod(dest,0755)
- except (AttributeError, os.error):
- pass
- logger.info("Generated interpreter %r.", script)
-
- generated.append(dest)
- return generated
-
py_script_template = script_header + '''\
+%(relative_paths_setup)s
-%(relative_paths_setup)s
import sys
sys.path[0:0] = [
- %(path)s,
- ]
+ %(path)s,
+ ]
_interactive = True
if len(sys.argv) > 1:
@@ -1151,6 +1480,235 @@
__import__("code").interact(banner="", local=globals())
'''
+# These are used only by the newer ``sitepackage_safe_scripts`` function.
+
+def _get_module_file(executable, name):
+ """Return a module's file path.
+
+ - executable is a path to the desired Python executable.
+ - name is the name of the (pure, not C) Python module.
+ """
+ cmd = [executable, "-Sc",
+ "import imp; "
+ "fp, path, desc = imp.find_module(%r); "
+ "fp.close; "
+ "print path" % (name,)]
+ env = os.environ.copy()
+ # We need to make sure that PYTHONPATH, which will often be set to
+ # include a custom buildout-generated site.py, is not set, or else
+ # we will not get an accurate value for the "real" site.py and
+ # sitecustomize.py.
+ env.pop('PYTHONPATH', None)
+ _proc = subprocess.Popen(
+ cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
+ stdout, stderr = _proc.communicate();
+ if _proc.returncode:
+ logger.info(
+ 'Could not find file for module %s:\n%s', name, stderr)
+ return None
+ # else: ...
+ res = stdout.strip()
+ if res.endswith('.pyc') or res.endswith('.pyo'):
+ raise RuntimeError('Cannot find uncompiled version of %s' % (name,))
+ if not os.path.exists(res):
+ raise RuntimeError(
+ 'File does not exist for module %s:\n%s' % (name, res))
+ return res
+
+def _generate_sitecustomize(dest, executable, initialization='',
+ exec_sitecustomize=False):
+ """Write a sitecustomize file with optional custom initialization.
+
+ The created script will execute the underlying Python's
+ sitecustomize if exec_sitecustomize is True.
+ """
+ sitecustomize_path = os.path.join(dest, 'sitecustomize.py')
+ sitecustomize = open(sitecustomize_path, 'w')
+ if initialization:
+ sitecustomize.write(initialization + '\n')
+ if exec_sitecustomize:
+ real_sitecustomize_path = _get_module_file(
+ executable, 'sitecustomize')
+ if real_sitecustomize_path:
+ real_sitecustomize = open(real_sitecustomize_path, 'r')
+ sitecustomize.write(
+ '\n# The following is from\n# %s\n' %
+ (real_sitecustomize_path,))
+ sitecustomize.write(real_sitecustomize.read())
+ real_sitecustomize.close()
+ sitecustomize.close()
+ return sitecustomize_path
+
+def _generate_site(dest, working_set, executable, extra_paths=(),
+ include_site_packages=False, relative_paths=False):
+ """Write a site.py file with eggs from working_set.
+
+ extra_paths will be added to the path. If include_site_packages is True,
+ paths from the underlying Python will be added.
+ """
+ path = _get_path(working_set, extra_paths)
+ site_path = os.path.join(dest, 'site.py')
+ egg_path_string, preamble = _relative_path_and_setup(
+ site_path, path, relative_paths, indent_level=2, omit_os_import=True)
+ if preamble:
+ preamble = '\n'.join(
+ [(line and ' %s' % (line,) or line)
+ for line in preamble.split('\n')])
+ original_path_setup = ''
+ if include_site_packages:
+ stdlib, site_paths = _get_system_paths(executable)
+ original_path_setup = original_path_snippet % (
+ _format_paths((repr(p) for p in site_paths), 2),)
+ distribution = working_set.find(
+ pkg_resources.Requirement.parse('setuptools'))
+ if distribution is not None:
+ # We need to worry about namespace packages.
+ if relative_paths:
+ location = _relativitize(
+ distribution.location,
+ os.path.normcase(os.path.abspath(site_path)),
+ relative_paths)
+ else:
+ location = repr(distribution.location)
+ preamble += namespace_include_site_packages_setup % (location,)
+ original_path_setup = (
+ addsitedir_namespace_originalpackages_snippet +
+ original_path_setup)
+ addsitepackages_marker = 'def addsitepackages('
+ enableusersite_marker = 'ENABLE_USER_SITE = '
+ successful_rewrite = False
+ real_site_path = _get_module_file(executable, 'site')
+ real_site = open(real_site_path, 'r')
+ site = open(site_path, 'w')
+ try:
+ for line in real_site.readlines():
+ if line.startswith(enableusersite_marker):
+ site.write(enableusersite_marker)
+ site.write('False # buildout does not support user sites.\n')
+ elif line.startswith(addsitepackages_marker):
+ site.write(addsitepackages_script % (
+ preamble, egg_path_string, original_path_setup))
+ site.write(line[len(addsitepackages_marker):])
+ successful_rewrite = True
+ else:
+ site.write(line)
+ finally:
+ site.close()
+ real_site.close()
+ if not successful_rewrite:
+ raise RuntimeError(
+ 'Buildout did not successfully rewrite %s to %s' %
+ (real_site_path, site_path))
+ return site_path
+
+namespace_include_site_packages_setup = '''
+ setuptools_path = %s
+ sys.path.append(setuptools_path)
+ known_paths.add(os.path.normcase(setuptools_path))
+ import pkg_resources'''
+
+addsitedir_namespace_originalpackages_snippet = '''
+ pkg_resources.working_set.add_entry(sitedir)'''
+
+original_path_snippet = '''
+ sys.__egginsert = len(buildout_paths) # Support distribute.
+ original_paths = [
+ %s
+ ]
+ for path in original_paths:
+ addsitedir(path, known_paths)'''
+
+addsitepackages_script = '''\
+def addsitepackages(known_paths):
+ """Add site packages, as determined by zc.buildout.
+
+ See original_addsitepackages, below, for the original version."""%s
+ buildout_paths = [
+ %s
+ ]
+ for path in buildout_paths:
+ sitedir, sitedircase = makepath(path)
+ if not sitedircase in known_paths and os.path.exists(sitedir):
+ sys.path.append(sitedir)
+ known_paths.add(sitedircase)%s
+ return known_paths
+
+def original_addsitepackages('''
+
+def _generate_interpreter(name, dest, executable, site_py_dest,
+ relative_paths=False):
+ """Write an interpreter script, using the site.py approach."""
+ full_name = os.path.join(dest, name)
+ site_py_dest_string, rpsetup = _relative_path_and_setup(
+ full_name, [site_py_dest], relative_paths, omit_os_import=True)
+ if rpsetup:
+ rpsetup += "\n"
+ if sys.platform == 'win32':
+ windows_import = '\nimport subprocess'
+ # os.exec* is a mess on Windows, particularly if the path
+ # to the executable has spaces and the Python is using MSVCRT.
+ # The standard fix is to surround the executable's path with quotes,
+ # but that has been unreliable in testing.
+ #
+ # Here's a demonstration of the problem. Given a Python
+ # compiled with a MSVCRT-based compiler, such as the free Visual
+ # C++ 2008 Express Edition, and an executable path with spaces
+ # in it such as the below, we see the following.
+ #
+ # >>> import os
+ # >>> p0 = 'C:\\Documents and Settings\\Administrator\\My Documents\\Downloads\\Python-2.6.4\\PCbuild\\python.exe'
+ # >>> os.path.exists(p0)
+ # True
+ # >>> os.execv(p0, [])
+ # Traceback (most recent call last):
+ # File "<stdin>", line 1, in <module>
+ # OSError: [Errno 22] Invalid argument
+ #
+ # That seems like a standard problem. The standard solution is
+ # to quote the path (see, for instance
+ # http://bugs.python.org/issue436259). However, this solution,
+ # and other variations, fail:
+ #
+ # >>> p1 = '"C:\\Documents and Settings\\Administrator\\My Documents\\Downloads\\Python-2.6.4\\PCbuild\\python.exe"'
+ # >>> os.execv(p1, [])
+ # Traceback (most recent call last):
+ # File "<stdin>", line 1, in <module>
+ # OSError: [Errno 22] Invalid argument
+ #
+ # We simply use subprocess instead, since it handles everything
+ # nicely, and the transparency of exec* (that is, not running,
+ # perhaps unexpectedly, in a subprocess) is arguably not a
+ # necessity, at least for many use cases.
+ execute = 'subprocess.call(argv, env=environ)'
+ else:
+ windows_import = ''
+ execute = 'os.execve(sys.executable, argv, environ)'
+ contents = interpreter_template % dict(
+ python=_safe_arg(executable),
+ dash_S=' -S',
+ site_dest=site_py_dest_string,
+ relative_paths_setup=rpsetup,
+ windows_import=windows_import,
+ execute=execute,
+ )
+ return _write_script(full_name, contents, 'interpreter')
+
+interpreter_template = script_header + '''
+import os
+import sys%(windows_import)s
+%(relative_paths_setup)s
+argv = [sys.executable] + sys.argv[1:]
+environ = os.environ.copy()
+path = %(site_dest)s
+if environ.get('PYTHONPATH'):
+ path = os.pathsep.join([path, environ['PYTHONPATH']])
+environ['PYTHONPATH'] = path
+%(execute)s
+'''
+
+# End of script generation code.
+############################################################################
+
runsetup_template = """
import sys
sys.path.insert(0, %(setupdir)r)
Modified: zc.buildout/trunk/src/zc/buildout/easy_install.txt
===================================================================
--- zc.buildout/trunk/src/zc/buildout/easy_install.txt 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/src/zc/buildout/easy_install.txt 2010-04-29 18:11:58 UTC (rev 111588)
@@ -89,6 +89,14 @@
for using dependency_links in preference to other
locations. Defaults to true.
+include_site_packages
+ A flag indicating whether Python's non-standard-library packages should
+ be available for finding dependencies. Defaults to true.
+
+ Paths outside of Python's standard library--or more precisely, those that
+ are not included when Python is started with the -S argument--are loosely
+ referred to as "site-packages" here.
+
relative_paths
Adjust egg paths so they are relative to the script path. This
allows scripts to work when scripts and eggs are moved, as long as
@@ -214,7 +222,9 @@
d other-1.0-py2.4.egg
We can request that eggs be unzipped even if they are zip safe. This
-can be useful when debugging.
+can be useful when debugging. (Note that Distribute will unzip eggs by
+default, so if you are using Distribute, most or all eggs will already be
+unzipped without this flag.)
>>> rmdir(dest)
>>> dest = tmpdir('sample-install')
@@ -399,6 +409,68 @@
>>> [d.version for d in ws]
['0.3', '1.1']
+Dependencies in Site Packages
+-----------------------------
+
+Paths outside of Python's standard library--or more precisely, those that are
+not included when Python is started with the -S argument--are loosely referred
+to as "site-packages" here. These site-packages are searched by default for
+distributions. This can be disabled, so that, for instance, a system Python
+can be used with buildout, cleaned of any packages installed by a user or
+system package manager.
+
+The default behavior can be controlled and introspected using
+zc.buildout.easy_install.include_site_packages.
+
+ >>> zc.buildout.easy_install.include_site_packages()
+ True
+
+Here's an example of using a Python executable that includes our dependencies.
+
+Our "py_path" will have the "demoneeded," and "demo" packages available.
+ We'll simply be asking for "demoneeded" here, but without any external
+ index or links.
+
+ >>> from zc.buildout.tests import create_sample_sys_install
+ >>> py_path, site_packages_path = make_py()
+ >>> create_sample_sys_install(site_packages_path)
+
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['demoneeded'], example_dest, links=[], executable=py_path,
+ ... index=None)
+ >>> [dist.project_name for dist in workingset]
+ ['demoneeded']
+
+That worked fine. Let's try again with site packages not allowed. We'll
+change the policy by changing the default. Notice that the function for
+changing the default value returns the previous value.
+
+ >>> zc.buildout.easy_install.include_site_packages(False)
+ True
+
+ >>> zc.buildout.easy_install.include_site_packages()
+ False
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['demoneeded'], example_dest, links=[], executable=py_path,
+ ... index=None)
+ Traceback (most recent call last):
+ ...
+ MissingDistribution: Couldn't find a distribution for 'demoneeded'.
+ >>> zc.buildout.easy_install.clear_index_cache()
+
+Now we'll reset the default.
+
+ >>> zc.buildout.easy_install.include_site_packages(True)
+ False
+
+ >>> zc.buildout.easy_install.include_site_packages()
+ True
+
Dependency links
----------------
@@ -521,25 +593,38 @@
Script generation
-----------------
-The easy_install module provides support for creating scripts from
-eggs. It provides a function similar to setuptools except that it
-provides facilities for baking a script's path into the script. This
-has two advantages:
+The easy_install module provides support for creating scripts from eggs.
+It provides two competing functions. One, ``scripts``, is a
+well-established approach to generating reliable scripts with a "clean"
+Python--e.g., one that does not have any packages in its site-packages.
+The other, ``sitepackage_safe_scripts``, is newer, a bit trickier, and is
+designed to work with a Python that has code in its site-packages, such
+as a system Python.
+Both are similar to setuptools except that they provides facilities for
+baking a script's path into the script. This has two advantages:
+
- The eggs to be used by a script are not chosen at run time, making
startup faster and, more importantly, deterministic.
-- The script doesn't have to import pkg_resources because the logic
- that pkg_resources would execute at run time is executed at
- script-creation time.
+- The script doesn't have to import pkg_resources because the logic that
+ pkg_resources would execute at run time is executed at script-creation
+ time. (There is an exception in ``sitepackage_safe_scripts`` if you
+ want to have your Python's site packages available, as discussed
+ below, but even in that case pkg_resources is only partially
+ activated, which can be a significant time savings.)
-The scripts method can be used to generate scripts. Let's create a
-destination directory for it to place them in:
- >>> import tempfile
+The ``scripts`` function
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``scripts`` function is the first way to generate scripts that we'll
+examine. It is the earlier approach that the package offered. Let's
+create a destination directory for it to place them in:
+
>>> bin = tmpdir('bin')
-Now, we'll use the scripts method to generate scripts in this directory
+Now, we'll use the scripts function to generate scripts in this directory
from the demo egg:
>>> import sys
@@ -736,8 +821,8 @@
>>> print system(os.path.join(bin, 'run')),
3 1
-Including extra paths in scripts
---------------------------------
+The ``scripts`` function: Including extra paths in scripts
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We can pass a keyword argument, extra paths, to cause additional paths
to be included in the a generated script:
@@ -762,8 +847,8 @@
if __name__ == '__main__':
eggrecipedemo.main()
-Providing script arguments
---------------------------
+The ``scripts`` function: Providing script arguments
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
An "argument" keyword argument can be used to pass arguments to an
entry point. The value passed is a source string to be placed between the
@@ -786,8 +871,8 @@
if __name__ == '__main__':
eggrecipedemo.main(1, 2)
-Passing initialization code
----------------------------
+The ``scripts`` function: Passing initialization code
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can also pass script initialization code:
@@ -812,8 +897,8 @@
if __name__ == '__main__':
eggrecipedemo.main(1, 2)
-Relative paths
---------------
+The ``scripts`` function: Relative paths
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sometimes, you want to be able to move a buildout directory around and
have scripts still work without having to rebuild them. We can
@@ -836,7 +921,7 @@
... interpreter='py',
... relative_paths=bo)
- >>> cat(bo, 'bin', 'run')
+ >>> cat(bo, 'bin', 'run') # doctest: +NORMALIZE_WHITESPACE
#!/usr/local/bin/python2.4
<BLANKLINE>
import os
@@ -868,7 +953,7 @@
We specified an interpreter and its paths are adjusted too:
- >>> cat(bo, 'bin', 'py')
+ >>> cat(bo, 'bin', 'py') # doctest: +NORMALIZE_WHITESPACE
#!/usr/local/bin/python2.4
<BLANKLINE>
import os
@@ -911,7 +996,571 @@
del _interactive
__import__("code").interact(banner="", local=globals())
+The ``sitepackage_safe_scripts`` function
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The newer function for creating scripts is ``sitepackage_safe_scripts``.
+It has the same basic functionality as the ``scripts`` function: it can
+create scripts to run arbitrary entry points, and to run a Python
+interpreter. The following are the differences from a user's
+perspective.
+
+- It can be used safely with a Python that has packages installed itself,
+ such as a system-installed Python.
+
+- In contrast to the interpreter generated by the ``scripts`` method, which
+ supports only a small subset of the usual Python executable's options,
+ the interpreter generated by ``sitepackage_safe_scripts`` supports all
+ of them. This makes it possible to use as full Python replacement for
+ scripts that need the distributions specified in your buildout.
+
+- Both the interpreter and the entry point scripts allow you to include the
+ site packages, and/or the sitecustomize, of the Python executable, if
+ desired.
+
+It works by creating site.py and sitecustomize.py files that set up the
+desired paths and initialization. These must be placed within an otherwise
+empty directory. Typically this is in a recipe's parts directory.
+
+Here's the simplest example, building an interpreter script.
+
+ >>> interpreter_dir = tmpdir('interpreter')
+ >>> interpreter_parts_dir = os.path.join(
+ ... interpreter_dir, 'parts', 'interpreter')
+ >>> interpreter_bin_dir = os.path.join(interpreter_dir, 'bin')
+ >>> mkdir(interpreter_bin_dir)
+ >>> mkdir(interpreter_dir, 'eggs')
+ >>> mkdir(interpreter_dir, 'parts')
+ >>> mkdir(interpreter_parts_dir)
+
+ >>> ws = zc.buildout.easy_install.install(
+ ... ['demo'], join(interpreter_dir, 'eggs'), links=[link_server],
+ ... index=link_server+'index/')
+ >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... interpreter='py')
+
+Depending on whether the machine being used is running Windows or not, this
+produces either three or four files. In both cases, we have site.py and
+sitecustomize.py generated in the parts/interpreter directory. For Windows,
+we have py.exe and py-script.py; for other operating systems, we have py.
+
+ >>> sitecustomize_path = os.path.join(
+ ... interpreter_parts_dir, 'sitecustomize.py')
+ >>> site_path = os.path.join(interpreter_parts_dir, 'site.py')
+ >>> interpreter_path = os.path.join(interpreter_bin_dir, 'py')
+ >>> if sys.platform == 'win32':
+ ... py_path = os.path.join(interpreter_bin_dir, 'py-script.py')
+ ... expected = [sitecustomize_path,
+ ... site_path,
+ ... os.path.join(interpreter_bin_dir, 'py.exe'),
+ ... py_path]
+ ... else:
+ ... py_path = interpreter_path
+ ... expected = [sitecustomize_path, site_path, py_path]
+ ...
+ >>> assert generated == expected, repr((generated, expected))
+
+We didn't ask for any initialization, and we didn't ask to use the underlying
+sitecustomization, so sitecustomize.py is empty.
+
+ >>> cat(sitecustomize_path)
+
+The interpreter script is simple. It puts the directory with the
+site.py and sitecustomize.py on the PYTHONPATH and (re)starts Python.
+
+ >>> cat(py_path)
+ #!/usr/bin/python -S
+ import os
+ import sys
+ <BLANKLINE>
+ argv = [sys.executable] + sys.argv[1:]
+ environ = os.environ.copy()
+ path = '/interpreter/parts/interpreter'
+ if environ.get('PYTHONPATH'):
+ path = os.pathsep.join([path, environ['PYTHONPATH']])
+ environ['PYTHONPATH'] = path
+ os.execve(sys.executable, argv, environ)
+
+The site.py file is a modified version of the underlying Python's site.py.
+The most important modification is that it has a different version of the
+addsitepackages function. It sets up the Python path, similarly to the
+behavior of the function it replaces. The following shows the part that
+buildout inserts, in the simplest case.
+
+ >>> sys.stdout.write('#\n'); cat(site_path)
+ ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
+ #...
+ def addsitepackages(known_paths):
+ """Add site packages, as determined by zc.buildout.
+ <BLANKLINE>
+ See original_addsitepackages, below, for the original version."""
+ buildout_paths = [
+ '/interpreter/eggs/demo-0.3-pyN.N.egg',
+ '/interpreter/eggs/demoneeded-1.1-pyN.N.egg'
+ ]
+ for path in buildout_paths:
+ sitedir, sitedircase = makepath(path)
+ if not sitedircase in known_paths and os.path.exists(sitedir):
+ sys.path.append(sitedir)
+ known_paths.add(sitedircase)
+ return known_paths
+ <BLANKLINE>
+ def original_addsitepackages(known_paths):...
+
+Here are some examples of the interpreter in use.
+
+ >>> print call_py(interpreter_path, "print 16+26")
+ 42
+ <BLANKLINE>
+ >>> res = call_py(interpreter_path, "import sys; print sys.path")
+ >>> print res # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
+ ['',
+ '/interpreter/parts/interpreter',
+ ...,
+ '/interpreter/eggs/demo-0.3-pyN.N.egg',
+ '/interpreter/eggs/demoneeded-1.1-pyN.N.egg']
+ <BLANKLINE>
+ >>> clean_paths = eval(res.strip()) # This is used later for comparison.
+
+If you provide initialization, it goes in sitecustomize.py.
+
+ >>> def reset_interpreter():
+ ... # This is necessary because, in our tests, the timestamps of the
+ ... # .pyc files are not outdated when we want them to be.
+ ... rmdir(interpreter_bin_dir)
+ ... mkdir(interpreter_bin_dir)
+ ... rmdir(interpreter_parts_dir)
+ ... mkdir(interpreter_parts_dir)
+ ...
+ >>> reset_interpreter()
+
+ >>> initialization_string = """\
+ ... import os
+ ... os.environ['FOO'] = 'bar baz bing shazam'"""
+ >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... interpreter='py', initialization=initialization_string)
+ >>> cat(sitecustomize_path)
+ import os
+ os.environ['FOO'] = 'bar baz bing shazam'
+ >>> print call_py(interpreter_path, "import os; print os.environ['FOO']")
+ bar baz bing shazam
+ <BLANKLINE>
+
+If you use relative paths, this affects the interpreter and site.py. (This is
+again the UNIX version; the Windows version uses subprocess instead of
+os.execve.)
+
+ >>> reset_interpreter()
+ >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... interpreter='py', relative_paths=interpreter_dir)
+ >>> cat(py_path)
+ #!/usr/bin/python -S
+ import os
+ import sys
+ <BLANKLINE>
+ join = os.path.join
+ base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
+ base = os.path.dirname(base)
+ <BLANKLINE>
+ argv = [sys.executable] + sys.argv[1:]
+ environ = os.environ.copy()
+ path = join(base, 'parts/interpreter')
+ if environ.get('PYTHONPATH'):
+ path = os.pathsep.join([path, environ['PYTHONPATH']])
+ environ['PYTHONPATH'] = path
+ os.execve(sys.executable, argv, environ)
+
+For site.py, we again show only the pertinent parts. Notice that the egg
+paths join a base to a path, as with the use of this argument in the
+``scripts`` function.
+
+ >>> sys.stdout.write('#\n'); cat(site_path) # doctest: +ELLIPSIS
+ #...
+ def addsitepackages(known_paths):
+ """Add site packages, as determined by zc.buildout.
+ <BLANKLINE>
+ See original_addsitepackages, below, for the original version."""
+ join = os.path.join
+ base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
+ base = os.path.dirname(base)
+ base = os.path.dirname(base)
+ buildout_paths = [
+ join(base, 'eggs/demo-0.3-pyN.N.egg'),
+ join(base, 'eggs/demoneeded-1.1-pyN.N.egg')
+ ]...
+
+The paths resolve in practice as you would expect.
+
+ >>> print call_py(interpreter_path,
+ ... "import sys, pprint; pprint.pprint(sys.path)")
+ ... # doctest: +ELLIPSIS
+ ['',
+ '/interpreter/parts/interpreter',
+ ...,
+ '/interpreter/eggs/demo-0.3-pyN.N.egg',
+ '/interpreter/eggs/demoneeded-1.1-pyN.N.egg']
+ <BLANKLINE>
+
+The ``extra_paths`` argument affects the path in site.py. Notice that
+/interpreter/other is added after the eggs.
+
+ >>> reset_interpreter()
+ >>> mkdir(interpreter_dir, 'other')
+ >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... interpreter='py', extra_paths=[join(interpreter_dir, 'other')])
+ >>> sys.stdout.write('#\n'); cat(site_path) # doctest: +ELLIPSIS
+ #...
+ def addsitepackages(known_paths):
+ """Add site packages, as determined by zc.buildout.
+ <BLANKLINE>
+ See original_addsitepackages, below, for the original version."""
+ buildout_paths = [
+ '/interpreter/eggs/demo-0.3-pyN.N.egg',
+ '/interpreter/eggs/demoneeded-1.1-pyN.N.egg',
+ '/interpreter/other'
+ ]...
+
+ >>> print call_py(interpreter_path,
+ ... "import sys, pprint; pprint.pprint(sys.path)")
+ ... # doctest: +ELLIPSIS
+ ['',
+ '/interpreter/parts/interpreter',
+ ...,
+ '/interpreter/eggs/demo-0.3-pyN.N.egg',
+ '/interpreter/eggs/demoneeded-1.1-pyN.N.egg',
+ '/interpreter/other']
+ <BLANKLINE>
+
+The ``sitepackage_safe_scripts`` function: using site-packages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``sitepackage_safe_scripts`` function supports including site
+packages. This has some advantages and some serious dangers.
+
+A typical reason to include site-packages is that it is easier to
+install one or more dependencies in your Python than it is with
+buildout. Some packages, such as lxml or Python PostgreSQL integration,
+have dependencies that can be much easier to build and/or install using
+other mechanisms, such as your operating system's package manager. By
+installing some core packages into your Python's site-packages, this can
+significantly simplify some application installations.
+
+However, doing this has a significant danger. One of the primary goals
+of buildout is to provide repeatability. Some packages (one of the
+better known Python openid packages, for instance) change their behavior
+depending on what packages are available. If Python curl bindings are
+available, these may be preferred by the library. If a certain XML
+package is installed, it may be preferred by the library. These hidden
+choices may cause small or large behavior differences. The fact that
+they can be rarely encountered can actually make it worse: you forget
+that this might be a problem, and debugging the differences can be
+difficult. If you allow site-packages to be included in your buildout,
+and the Python you use is not managed precisely by your application (for
+instance, it is a system Python), you open yourself up to these
+possibilities. Don't be unaware of the dangers.
+
+That explained, let's see how it works. If you don't use namespace packages,
+this is very straightforward.
+
+ >>> reset_interpreter()
+ >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... interpreter='py', include_site_packages=True)
+ >>> sys.stdout.write('#\n'); cat(site_path)
+ ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
+ #...
+ def addsitepackages(known_paths):
+ """Add site packages, as determined by zc.buildout.
+ <BLANKLINE>
+ See original_addsitepackages, below, for the original version."""
+ buildout_paths = [
+ '/interpreter/eggs/demo-0.3-pyN.N.egg',
+ '/interpreter/eggs/demoneeded-1.1-pyN.N.egg'
+ ]
+ for path in buildout_paths:
+ sitedir, sitedircase = makepath(path)
+ if not sitedircase in known_paths and os.path.exists(sitedir):
+ sys.path.append(sitedir)
+ known_paths.add(sitedircase)
+ sys.__egginsert = len(buildout_paths) # Support distribute.
+ original_paths = [
+ ...
+ ]
+ for path in original_paths:
+ addsitedir(path, known_paths)
+ return known_paths
+ <BLANKLINE>
+ def original_addsitepackages(known_paths):...
+
+It simply adds the original paths using addsitedir after the code to add the
+buildout paths.
+
+Here's an example of the new script in use. Other documents and tests in
+this package give the feature a more thorough workout, but this should
+give you an idea of the feature.
+
+ >>> res = call_py(interpreter_path, "import sys; print sys.path")
+ >>> print res # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
+ ['',
+ '/interpreter/parts/interpreter',
+ ...,
+ '/interpreter/eggs/demo-0.3-py2.4.egg',
+ '/interpreter/eggs/demoneeded-1.1-py2.4.egg',
+ ...]
+ <BLANKLINE>
+
+The clean_paths gathered earlier is a subset of this full list of paths.
+
+ >>> full_paths = eval(res.strip())
+ >>> len(clean_paths) < len(full_paths)
+ True
+ >>> set(os.path.normpath(p) for p in clean_paths).issubset(
+ ... os.path.normpath(p) for p in full_paths)
+ True
+
+Unfortunately, because of how setuptools namespace packages are implemented
+differently for operating system packages (debs or rpms) as opposed to
+standard setuptools installation, there's a slightly trickier dance if you
+use them. To show this we'll needs some extra eggs that use namespaces.
+We'll use the ``tellmy.fortune`` package, which we'll need to make an initial
+call to another text fixture to create.
+
+ >>> from zc.buildout.tests import create_sample_namespace_eggs
+ >>> namespace_eggs = tmpdir('namespace_eggs')
+ >>> create_sample_namespace_eggs(namespace_eggs)
+
+ >>> reset_interpreter()
+ >>> ws = zc.buildout.easy_install.install(
+ ... ['demo', 'tellmy.fortune'], join(interpreter_dir, 'eggs'),
+ ... links=[link_server, namespace_eggs], index=link_server+'index/')
+ >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... interpreter='py', include_site_packages=True)
+ >>> sys.stdout.write('#\n'); cat(site_path)
+ ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
+ #...
+ def addsitepackages(known_paths):
+ """Add site packages, as determined by zc.buildout.
+ <BLANKLINE>
+ See original_addsitepackages, below, for the original version."""
+ setuptools_path = '...setuptools...'
+ sys.path.append(setuptools_path)
+ known_paths.add(os.path.normcase(setuptools_path))
+ import pkg_resources
+ buildout_paths = [
+ '/interpreter/eggs/demo-0.3-pyN.N.egg',
+ '/interpreter/eggs/tellmy.fortune-1.0-pyN.N.egg',
+ '...setuptools...',
+ '/interpreter/eggs/demoneeded-1.1-pyN.N.egg'
+ ]
+ for path in buildout_paths:
+ sitedir, sitedircase = makepath(path)
+ if not sitedircase in known_paths and os.path.exists(sitedir):
+ sys.path.append(sitedir)
+ known_paths.add(sitedircase)
+ pkg_resources.working_set.add_entry(sitedir)
+ sys.__egginsert = len(buildout_paths) # Support distribute.
+ original_paths = [
+ ...
+ ]
+ for path in original_paths:
+ addsitedir(path, known_paths)
+ return known_paths
+ <BLANKLINE>
+ def original_addsitepackages(known_paths):...
+
+ >>> print call_py(interpreter_path, "import sys; print sys.path")
+ ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
+ ['',
+ '/interpreter/parts/interpreter',
+ ...,
+ '...setuptools...',
+ '/interpreter/eggs/demo-0.3-pyN.N.egg',
+ '/interpreter/eggs/tellmy.fortune-1.0-pyN.N.egg',
+ '/interpreter/eggs/demoneeded-1.1-pyN.N.egg',
+ ...]
+
+As you can see, the script now first imports pkg_resources. Then we
+need to process egg files specially to look for namespace packages there
+*before* we process process lines in .pth files that use the "import"
+feature--lines that might be part of the setuptools namespace package
+implementation for system packages, as mentioned above, and that must
+come after processing egg namespaces.
+
+The most complex that this function gets is if you use namespace packages,
+include site-packages, and use relative paths. For completeness, we'll look
+at that result.
+
+ >>> reset_interpreter()
+ >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... interpreter='py', include_site_packages=True,
+ ... relative_paths=interpreter_dir)
+ >>> sys.stdout.write('#\n'); cat(site_path)
+ ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
+ #...
+ def addsitepackages(known_paths):
+ """Add site packages, as determined by zc.buildout.
+ <BLANKLINE>
+ See original_addsitepackages, below, for the original version."""
+ join = os.path.join
+ base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
+ base = os.path.dirname(base)
+ base = os.path.dirname(base)
+ setuptools_path = '...setuptools...'
+ sys.path.append(setuptools_path)
+ known_paths.add(os.path.normcase(setuptools_path))
+ import pkg_resources
+ buildout_paths = [
+ join(base, 'eggs/demo-0.3-pyN.N.egg'),
+ join(base, 'eggs/tellmy.fortune-1.0-pyN.N.egg'),
+ '...setuptools...',
+ join(base, 'eggs/demoneeded-1.1-pyN.N.egg')
+ ]
+ for path in buildout_paths:
+ sitedir, sitedircase = makepath(path)
+ if not sitedircase in known_paths and os.path.exists(sitedir):
+ sys.path.append(sitedir)
+ known_paths.add(sitedircase)
+ pkg_resources.working_set.add_entry(sitedir)
+ sys.__egginsert = len(buildout_paths) # Support distribute.
+ original_paths = [
+ ...
+ ]
+ for path in original_paths:
+ addsitedir(path, known_paths)
+ return known_paths
+ <BLANKLINE>
+ def original_addsitepackages(known_paths):...
+
+ >>> print call_py(interpreter_path, "import sys; print sys.path")
+ ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
+ ['',
+ '/interpreter/parts/interpreter',
+ ...,
+ '...setuptools...',
+ '/interpreter/eggs/demo-0.3-pyN.N.egg',
+ '/interpreter/eggs/tellmy.fortune-1.0-pyN.N.egg',
+ '/interpreter/eggs/demoneeded-1.1-pyN.N.egg',
+ ...]
+
+The ``exec_sitecustomize`` argument does the same thing for the
+sitecustomize module--it allows you to include the code from the
+sitecustomize module in the underlying Python if you set the argument to
+True. The z3c.recipe.scripts package sets up the full environment necessary
+to demonstrate this piece.
+
+The ``sitepackage_safe_scripts`` function: writing scripts for entry points
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+All of the examples so far for this function have been creating
+interpreters. The function can also write scripts for entry
+points. They are almost identical to the scripts that we saw for the
+``scripts`` function except that they ``import site`` after setting the
+sys.path to include our custom site.py and sitecustomize.py files. These
+files then initialize the Python environment as we have already seen. Let's
+see a simple example.
+
+ >>> reset_interpreter()
+ >>> ws = zc.buildout.easy_install.install(
+ ... ['demo'], join(interpreter_dir, 'eggs'), links=[link_server],
+ ... index=link_server+'index/')
+ >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... reqs=['demo'])
+
+As before, in Windows, 2 files are generated for each script. A script
+file, ending in '-script.py', and an exe file that allows the script
+to be invoked directly without having to specify the Python
+interpreter and without having to provide a '.py' suffix. This is in addition
+to the site.py and sitecustomize.py files that are generated as with our
+interpreter examples above.
+
+ >>> if sys.platform == 'win32':
+ ... demo_path = os.path.join(interpreter_bin_dir, 'demo-script.py')
+ ... expected = [sitecustomize_path,
+ ... site_path,
+ ... os.path.join(interpreter_bin_dir, 'demo.exe'),
+ ... demo_path]
+ ... else:
+ ... demo_path = os.path.join(interpreter_bin_dir, 'demo')
+ ... expected = [sitecustomize_path, site_path, demo_path]
+ ...
+ >>> assert generated == expected, repr((generated, expected))
+
+The demo script runs the entry point defined in the demo egg:
+
+ >>> cat(demo_path) # doctest: +NORMALIZE_WHITESPACE
+ #!/usr/local/bin/python2.4 -S
+ <BLANKLINE>
+ import sys
+ sys.path[0:0] = [
+ '/interpreter/parts/interpreter',
+ ]
+ <BLANKLINE>
+ <BLANKLINE>
+ import os
+ path = sys.path[0]
+ if os.environ.get('PYTHONPATH'):
+ path = os.pathsep.join([path, os.environ['PYTHONPATH']])
+ os.environ['PYTHONPATH'] = path
+ import site # imports custom buildout-generated site.py
+ <BLANKLINE>
+ import eggrecipedemo
+ <BLANKLINE>
+ if __name__ == '__main__':
+ eggrecipedemo.main()
+
+ >>> demo_call = join(interpreter_bin_dir, 'demo')
+ >>> if sys.platform == 'win32':
+ ... demo_call = '"%s"' % demo_call
+ >>> print system(demo_call)
+ 3 1
+ <BLANKLINE>
+
+There are a few differences from the ``scripts`` function. First, the
+``reqs`` argument (an iterable of string requirements or entry point
+tuples) is a keyword argument here. We see that in the example above.
+Second, the ``arguments`` argument is now named ``script_arguments`` to
+try and clarify that it does not affect interpreters. While the
+``initialization`` argument continues to affect both the interpreters
+and the entry point scripts, if you have initialization that is only
+pertinent to the entry point scripts, you can use the
+``script_initialization`` argument.
+
+Let's see ``script_arguments`` and ``script_initialization`` in action.
+
+ >>> reset_interpreter()
+ >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... reqs=['demo'], script_arguments='1, 2',
+ ... script_initialization='import os\nos.chdir("foo")')
+
+ >>> cat(demo_path) # doctest: +NORMALIZE_WHITESPACE
+ #!/usr/local/bin/python2.4 -S
+ import sys
+ sys.path[0:0] = [
+ '/interpreter/parts/interpreter',
+ ]
+ <BLANKLINE>
+ import os
+ path = sys.path[0]
+ if os.environ.get('PYTHONPATH'):
+ path = os.pathsep.join([path, os.environ['PYTHONPATH']])
+ os.environ['PYTHONPATH'] = path
+ import site # imports custom buildout-generated site.py
+ import os
+ os.chdir("foo")
+ <BLANKLINE>
+ import eggrecipedemo
+ <BLANKLINE>
+ if __name__ == '__main__':
+ eggrecipedemo.main(1, 2)
+
Handling custom build options for extensions provided in source distributions
-----------------------------------------------------------------------------
Modified: zc.buildout/trunk/src/zc/buildout/testing.py
===================================================================
--- zc.buildout/trunk/src/zc/buildout/testing.py 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/src/zc/buildout/testing.py 2010-04-29 18:11:58 UTC (rev 111588)
@@ -28,6 +28,7 @@
import subprocess
import sys
import tempfile
+import textwrap
import threading
import time
import urllib2
@@ -109,6 +110,16 @@
e.close()
return result
+def call_py(interpreter, cmd, flags=None):
+ if sys.platform == 'win32':
+ args = ['"%s"' % arg for arg in (interpreter, flags, cmd) if arg]
+ args.insert(-1, '"-c"')
+ return system('"%s"' % ' '.join(args))
+ else:
+ cmd = repr(cmd)
+ return system(
+ ' '.join(arg for arg in (interpreter, flags, '-c', cmd) if arg))
+
def get(url):
return urllib2.urlopen(url).read()
@@ -120,7 +131,11 @@
args = [zc.buildout.easy_install._safe_arg(arg)
for arg in args]
args.insert(0, '-q')
- args.append(dict(os.environ, PYTHONPATH=setuptools_location))
+ env = dict(os.environ)
+ if executable == sys.executable:
+ env['PYTHONPATH'] = setuptools_location
+ # else pass an executable that has setuptools! See testselectingpython.py.
+ args.append(env)
here = os.getcwd()
try:
@@ -139,6 +154,11 @@
def bdist_egg(setup, executable, dest):
_runsetup(setup, executable, 'bdist_egg', '-d', dest)
+def sys_install(setup, dest):
+ _runsetup(setup, sys.executable, 'install', '--install-purelib', dest,
+ '--record', os.path.join(dest, '__added_files__'),
+ '--single-version-externally-managed')
+
def find_python(version):
e = os.environ.get('PYTHON%s' % version)
if e is not None:
@@ -206,14 +226,64 @@
time.sleep(0.01)
raise ValueError('Timed out waiting for: '+label)
+def get_installer_values():
+ """Get the current values for the easy_install module.
+
+ This is necessary because instantiating a Buildout will force the
+ Buildout's values on the installer.
+
+ Returns a dict of names-values suitable for set_installer_values."""
+ names = ('default_versions', 'download_cache', 'install_from_cache',
+ 'prefer_final', 'include_site_packages',
+ 'allowed_eggs_from_site_packages', 'use_dependency_links',
+ 'allow_picked_versions', 'always_unzip'
+ )
+ values = {}
+ for name in names:
+ values[name] = getattr(zc.buildout.easy_install, name)()
+ return values
+
+def set_installer_values(values):
+ """Set the given values on the installer."""
+ for name, value in values.items():
+ getattr(zc.buildout.easy_install, name)(value)
+
+def make_buildout(executable=None):
+ """Make a buildout that uses this version of zc.buildout."""
+ # Create a basic buildout.cfg to avoid a warning from buildout.
+ open('buildout.cfg', 'w').write(
+ "[buildout]\nparts =\n"
+ )
+ # Get state of installer defaults so we can reinstate them (instantiating
+ # a Buildout will force the Buildout's defaults on the installer).
+ installer_values = get_installer_values()
+ # Use the buildout bootstrap command to create a buildout
+ config = [
+ ('buildout', 'log-level', 'WARNING'),
+ # trick bootstrap into putting the buildout develop egg
+ # in the eggs dir.
+ ('buildout', 'develop-eggs-directory', 'eggs'),
+ ]
+ if executable is not None:
+ config.append(('buildout', 'executable', executable))
+ zc.buildout.buildout.Buildout(
+ 'buildout.cfg', config,
+ user_defaults=False,
+ ).bootstrap([])
+ # Create the develop-eggs dir, which didn't get created the usual
+ # way due to the trick above:
+ os.mkdir('develop-eggs')
+ # Reinstate the default values of the installer.
+ set_installer_values(installer_values)
+
def buildoutSetUp(test):
test.globs['__tear_downs'] = __tear_downs = []
test.globs['register_teardown'] = register_teardown = __tear_downs.append
- prefer_final = zc.buildout.easy_install.prefer_final()
+ installer_values = get_installer_values()
register_teardown(
- lambda: zc.buildout.easy_install.prefer_final(prefer_final)
+ lambda: set_installer_values(installer_values)
)
here = os.getcwd()
@@ -259,34 +329,58 @@
sample = tmpdir('sample-buildout')
os.chdir(sample)
+ make_buildout()
- # Create a basic buildout.cfg to avoid a warning from buildout:
- open('buildout.cfg', 'w').write(
- "[buildout]\nparts =\n"
- )
-
- # Use the buildout bootstrap command to create a buildout
- zc.buildout.buildout.Buildout(
- 'buildout.cfg',
- [('buildout', 'log-level', 'WARNING'),
- # trick bootstrap into putting the buildout develop egg
- # in the eggs dir.
- ('buildout', 'develop-eggs-directory', 'eggs'),
- ]
- ).bootstrap([])
-
-
-
- # Create the develop-eggs dir, which didn't get created the usual
- # way due to the trick above:
- os.mkdir('develop-eggs')
-
def start_server(path):
port, thread = _start_server(path, name=path)
url = 'http://localhost:%s/' % port
register_teardown(lambda: stop_server(url, thread))
return url
+ def make_py(initialization=''):
+ """Returns paths to new executable and to its site-packages.
+ """
+ buildout = tmpdir('executable_buildout')
+ site_packages_dir = os.path.join(buildout, 'site-packages')
+ mkdir(site_packages_dir)
+ old_wd = os.getcwd()
+ os.chdir(buildout)
+ make_buildout()
+ # Normally we don't process .pth files in extra-paths. We want to
+ # in this case so that we can test with setuptools system installs
+ # (--single-version-externally-managed), which use .pth files.
+ initialization = (
+ ('import sys\n'
+ 'import site\n'
+ 'known_paths = set(sys.path)\n'
+ 'site_packages_dir = %r\n'
+ 'site.addsitedir(site_packages_dir, known_paths)\n'
+ ) % (site_packages_dir,)) + initialization
+ initialization = '\n'.join(
+ ' ' + line for line in initialization.split('\n'))
+ install_develop(
+ 'zc.recipe.egg', os.path.join(buildout, 'develop-eggs'))
+ install_develop(
+ 'z3c.recipe.scripts', os.path.join(buildout, 'develop-eggs'))
+ write('buildout.cfg', textwrap.dedent('''\
+ [buildout]
+ parts = py
+
+ [py]
+ recipe = z3c.recipe.scripts
+ interpreter = py
+ initialization =
+ %(initialization)s
+ extra-paths = %(site-packages)s
+ eggs = setuptools
+ ''') % {
+ 'initialization': initialization,
+ 'site-packages': site_packages_dir})
+ system(os.path.join(buildout, 'bin', 'buildout'))
+ os.chdir(old_wd)
+ return (
+ os.path.join(buildout, 'bin', 'py'), site_packages_dir)
+
test.globs.update(dict(
sample_buildout = sample,
ls = ls,
@@ -297,6 +391,7 @@
tmpdir = tmpdir,
write = write,
system = system,
+ call_py = call_py,
get = get,
cd = (lambda *path: os.chdir(os.path.join(*path))),
join = os.path.join,
@@ -305,10 +400,9 @@
start_server = start_server,
buildout = os.path.join(sample, 'bin', 'buildout'),
wait_until = wait_until,
+ make_py = make_py
))
- zc.buildout.easy_install.prefer_final(prefer_final)
-
def buildoutTearDown(test):
for f in test.globs['__tear_downs']:
f()
Modified: zc.buildout/trunk/src/zc/buildout/tests.py
===================================================================
--- zc.buildout/trunk/src/zc/buildout/tests.py 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/src/zc/buildout/tests.py 2010-04-29 18:11:58 UTC (rev 111588)
@@ -53,6 +53,7 @@
>>> ls('develop-eggs')
- foo.egg-link
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
"""
@@ -84,6 +85,7 @@
>>> ls('develop-eggs')
- foo.egg-link
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
>>> print system(join('bin', 'buildout')+' -vvv'), # doctest: +ELLIPSIS
@@ -383,7 +385,65 @@
Error: Couldn't find a distribution for 'demoneeded'.
"""
+def show_eggs_from_site_packages():
+ """
+Sometimes you want to know what eggs are coming from site-packages. This
+might be for a diagnostic, or so that you can get a starting value for the
+allowed-eggs-from-site-packages option. The -v flag will also include this
+information.
+Our "py_path" has the "demoneeded," "demo"
+packages available. We'll ask for "bigdemo," which will get both of them.
+
+Here's our set up.
+
+ >>> py_path, site_packages_path = make_py()
+ >>> create_sample_sys_install(site_packages_path)
+
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... parts = eggs
+ ... prefer-final = true
+ ... find-links = %(link_server)s
+ ...
+ ... [primed_python]
+ ... executable = %(py_path)s
+ ...
+ ... [eggs]
+ ... recipe = zc.recipe.egg:eggs
+ ... python = primed_python
+ ... eggs = bigdemo
+ ... ''' % globals())
+
+Now here is the output. The lines that begin with "Egg from site-packages:"
+indicate the eggs from site-packages that have been selected. You'll see
+we have two: demo 0.3 and demoneeded 1.1.
+
+ >>> print system(buildout+" -v")
+ Installing 'zc.buildout', 'setuptools'.
+ We have a develop egg: zc.buildout V
+ We have the best distribution that satisfies 'setuptools'.
+ Picked: setuptools = V
+ Installing 'zc.recipe.egg'.
+ We have a develop egg: zc.recipe.egg V
+ Installing eggs.
+ Installing 'bigdemo'.
+ We have no distributions for bigdemo that satisfies 'bigdemo'.
+ Getting distribution for 'bigdemo'.
+ Got bigdemo 0.1.
+ Picked: bigdemo = 0.1
+ Getting required 'demo'
+ required by bigdemo 0.1.
+ We have a develop egg: demo V
+ Egg from site-packages: demo 0.3
+ Getting required 'demoneeded'
+ required by demo 0.3.
+ We have a develop egg: demoneeded V
+ Egg from site-packages: demoneeded 1.1
+ <BLANKLINE>
+ """
+
def test_comparing_saved_options_with_funny_characters():
"""
If an option has newlines, extra/odd spaces or a %, we need to make sure
@@ -668,6 +728,7 @@
>>> ls('develop-eggs')
- foox.egg-link
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
Create another:
@@ -692,6 +753,7 @@
>>> ls('develop-eggs')
- foox.egg-link
- fooy.egg-link
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
Remove one:
@@ -709,6 +771,7 @@
>>> ls('develop-eggs')
- fooy.egg-link
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
Remove the other:
@@ -723,6 +786,7 @@
All gone
>>> ls('develop-eggs')
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
'''
@@ -797,9 +861,11 @@
... + join(sample_buildout, 'eggs'))
>>> ls('develop-eggs')
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
- >>> ls('eggs') # doctest: +ELLIPSIS
+ >>> print 'START ->'; ls('eggs') # doctest: +ELLIPSIS
+ START...
- foox-0.0.0-py2.4.egg
...
@@ -1769,6 +1835,658 @@
1 2
"""
+def versions_section_ignored_for_dependency_in_favor_of_site_packages():
+ r"""
+This is a test for a bugfix.
+
+The error showed itself when at least two dependencies were in a shared
+location like site-packages, and the first one met the "versions" setting. The
+first dependency would be added, but subsequent dependencies from the same
+location (e.g., site-packages) would use the version of the package found in
+the shared location, ignoring the version setting.
+
+We begin with a Python that has demoneeded version 1.1 installed and a
+demo version 0.3, all in a site-packages-like shared directory. We need
+to create this. ``eggrecipedemo.main()`` shows the number after the dot
+(that is, ``X`` in ``1.X``), for the demo package and the demoneeded
+package, so this demonstrates that our Python does in fact have demo
+version 0.3 and demoneeded version 1.1.
+
+ >>> py_path = make_py_with_system_install(make_py, sample_eggs)
+ >>> print call_py(
+ ... py_path,
+ ... "import tellmy.version; print tellmy.version.__version__"),
+ 1.1
+
+Now here's a setup that would expose the bug, using the
+zc.buildout.easy_install API.
+
+ >>> example_dest = tmpdir('example_dest')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['tellmy.version'], example_dest, links=[sample_eggs],
+ ... executable=py_path,
+ ... index=None,
+ ... versions={'tellmy.version': '1.0'})
+ >>> for dist in workingset:
+ ... res = str(dist)
+ ... if res.startswith('tellmy.version'):
+ ... print res
+ ... break
+ tellmy.version 1.0
+
+Before the bugfix, the desired tellmy.version distribution would have
+been blocked the one in site-packages.
+"""
+
+def handle_namespace_package_in_both_site_packages_and_buildout_eggs():
+ r"""
+If you have the same namespace package in both site-packages and in
+buildout, we need to be very careful that faux-Python-executables and
+scripts generated by easy_install.sitepackage_safe_scripts correctly
+combine the two. We show this with the local recipe that uses the
+function, z3c.recipe.scripts.
+
+To demonstrate this, we will create three packages: tellmy.version 1.0,
+tellmy.version 1.1, and tellmy.fortune 1.0. tellmy.version 1.1 is installed.
+
+ >>> py_path = make_py_with_system_install(make_py, sample_eggs)
+ >>> print call_py(
+ ... py_path,
+ ... "import tellmy.version; print tellmy.version.__version__")
+ 1.1
+ <BLANKLINE>
+
+Now we will create a buildout that creates a script and a faux-Python script.
+We want to see that both can successfully import the specified versions of
+tellmy.version and tellmy.fortune.
+
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... parts = eggs
+ ... find-links = %(link_server)s
+ ...
+ ... [primed_python]
+ ... executable = %(py_path)s
+ ...
+ ... [eggs]
+ ... recipe = z3c.recipe.scripts
+ ... python = primed_python
+ ... interpreter = py
+ ... include-site-packages = true
+ ... eggs = tellmy.version == 1.0
+ ... tellmy.fortune == 1.0
+ ... demo
+ ... script-initialization =
+ ... import tellmy.version
+ ... print tellmy.version.__version__
+ ... import tellmy.fortune
+ ... print tellmy.fortune.__version__
+ ... ''' % globals())
+
+ >>> print system(buildout)
+ Installing eggs.
+ Getting distribution for 'tellmy.version==1.0'.
+ Got tellmy.version 1.0.
+ Getting distribution for 'tellmy.fortune==1.0'.
+ Got tellmy.fortune 1.0.
+ Getting distribution for 'demo'.
+ Got demo 0.4c1.
+ Getting distribution for 'demoneeded'.
+ Got demoneeded 1.2c1.
+ Generated script '/sample-buildout/bin/demo'.
+ Generated interpreter '/sample-buildout/bin/py'.
+ <BLANKLINE>
+
+Finally, we are ready to see if it worked. Prior to the bug fix that
+this tests, the results of both calls below was the following::
+
+ 1.1
+ Traceback (most recent call last):
+ ...
+ ImportError: No module named fortune
+ <BLANKLINE>
+
+In other words, we got the site-packages version of tellmy.version, and
+we could not import tellmy.fortune at all. The following are the correct
+results for the interpreter and for the script.
+
+ >>> print call_py(
+ ... join('bin', 'py'),
+ ... "import tellmy.version; " +
+ ... "print tellmy.version.__version__; " +
+ ... "import tellmy.fortune; " +
+ ... "print tellmy.fortune.__version__") # doctest: +ELLIPSIS
+ 1.0
+ 1.0...
+
+ >>> print system(join('bin', 'demo'))
+ 1.0
+ 1.0
+ 4 2
+ <BLANKLINE>
+ """
+
+def handle_sys_path_version_hack():
+ r"""
+This is a test for a bugfix.
+
+If you use a Python that has a different version of one of your
+dependencies, and the new package tries to do sys.path tricks in the
+setup.py to get a __version__, and it uses namespace packages, the older
+package will be loaded first, making the setup version the wrong number.
+While very arguably packages simply shouldn't do this, some do, and we
+don't want buildout to fall over when they do.
+
+To demonstrate this, we will need to create a distribution that has one of
+these unpleasant tricks, and a Python that has an older version installed.
+
+ >>> py_path, site_packages_path = make_py()
+ >>> for version in ('1.0', '1.1'):
+ ... tmp = tempfile.mkdtemp()
+ ... try:
+ ... write(tmp, 'README.txt', '')
+ ... mkdir(tmp, 'src')
+ ... mkdir(tmp, 'src', 'tellmy')
+ ... write(tmp, 'src', 'tellmy', '__init__.py',
+ ... "__import__("
+ ... "'pkg_resources').declare_namespace(__name__)\n")
+ ... mkdir(tmp, 'src', 'tellmy', 'version')
+ ... write(tmp, 'src', 'tellmy', 'version',
+ ... '__init__.py', '__version__=%r\n' % version)
+ ... write(
+ ... tmp, 'setup.py',
+ ... "from setuptools import setup\n"
+ ... "import sys\n"
+ ... "sys.path.insert(0, 'src')\n"
+ ... "from tellmy.version import __version__\n"
+ ... "setup(\n"
+ ... " name='tellmy.version',\n"
+ ... " package_dir = {'': 'src'},\n"
+ ... " packages = ['tellmy', 'tellmy.version'],\n"
+ ... " install_requires = ['setuptools'],\n"
+ ... " namespace_packages=['tellmy'],\n"
+ ... " zip_safe=True, version=__version__,\n"
+ ... " author='bob', url='bob', author_email='bob')\n"
+ ... )
+ ... zc.buildout.testing.sdist(tmp, sample_eggs)
+ ... if version == '1.0':
+ ... # We install the 1.0 version in site packages the way a
+ ... # system packaging system (debs, rpms) would do it.
+ ... zc.buildout.testing.sys_install(tmp, site_packages_path)
+ ... finally:
+ ... shutil.rmtree(tmp)
+ >>> print call_py(
+ ... py_path,
+ ... "import tellmy.version; print tellmy.version.__version__")
+ 1.0
+ <BLANKLINE>
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... parts = eggs
+ ... find-links = %(sample_eggs)s
+ ...
+ ... [primed_python]
+ ... executable = %(py_path)s
+ ...
+ ... [eggs]
+ ... recipe = zc.recipe.egg:eggs
+ ... python = primed_python
+ ... eggs = tellmy.version == 1.1
+ ... ''' % globals())
+
+Before the bugfix, running this buildout would generate this error:
+
+ Installing eggs.
+ Getting distribution for 'tellmy.version==1.1'.
+ Installing tellmy.version 1.1
+ Caused installation of a distribution:
+ tellmy.version 1.0
+ with a different version.
+ Got None.
+ While:
+ Installing eggs.
+ Error: There is a version conflict.
+ We already have: tellmy.version 1.0
+ <BLANKLINE>
+
+You can see the copiously commented fix for this in easy_install.py (see
+zc.buildout.easy_install.Installer._call_easy_install and particularly
+the comment leading up to zc.buildout.easy_install._easy_install_cmd).
+Now the install works correctly, as seen here.
+
+ >>> print system(buildout)
+ Installing eggs.
+ Getting distribution for 'tellmy.version==1.1'.
+ Got tellmy.version 1.1.
+ <BLANKLINE>
+
+ """
+
+def isolated_include_site_packages():
+ """
+
+This is an isolated test of the include_site_packages functionality, passing
+the argument directly to install, overriding a default.
+
+Our "py_path" has the "demoneeded" and "demo" packages available. We'll
+simply be asking for "demoneeded" here.
+
+ >>> py_path, site_packages_path = make_py()
+ >>> create_sample_sys_install(site_packages_path)
+ >>> zc.buildout.easy_install.include_site_packages(False)
+ True
+
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['demoneeded'], example_dest, links=[], executable=py_path,
+ ... index=None, include_site_packages=True)
+ >>> [dist.project_name for dist in workingset]
+ ['demoneeded']
+
+That worked fine. Let's try again with site packages not allowed (and
+reversing the default).
+
+ >>> zc.buildout.easy_install.include_site_packages(True)
+ False
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['demoneeded'], example_dest, links=[], executable=py_path,
+ ... index=None, include_site_packages=False)
+ Traceback (most recent call last):
+ ...
+ MissingDistribution: Couldn't find a distribution for 'demoneeded'.
+
+That's a failure, as expected.
+
+Now we explore an important edge case.
+
+Some system Pythons include setuptools (and other Python packages) in their
+site-packages (or equivalent) using a .egg-info directory. The pkg_resources
+module (from setuptools) considers a package installed using .egg-info to be a
+develop egg.
+
+zc.buildout.buildout.Buildout.bootstrap will make setuptools and zc.buildout
+available to the buildout via the eggs directory, for normal eggs; or the
+develop-eggs directory, for develop-eggs.
+
+If setuptools or zc.buildout is found in site-packages and considered by
+pkg_resources to be a develop egg, then the bootstrap code will use a .egg-link
+in the local develop-eggs, pointing to site-packages, in its entirety. Because
+develop-eggs must always be available for searching for distributions, this
+indirectly brings site-packages back into the search path for distributions.
+
+Because of this, we have to take special care that we still exclude
+site-packages even in this case. See the comments about site packages in the
+Installer._satisfied and Installer._obtain methods for the implementation
+(as of this writing).
+
+In this demonstration, we insert a link to the "demoneeded" distribution
+in our develop-eggs, which would bring the package back in, except for
+the special care we have taken to exclude it.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> mkdir(example_dest, 'develop-eggs')
+ >>> write(example_dest, 'develop-eggs', 'demoneeded.egg-link',
+ ... site_packages_path)
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['demoneeded'], example_dest, links=[],
+ ... path=[join(example_dest, 'develop-eggs')],
+ ... executable=py_path,
+ ... index=None, include_site_packages=False)
+ Traceback (most recent call last):
+ ...
+ MissingDistribution: Couldn't find a distribution for 'demoneeded'.
+
+The MissingDistribution error shows that buildout correctly excluded the
+"site-packages" source even though it was indirectly included in the path
+via a .egg-link file.
+
+ """
+
+def allowed_eggs_from_site_packages():
+ """
+Sometimes you need or want to control what eggs from site-packages are used.
+The allowed-eggs-from-site-packages option allows you to specify a whitelist of
+project names that may be included from site-packages. You can use globs to
+specify the value. It defaults to a single value of '*', indicating that any
+package may come from site-packages.
+
+This option interacts with include-site-packages in the following ways.
+
+If include-site-packages is true, then allowed-eggs-from-site-packages filters
+what eggs from site-packages may be chosen. If allowed-eggs-from-site-packages
+is an empty list, then no eggs from site-packages are chosen, but site-packages
+will still be included at the end of path lists.
+
+If include-site-packages is false, allowed-eggs-from-site-packages is
+irrelevant.
+
+This test shows the interaction with the zc.buildout.easy_install API. Another
+test below (allow_site_package_eggs_option) shows using it with a buildout.cfg.
+
+Our "py_path" has the "demoneeded" and "demo" packages available. We'll
+simply be asking for "demoneeded" here.
+
+ >>> py_path, site_packages_path = make_py()
+ >>> create_sample_sys_install(site_packages_path)
+
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['demoneeded'], example_dest, links=[], executable=py_path,
+ ... index=None,
+ ... allowed_eggs_from_site_packages=['demoneeded', 'other'])
+ >>> [dist.project_name for dist in workingset]
+ ['demoneeded']
+
+That worked fine. It would work fine for a glob too.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['demoneeded'], example_dest, links=[], executable=py_path,
+ ... index=None,
+ ... allowed_eggs_from_site_packages=['?emon*', 'other'])
+ >>> [dist.project_name for dist in workingset]
+ ['demoneeded']
+
+But now let's try again with 'demoneeded' not allowed.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['demoneeded'], example_dest, links=[], executable=py_path,
+ ... index=None,
+ ... allowed_eggs_from_site_packages=['demo'])
+ Traceback (most recent call last):
+ ...
+ MissingDistribution: Couldn't find a distribution for 'demoneeded'.
+
+Here's the same, but with an empty list.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['demoneeded'], example_dest, links=[], executable=py_path,
+ ... index=None,
+ ... allowed_eggs_from_site_packages=[])
+ Traceback (most recent call last):
+ ...
+ MissingDistribution: Couldn't find a distribution for 'demoneeded'.
+
+Of course, this doesn't stop us from getting a package from elsewhere. Here,
+we add a link server.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['demoneeded'], example_dest, executable=py_path,
+ ... links=[link_server], index=link_server+'index/',
+ ... allowed_eggs_from_site_packages=['other'])
+ >>> [dist.project_name for dist in workingset]
+ ['demoneeded']
+ >>> [dist.location for dist in workingset]
+ ['/site-packages-example-install/demoneeded-1.1-py2.6.egg']
+
+Finally, here's an example of an interaction: we say that it is OK to
+allow the "demoneeded" egg to come from site-packages, but we don't
+include-site-packages.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['demoneeded'], example_dest, links=[], executable=py_path,
+ ... index=None, include_site_packages=False,
+ ... allowed_eggs_from_site_packages=['demoneeded'])
+ Traceback (most recent call last):
+ ...
+ MissingDistribution: Couldn't find a distribution for 'demoneeded'.
+
+ """
+
+def subprocesses_have_same_environment_by_default():
+ """
+The scripts generated by sitepackage_safe_scripts set the PYTHONPATH so that,
+if the environment is maintained (the default behavior), subprocesses get
+the same Python packages.
+
+First, we set up a script and an interpreter.
+
+ >>> interpreter_dir = tmpdir('interpreter')
+ >>> interpreter_parts_dir = os.path.join(
+ ... interpreter_dir, 'parts', 'interpreter')
+ >>> interpreter_bin_dir = os.path.join(interpreter_dir, 'bin')
+ >>> mkdir(interpreter_bin_dir)
+ >>> mkdir(interpreter_dir, 'eggs')
+ >>> mkdir(interpreter_dir, 'parts')
+ >>> mkdir(interpreter_parts_dir)
+ >>> ws = zc.buildout.easy_install.install(
+ ... ['demo'], join(interpreter_dir, 'eggs'), links=[link_server],
+ ... index=link_server+'index/')
+ >>> test = (
+ ... "import subprocess, sys; subprocess.call("
+ ... "[sys.executable, '-c', "
+ ... "'import eggrecipedemo; print eggrecipedemo.x'])")
+ >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... reqs=['demo'], interpreter='py',
+ ... script_initialization=test + '; sys.exit(0)')
+
+This works for the script.
+
+ >>> print system(join(interpreter_bin_dir, 'demo'))
+ 3
+ <BLANKLINE>
+
+This also works for the generated interpreter.
+
+ >>> print call_py(join(interpreter_bin_dir, 'py'), test)
+ 3
+ <BLANKLINE>
+
+If you have a PYTHONPATH in your environment, it will be honored, after
+the buildout-generated path.
+
+ >>> original_pythonpath = os.environ.get('PYTHONPATH')
+ >>> os.environ['PYTHONPATH'] = 'foo'
+ >>> test = (
+ ... "import subprocess, sys; subprocess.call("
+ ... "[sys.executable, '-c', "
+ ... "'import sys, pprint; pprint.pprint(sys.path)'])")
+ >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... reqs=['demo'], interpreter='py',
+ ... script_initialization=test + '; sys.exit(0)')
+
+This works for the script. As you can see, /sample_buildout/foo is included
+right after the "parts" directory that contains site.py and sitecustomize.py.
+You can also see, actually more easily than in the other example, that we
+have the desired eggs available.
+
+ >>> print system(join(interpreter_bin_dir, 'demo')), # doctest: +ELLIPSIS
+ ['',
+ '/interpreter/parts/interpreter',
+ '/sample-buildout/foo',
+ ...
+ '/interpreter/eggs/demo-0.3-pyN.N.egg',
+ '/interpreter/eggs/demoneeded-1.1-pyN.N.egg']
+
+This also works for the generated interpreter, with identical results.
+
+ >>> print call_py(join(interpreter_bin_dir, 'py'), test),
+ ... # doctest: +ELLIPSIS
+ ['',
+ '/interpreter/parts/interpreter',
+ '/sample-buildout/foo',
+ ...
+ '/interpreter/eggs/demo-0.3-pyN.N.egg',
+ '/interpreter/eggs/demoneeded-1.1-pyN.N.egg']
+
+ >>> # Cleanup
+ >>> if original_pythonpath:
+ ... os.environ['PYTHONPATH'] = original_pythonpath
+ ... else:
+ ... del os.environ['PYTHONPATH']
+ ...
+
+ """
+
+def bootstrap_makes_buildout_that_works_with_system_python():
+ r"""
+In order to work smoothly with a system Python, bootstrapping creates
+the buildout script with
+zc.buildout.easy_install.sitepackage_safe_scripts. If it did not, a
+variety of problems might happen. For instance, if another version of
+buildout or setuptools is installed in the site-packages than is
+desired, it may cause a problem.
+
+A problem actually experienced in the field is when
+a recipe wants a different version of a dependency that is installed in
+site-packages. We will create a similar situation, and show that it is now
+handled.
+
+First let's write a dummy recipe.
+
+ >>> mkdir(sample_buildout, 'recipes')
+ >>> write(sample_buildout, 'recipes', 'dummy.py',
+ ... '''
+ ... import logging, os, zc.buildout
+ ... class Dummy:
+ ... def __init__(self, buildout, name, options):
+ ... pass
+ ... def install(self):
+ ... return ()
+ ... def update(self):
+ ... pass
+ ... ''')
+ >>> write(sample_buildout, 'recipes', 'setup.py',
+ ... '''
+ ... from setuptools import setup
+ ...
+ ... setup(
+ ... name = "recipes",
+ ... entry_points = {'zc.buildout': ['dummy = dummy:Dummy']},
+ ... install_requires = 'demoneeded==1.2c1',
+ ... )
+ ... ''')
+ >>> write(sample_buildout, 'recipes', 'README.txt', " ")
+
+Now we'll try to use it with a Python that has a different version of
+demoneeded installed.
+
+ >>> py_path, site_packages_path = make_py()
+ >>> create_sample_sys_install(site_packages_path)
+ >>> rmdir('develop-eggs')
+ >>> from zc.buildout.testing import make_buildout
+ >>> make_buildout(executable=py_path)
+ >>> write(sample_buildout, 'buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... develop = recipes
+ ... parts = dummy
+ ... find-links = %(link_server)s
+ ... executable = %(py_path)s
+ ...
+ ... [dummy]
+ ... recipe = recipes:dummy
+ ... ''' % globals())
+
+Now we actually run the buildout. Before the change, we got the following
+error:
+
+ Develop: '/sample-buildout/recipes'
+ While:
+ Installing.
+ Getting section dummy.
+ Initializing section dummy.
+ Installing recipe recipes.
+ Error: There is a version conflict.
+ We already have: demoneeded 1.1
+ but recipes 0.0.0 requires 'demoneeded==1.2c1'.
+
+Now, it is handled smoothly.
+
+ >>> print system(buildout)
+ Develop: '/sample-buildout/recipes'
+ Getting distribution for 'demoneeded==1.2c1'.
+ Got demoneeded 1.2c1.
+ Installing dummy.
+ <BLANKLINE>
+
+Here's the same story with a namespace package, which has some additional
+complications behind the scenes. First, a recipe, in the "tellmy" namespace.
+
+ >>> mkdir(sample_buildout, 'ns')
+ >>> mkdir(sample_buildout, 'ns', 'tellmy')
+ >>> write(sample_buildout, 'ns', 'tellmy', '__init__.py',
+ ... "__import__('pkg_resources').declare_namespace(__name__)\n")
+ >>> mkdir(sample_buildout, 'ns', 'tellmy', 'recipes')
+ >>> write(sample_buildout, 'ns', 'tellmy', 'recipes', '__init__.py', ' ')
+ >>> write(sample_buildout, 'ns', 'tellmy', 'recipes', 'dummy.py',
+ ... '''
+ ... import logging, os, zc.buildout
+ ... class Dummy:
+ ... def __init__(self, buildout, name, options):
+ ... pass
+ ... def install(self):
+ ... return ()
+ ... def update(self):
+ ... pass
+ ... ''')
+ >>> write(sample_buildout, 'ns', 'setup.py',
+ ... '''
+ ... from setuptools import setup
+ ... setup(
+ ... name="tellmy.recipes",
+ ... packages=['tellmy', 'tellmy.recipes'],
+ ... install_requires=['setuptools'],
+ ... namespace_packages=['tellmy'],
+ ... entry_points = {'zc.buildout':
+ ... ['dummy = tellmy.recipes.dummy:Dummy']},
+ ... )
+ ... ''')
+
+Now, a buildout that uses it.
+
+ >>> create_sample_namespace_eggs(sample_eggs, site_packages_path)
+ >>> rmdir('develop-eggs')
+ >>> from zc.buildout.testing import make_buildout
+ >>> make_buildout(executable=py_path)
+ >>> write(sample_buildout, 'buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... develop = ns
+ ... recipes
+ ... parts = dummy
+ ... find-links = %(link_server)s
+ ... executable = %(py_path)s
+ ...
+ ... [dummy]
+ ... recipe = tellmy.recipes:dummy
+ ... ''' % globals())
+
+Now we actually run the buildout.
+
+ >>> print system(buildout)
+ Develop: '/sample-buildout/ns'
+ Develop: '/sample-buildout/recipes'
+ Uninstalling dummy.
+ Installing dummy.
+ <BLANKLINE>
+
+ """
+
if sys.version_info > (2, 4):
def test_exit_codes():
"""
@@ -2367,6 +3085,7 @@
>>> ls('develop-eggs')
- foo.egg-link
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
"""
@@ -2385,6 +3104,13 @@
>>> write('foo.py', '')
>>> _ = system(buildout+' setup . sdist')
+ >>> if zc.buildout.easy_install.is_distribute:
+ ... distribute_version = 'distribute = %s' % (
+ ... pkg_resources.working_set.find(
+ ... pkg_resources.Requirement.parse('distribute')).version,)
+ ... else:
+ ... distribute_version = ''
+ ...
>>> write('buildout.cfg',
... '''
... [buildout]
@@ -2396,12 +3122,14 @@
... [versions]
... setuptools = %s
... foo = 1
+ ... %s
...
... [foo]
... recipe = zc.recipe.egg
... eggs = foo
- ... ''' % pkg_resources.working_set.find(
- ... pkg_resources.Requirement.parse('setuptools')).version)
+ ... ''' % (pkg_resources.working_set.find(
+ ... pkg_resources.Requirement.parse('setuptools')).version,
+ ... distribute_version))
>>> print system(buildout),
Installing foo.
@@ -2654,26 +3382,102 @@
######################################################################
-def create_sample_eggs(test, executable=sys.executable):
- write = test.globs['write']
- dest = test.globs['sample_eggs']
- tmp = tempfile.mkdtemp()
- try:
- write(tmp, 'README.txt', '')
+def make_py_with_system_install(make_py, sample_eggs):
+ py_path, site_packages_path = make_py()
+ create_sample_namespace_eggs(sample_eggs, site_packages_path)
+ return py_path
- for i in (0, 1, 2):
- write(tmp, 'eggrecipedemoneeded.py', 'y=%s\ndef f():\n pass' % i)
- c1 = i==2 and 'c1' or ''
+def create_sample_namespace_eggs(dest, site_packages_path=None):
+ from zc.buildout.testing import write, mkdir
+ for pkg, version in (('version', '1.0'), ('version', '1.1'),
+ ('fortune', '1.0')):
+ tmp = tempfile.mkdtemp()
+ try:
+ write(tmp, 'README.txt', '')
+ mkdir(tmp, 'src')
+ mkdir(tmp, 'src', 'tellmy')
+ write(tmp, 'src', 'tellmy', '__init__.py',
+ "__import__("
+ "'pkg_resources').declare_namespace(__name__)\n")
+ mkdir(tmp, 'src', 'tellmy', pkg)
+ write(tmp, 'src', 'tellmy', pkg,
+ '__init__.py', '__version__=%r\n' % version)
write(
tmp, 'setup.py',
"from setuptools import setup\n"
- "setup(name='demoneeded', py_modules=['eggrecipedemoneeded'],"
- " zip_safe=True, version='1.%s%s', author='bob', url='bob', "
- "author_email='bob')\n"
- % (i, c1)
+ "setup(\n"
+ " name='tellmy.%(pkg)s',\n"
+ " package_dir = {'': 'src'},\n"
+ " packages = ['tellmy', 'tellmy.%(pkg)s'],\n"
+ " install_requires = ['setuptools'],\n"
+ " namespace_packages=['tellmy'],\n"
+ " zip_safe=True, version=%(version)r,\n"
+ " author='bob', url='bob', author_email='bob')\n"
+ % locals()
)
zc.buildout.testing.sdist(tmp, dest)
+ if (site_packages_path and pkg == 'version' and version == '1.1'):
+ # We install the 1.1 version in site packages the way a
+ # system packaging system (debs, rpms) would do it.
+ zc.buildout.testing.sys_install(tmp, site_packages_path)
+ finally:
+ shutil.rmtree(tmp)
+def _write_eggrecipedemoneeded(tmp, minor_version, suffix=''):
+ from zc.buildout.testing import write
+ write(tmp, 'README.txt', '')
+ write(tmp, 'eggrecipedemoneeded.py',
+ 'y=%s\ndef f():\n pass' % minor_version)
+ write(
+ tmp, 'setup.py',
+ "from setuptools import setup\n"
+ "setup(name='demoneeded', py_modules=['eggrecipedemoneeded'],"
+ " zip_safe=True, version='1.%s%s', author='bob', url='bob', "
+ "author_email='bob')\n"
+ % (minor_version, suffix)
+ )
+
+def _write_eggrecipedemo(tmp, minor_version, suffix=''):
+ from zc.buildout.testing import write
+ write(tmp, 'README.txt', '')
+ write(
+ tmp, 'eggrecipedemo.py',
+ 'import eggrecipedemoneeded\n'
+ 'x=%s\n'
+ 'def main(): print x, eggrecipedemoneeded.y\n'
+ % minor_version)
+ write(
+ tmp, 'setup.py',
+ "from setuptools import setup\n"
+ "setup(name='demo', py_modules=['eggrecipedemo'],"
+ " install_requires = 'demoneeded',"
+ " entry_points={'console_scripts': "
+ "['demo = eggrecipedemo:main']},"
+ " zip_safe=True, version='0.%s%s')\n" % (minor_version, suffix)
+ )
+
+def create_sample_sys_install(site_packages_path):
+ for creator, minor_version in (
+ (_write_eggrecipedemoneeded, 1),
+ (_write_eggrecipedemo, 3)):
+ # Write the files and install in site_packages_path.
+ tmp = tempfile.mkdtemp()
+ try:
+ creator(tmp, minor_version)
+ zc.buildout.testing.sys_install(tmp, site_packages_path)
+ finally:
+ shutil.rmtree(tmp)
+
+def create_sample_eggs(test, executable=sys.executable):
+ from zc.buildout.testing import write
+ dest = test.globs['sample_eggs']
+ tmp = tempfile.mkdtemp()
+ try:
+ for i in (0, 1, 2):
+ suffix = i==2 and 'c1' or ''
+ _write_eggrecipedemoneeded(tmp, i, suffix)
+ zc.buildout.testing.sdist(tmp, dest)
+
write(
tmp, 'setup.py',
"from setuptools import setup\n"
@@ -2685,22 +3489,8 @@
os.remove(os.path.join(tmp, 'eggrecipedemoneeded.py'))
for i in (1, 2, 3, 4):
- write(
- tmp, 'eggrecipedemo.py',
- 'import eggrecipedemoneeded\n'
- 'x=%s\n'
- 'def main(): print x, eggrecipedemoneeded.y\n'
- % i)
- c1 = i==4 and 'c1' or ''
- write(
- tmp, 'setup.py',
- "from setuptools import setup\n"
- "setup(name='demo', py_modules=['eggrecipedemo'],"
- " install_requires = 'demoneeded',"
- " entry_points={'console_scripts': "
- "['demo = eggrecipedemo:main']},"
- " zip_safe=True, version='0.%s%s')\n" % (i, c1)
- )
+ suffix = i==4 and 'c1' or ''
+ _write_eggrecipedemo(tmp, i, suffix)
zc.buildout.testing.bdist_egg(tmp, executable, dest)
write(tmp, 'eggrecipebigdemo.py', 'import eggrecipedemo')
@@ -2776,6 +3566,7 @@
test.globs['sample_eggs'])
test.globs['update_extdemo'] = lambda : add_source_dist(test, 1.5)
zc.buildout.testing.install_develop('zc.recipe.egg', test)
+ zc.buildout.testing.install_develop('z3c.recipe.scripts', test)
egg_parse = re.compile('([0-9a-zA-Z_.]+)-([0-9a-zA-Z_.]+)-py(\d[.]\d).egg$'
).match
@@ -2818,7 +3609,8 @@
here = os.getcwd()
os.chdir(os.path.dirname(dist.location))
assert os.spawnle(
- os.P_WAIT, sys.executable, zc.buildout.easy_install._safe_arg (sys.executable),
+ os.P_WAIT, sys.executable,
+ zc.buildout.easy_install._safe_arg(sys.executable),
os.path.join(os.path.dirname(dist.location), 'setup.py'),
'-q', 'bdist_egg', '-d', eggs,
dict(os.environ,
@@ -2841,13 +3633,15 @@
# now let's make the new releases
makeNewRelease('zc.buildout', ws, new_releases)
- makeNewRelease('setuptools', ws, new_releases)
-
os.mkdir(os.path.join(new_releases, 'zc.buildout'))
- os.mkdir(os.path.join(new_releases, 'setuptools'))
+ if zc.buildout.easy_install.is_distribute:
+ makeNewRelease('distribute', ws, new_releases)
+ os.mkdir(os.path.join(new_releases, 'distribute'))
+ else:
+ makeNewRelease('setuptools', ws, new_releases)
+ os.mkdir(os.path.join(new_releases, 'setuptools'))
-
normalize_bang = (
re.compile(re.escape('#!'+
zc.buildout.easy_install._safe_arg(sys.executable))),
@@ -2869,7 +3663,8 @@
'__buildout_signature__ = recipes-SSSSSSSSSSS'),
(re.compile('executable = [\S ]+python\S*', re.I),
'executable = python'),
- (re.compile('[-d] setuptools-\S+[.]egg'), 'setuptools.egg'),
+ (re.compile('[-d] (setuptools|distribute)-\S+[.]egg'),
+ 'setuptools.egg'),
(re.compile('zc.buildout(-\S+)?[.]egg(-link)?'),
'zc.buildout.egg'),
(re.compile('creating \S*setup.cfg'), 'creating setup.cfg'),
@@ -2884,6 +3679,7 @@
r'when that file already exists: '),
'[Errno 17] File exists: '
),
+ (re.compile('distribute'), 'setuptools'),
])
),
doctest.DocFileSuite(
@@ -2913,9 +3709,18 @@
(re.compile('(zc.buildout|setuptools)-\d+[.]\d+\S*'
'-py\d.\d.egg'),
'\\1.egg'),
+ (re.compile('distribute-\d+[.]\d+\S*'
+ '-py\d.\d.egg'),
+ 'setuptools.egg'),
(re.compile('(zc.buildout|setuptools)( version)? \d+[.]\d+\S*'),
'\\1 V.V'),
- (re.compile('[-d] setuptools'), '- setuptools'),
+ (re.compile('distribute( version)? \d+[.]\d+\S*'),
+ 'setuptools V.V'),
+ (re.compile('[-d] (setuptools|distribute)'), '- setuptools'),
+ (re.compile('distribute'), 'setuptools'),
+ (re.compile("\nUnused options for buildout: "
+ "'(distribute|setuptools)\-version'\."),
+ '')
])
),
@@ -2931,9 +3736,17 @@
zc.buildout.testing.normalize_egg_py,
normalize_bang,
(re.compile('extdemo[.]pyd'), 'extdemo.so'),
- (re.compile('[-d] setuptools-\S+[.]egg'), 'setuptools.egg'),
+ (re.compile('[-d] (setuptools|distribute)-\S+[.]egg'),
+ 'setuptools.egg'),
(re.compile(r'\\[\\]?'), '/'),
(re.compile(r'\#!\S+\bpython\S*'), '#!/usr/bin/python'),
+ # Normalize generate_script's Windows interpreter to UNIX:
+ (re.compile(r'\nimport subprocess\n'), '\n'),
+ (re.compile('subprocess\\.call\\(argv, env=environ\\)'),
+ 'os.execve(sys.executable, argv, environ)'),
+ (re.compile('distribute'), 'setuptools'),
+ # Distribute unzips eggs by default.
+ (re.compile('\- demoneeded'), 'd demoneeded'),
]+(sys.version_info < (2, 5) and [
(re.compile('.*No module named runpy.*', re.S), ''),
(re.compile('.*usage: pdb.py scriptfile .*', re.S), ''),
@@ -2965,7 +3778,7 @@
zc.buildout.testing.normalize_egg_py,
(re.compile("buildout: Running \S*setup.py"),
'buildout: Running setup.py'),
- (re.compile('setuptools-\S+-'),
+ (re.compile('(setuptools|distribute)-\S+-'),
'setuptools.egg'),
(re.compile('zc.buildout-\S+-'),
'zc.buildout.egg'),
@@ -2973,7 +3786,7 @@
'File "one.py"'),
(re.compile(r'We have a develop egg: (\S+) (\S+)'),
r'We have a develop egg: \1 V'),
- (re.compile('Picked: setuptools = \S+'),
+ (re.compile('Picked: (setuptools|distribute) = \S+'),
'Picked: setuptools = V'),
(re.compile(r'\\[\\]?'), '/'),
(re.compile(
@@ -2981,6 +3794,12 @@
'-q develop -mxN -d /sample-buildout/develop-eggs'
),
(re.compile(r'^[*]...'), '...'),
+ # for bug_92891_bootstrap_crashes_with_egg_recipe_in_buildout_section
+ (re.compile(r"Unused options for buildout: 'eggs' 'scripts'\."),
+ "Unused options for buildout: 'scripts' 'eggs'."),
+ (re.compile('distribute'), 'setuptools'),
+ # Distribute unzips eggs by default.
+ (re.compile('\- demoneeded'), 'd demoneeded'),
]),
),
zc.buildout.testselectingpython.test_suite(),
@@ -3040,6 +3859,8 @@
zc.buildout.testing.normalize_script,
normalize_bang,
(re.compile('Downloading.*setuptools.*egg\n'), ''),
+ (re.compile('options:'), 'Options:'),
+ (re.compile('usage:'), 'Usage:'),
]),
))
Modified: zc.buildout/trunk/src/zc/buildout/testselectingpython.py
===================================================================
--- zc.buildout/trunk/src/zc/buildout/testselectingpython.py 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/src/zc/buildout/testselectingpython.py 2010-04-29 18:11:58 UTC (rev 111588)
@@ -11,7 +11,7 @@
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
-import os, re, sys, unittest
+import os, re, subprocess, sys, textwrap, unittest
from zope.testing import doctest, renormalizing
import zc.buildout.tests
import zc.buildout.testing
@@ -42,6 +42,33 @@
def multi_python(test):
other_executable = zc.buildout.testing.find_python(other_version)
+ command = textwrap.dedent('''\
+ try:
+ import setuptools
+ except ImportError:
+ import sys
+ sys.exit(1)
+ ''')
+ if subprocess.call([other_executable, '-c', command],
+ env=os.environ):
+ # the other executable does not have setuptools. Get setuptools.
+ # We will do this using the same tools we are testing, for better or
+ # worse. Alternatively, we could try using bootstrap.
+ executable_dir = test.globs['tmpdir']('executable_dir')
+ executable_parts = os.path.join(executable_dir, 'parts')
+ test.globs['mkdir'](executable_parts)
+ ws = zc.buildout.easy_install.install(
+ ['setuptools'], executable_dir,
+ index='http://www.python.org/pypi/',
+ always_unzip=True, executable=other_executable)
+ zc.buildout.easy_install.sitepackage_safe_scripts(
+ executable_dir, ws, other_executable, executable_parts,
+ reqs=['setuptools'], interpreter='py')
+ original_executable = other_executable
+ other_executable = os.path.join(executable_dir, 'py')
+ assert not subprocess.call(
+ [other_executable, '-c', command], env=os.environ), (
+ 'test set up failed')
sample_eggs = test.globs['tmpdir']('sample_eggs')
os.mkdir(os.path.join(sample_eggs, 'index'))
test.globs['sample_eggs'] = sample_eggs
Modified: zc.buildout/trunk/src/zc/buildout/update.txt
===================================================================
--- zc.buildout/trunk/src/zc/buildout/update.txt 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/src/zc/buildout/update.txt 2010-04-29 18:11:58 UTC (rev 111588)
@@ -78,25 +78,30 @@
zc.buildout 99.99
setuptools 99.99
-Our buildout script has been updated to use the new eggs:
+Our buildout script's site.py has been updated to use the new eggs:
- >>> cat(sample_buildout, 'bin', 'buildout')
- #!/usr/local/bin/python2.4
+ >>> cat(sample_buildout, 'parts', 'buildout', 'site.py')
+ ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
+ "...
+ def addsitepackages(known_paths):
+ """Add site packages, as determined by zc.buildout.
<BLANKLINE>
- import sys
- sys.path[0:0] = [
- '/sample-buildout/eggs/zc.buildout-99.99-py2.4.egg',
- '/sample-buildout/eggs/setuptools-99.99-py2.4.egg',
- ]
- <BLANKLINE>
- import zc.buildout.buildout
- <BLANKLINE>
- if __name__ == '__main__':
- zc.buildout.buildout.main()
+ See original_addsitepackages, below, for the original version."""
+ buildout_paths = [
+ '/sample-buildout/eggs/zc.buildout-99.99-pyN.N.egg',
+ '/sample-buildout/eggs/setuptools-99.99-pyN.N.egg'
+ ]
+ for path in buildout_paths:
+ sitedir, sitedircase = makepath(path)
+ if not sitedircase in known_paths and os.path.exists(sitedir):
+ sys.path.append(sitedir)
+ known_paths.add(sitedircase)
+ return known_paths
+ ...
Now, let's recreate the sample buildout. If we specify constraints on
-the versions of zc.buildout and setuptools to use, running the
-buildout will install earlier versions of these packages:
+the versions of zc.buildout and setuptools (or distribute) to use,
+running the buildout will install earlier versions of these packages:
>>> write(sample_buildout, 'buildout.cfg',
... """
@@ -107,6 +112,7 @@
... develop = showversions
... zc.buildout-version = < 99
... setuptools-version = < 99
+ ... distribute-version = < 99
...
... [show-versions]
... recipe = showversions
@@ -119,7 +125,6 @@
zc.buildout version 1.0.0,
setuptools version 0.6;
restarting.
- Generated script '/sample-buildout/bin/buildout'.
Develop: '/sample-buildout/showversions'
Updating show-versions.
zc.buildout 1.0.0
Modified: zc.buildout/trunk/zc.recipe.egg_/setup.py
===================================================================
--- zc.buildout/trunk/zc.recipe.egg_/setup.py 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/zc.recipe.egg_/setup.py 2010-04-29 18:11:58 UTC (rev 111588)
@@ -16,7 +16,7 @@
$Id$
"""
-version = '0'
+version = '1.2.3dev'
import os
from setuptools import setup, find_packages
@@ -66,7 +66,7 @@
package_dir = {'':'src'},
namespace_packages = ['zc', 'zc.recipe'],
install_requires = [
- 'zc.buildout >=1.2.0',
+ 'zc.buildout >=1.5.0dev',
'setuptools'],
tests_require = ['zope.testing'],
test_suite = name+'.tests.test_suite',
Modified: zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/README.txt
===================================================================
--- zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/README.txt 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/README.txt 2010-04-29 18:11:58 UTC (rev 111588)
@@ -154,6 +154,8 @@
interpreter
The name of a script to generate that allows access to a Python
interpreter that has the path set based on the eggs installed.
+ (See the ``z3c.recipe.scripts`` recipe for a more full-featured
+ interpreter.)
extra-paths
Extra paths to include in a generated script.
@@ -577,7 +579,7 @@
- demo
- other
- >>> cat(sample_buildout, 'bin', 'other')
+ >>> cat(sample_buildout, 'bin', 'other') # doctest: +NORMALIZE_WHITESPACE
#!/usr/local/bin/python2.4
<BLANKLINE>
import sys
@@ -640,3 +642,4 @@
Uninstalling bigdemo.
Installing demo.
Generated script '/sample-buildout/bin/foo'.
+
Modified: zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/api.txt
===================================================================
--- zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/api.txt 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/api.txt 2010-04-29 18:11:58 UTC (rev 111588)
@@ -117,6 +117,7 @@
extras = other
find-links = http://localhost:27071/
index = http://localhost:27071/index
+ python = buildout
recipe = sample
If we use the extra-paths option:
Modified: zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/custom.txt
===================================================================
--- zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/custom.txt 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/custom.txt 2010-04-29 18:11:58 UTC (rev 111588)
@@ -150,6 +150,7 @@
>>> ls(sample_buildout, 'develop-eggs')
d extdemo-1.4-py2.4-unix-i686.egg
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
Note that no scripts or dependencies are installed. To install
@@ -231,6 +232,7 @@
>>> ls(sample_buildout, 'develop-eggs')
- demo.egg-link
d extdemo-1.4-py2.4-unix-i686.egg
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
But if we run the buildout in the default on-line and newest modes, we
@@ -248,6 +250,7 @@
- demo.egg-link
d extdemo-1.4-py2.4-linux-i686.egg
d extdemo-1.5-py2.4-linux-i686.egg
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
Controlling the version used
@@ -287,6 +290,7 @@
>>> ls(sample_buildout, 'develop-eggs')
- demo.egg-link
d extdemo-1.4-py2.4-linux-i686.egg
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
@@ -553,6 +557,7 @@
>>> ls('develop-eggs')
- demo.egg-link
- extdemo.egg-link
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
and the extdemo now has a built extension:
Modified: zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/egg.py
===================================================================
--- zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/egg.py 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/egg.py 2010-04-29 18:11:58 UTC (rev 111588)
@@ -19,14 +19,17 @@
import logging, os, re, zipfile
import zc.buildout.easy_install
+
class Eggs(object):
+ include_site_packages = allowed_eggs = None
+
def __init__(self, buildout, name, options):
self.buildout = buildout
- self.name = name
+ self.name = self.default_eggs = name
self.options = options
- links = options.get('find-links',
- buildout['buildout'].get('find-links'))
+ b_options = buildout['buildout']
+ links = options.get('find-links', b_options['find-links'])
if links:
links = links.split()
options['find-links'] = '\n'.join(links)
@@ -34,25 +37,25 @@
links = ()
self.links = links
- index = options.get('index', buildout['buildout'].get('index'))
+ index = options.get('index', b_options.get('index'))
if index is not None:
options['index'] = index
self.index = index
- allow_hosts = buildout['buildout'].get('allow-hosts', '*')
+ allow_hosts = b_options['allow-hosts']
allow_hosts = tuple([host.strip() for host in allow_hosts.split('\n')
if host.strip()!=''])
self.allow_hosts = allow_hosts
- options['eggs-directory'] = buildout['buildout']['eggs-directory']
+ options['eggs-directory'] = b_options['eggs-directory']
options['_e'] = options['eggs-directory'] # backward compat.
- options['develop-eggs-directory'
- ] = buildout['buildout']['develop-eggs-directory']
+ options['develop-eggs-directory'] = b_options['develop-eggs-directory']
options['_d'] = options['develop-eggs-directory'] # backward compat.
- assert options.get('unzip') in ('true', 'false', None)
+ # verify that this is None, 'true' or 'false'
+ get_bool(options, 'unzip')
- python = options.get('python', buildout['buildout']['python'])
+ python = options.setdefault('python', b_options['python'])
options['executable'] = buildout[python]['executable']
def working_set(self, extra=()):
@@ -61,31 +64,36 @@
This is intended for reuse by similar recipes.
"""
options = self.options
+ b_options = self.buildout['buildout']
distributions = [
r.strip()
- for r in options.get('eggs', self.name).split('\n')
+ for r in options.get('eggs', self.default_eggs).split('\n')
if r.strip()]
orig_distributions = distributions[:]
distributions.extend(extra)
- if self.buildout['buildout'].get('offline') == 'true':
+ if b_options.get('offline') == 'true':
ws = zc.buildout.easy_install.working_set(
distributions, options['executable'],
- [options['develop-eggs-directory'], options['eggs-directory']]
+ [options['develop-eggs-directory'],
+ options['eggs-directory']],
+ include_site_packages=self.include_site_packages,
+ allowed_eggs_from_site_packages=self.allowed_eggs,
)
else:
kw = {}
- if options.get('unzip'):
+ if 'unzip' in options:
kw['always_unzip'] = get_bool(options, 'unzip')
-
ws = zc.buildout.easy_install.install(
distributions, options['eggs-directory'],
links=self.links,
index=self.index,
executable=options['executable'],
path=[options['develop-eggs-directory']],
- newest=self.buildout['buildout'].get('newest') == 'true',
+ newest=b_options.get('newest') == 'true',
+ include_site_packages=self.include_site_packages,
+ allowed_eggs_from_site_packages=self.allowed_eggs,
allow_hosts=self.allow_hosts,
**kw)
@@ -97,16 +105,19 @@
update = install
-class Scripts(Eggs):
+class ScriptBase(Eggs):
+
def __init__(self, buildout, name, options):
- super(Scripts, self).__init__(buildout, name, options)
+ super(ScriptBase, self).__init__(buildout, name, options)
- options['bin-directory'] = buildout['buildout']['bin-directory']
+ b_options = buildout['buildout']
+
+ options['bin-directory'] = b_options['bin-directory']
options['_b'] = options['bin-directory'] # backward compat.
self.extra_paths = [
- os.path.join(buildout['buildout']['directory'], p.strip())
+ os.path.join(b_options['directory'], p.strip())
for p in options.get('extra-paths', '').split('\n')
if p.strip()
]
@@ -115,11 +126,9 @@
relative_paths = options.get(
- 'relative-paths',
- buildout['buildout'].get('relative-paths', 'false')
- )
+ 'relative-paths', b_options.get('relative-paths', 'false'))
if relative_paths == 'true':
- options['buildout-directory'] = buildout['buildout']['directory']
+ options['buildout-directory'] = b_options['directory']
self._relative_paths = options['buildout-directory']
else:
self._relative_paths = ''
@@ -128,12 +137,13 @@
parse_entry_point = re.compile(
'([^=]+)=(\w+(?:[.]\w+)*):(\w+(?:[.]\w+)*)$'
).match
+
def install(self):
reqs, ws = self.working_set()
options = self.options
scripts = options.get('scripts')
- if scripts or scripts is None:
+ if scripts or scripts is None or options.get('interpreter'):
if scripts is not None:
scripts = scripts.split()
scripts = dict([
@@ -157,22 +167,32 @@
name = dist.project_name
if name != 'setuptools' and name not in reqs:
reqs.append(name)
-
- return zc.buildout.easy_install.scripts(
- reqs, ws, options['executable'],
- options['bin-directory'],
- scripts=scripts,
- extra_paths=self.extra_paths,
- interpreter=options.get('interpreter'),
- initialization=options.get('initialization', ''),
- arguments=options.get('arguments', ''),
- relative_paths=self._relative_paths,
- )
-
+ return self._install(reqs, ws, scripts)
return ()
update = install
+ def _install(self, reqs, ws, scripts):
+ # Subclasses implement this.
+ raise NotImplementedError()
+
+
+class Scripts(ScriptBase):
+
+ def _install(self, reqs, ws, scripts):
+ options = self.options
+ return zc.buildout.easy_install.scripts(
+ reqs, ws, options['executable'],
+ options['bin-directory'],
+ scripts=scripts,
+ extra_paths=self.extra_paths,
+ interpreter=options.get('interpreter'),
+ initialization=options.get('initialization', ''),
+ arguments=options.get('arguments', ''),
+ relative_paths=self._relative_paths
+ )
+
+
def get_bool(options, name, default=False):
value = options.get(name)
if not value:
Modified: zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/selecting-python.txt
===================================================================
--- zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/selecting-python.txt 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/selecting-python.txt 2010-04-29 18:11:58 UTC (rev 111588)
@@ -35,7 +35,7 @@
... index = http://www.python.org/pypi/
...
... [python2.4]
- ... executable = %(python23)s
+ ... executable = %(python24)s
...
... [demo]
... recipe = zc.recipe.egg
@@ -43,7 +43,7 @@
... find-links = %(server)s
... python = python2.4
... interpreter = py-demo
- ... """ % dict(server=link_server, python23=other_executable))
+ ... """ % dict(server=link_server, python24=other_executable))
Now, if we run the buildout:
Modified: zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/tests.py
===================================================================
--- zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/tests.py 2010-04-29 17:29:39 UTC (rev 111587)
+++ zc.buildout/trunk/zc.recipe.egg_/src/zc/recipe/egg/tests.py 2010-04-29 18:11:58 UTC (rev 111588)
@@ -50,9 +50,12 @@
zc.buildout.tests.normalize_bang,
(re.compile('zc.buildout(-\S+)?[.]egg(-link)?'),
'zc.buildout.egg'),
- (re.compile('[-d] setuptools-[^-]+-'), 'setuptools-X-'),
+ (re.compile('[-d] (setuptools|distribute)-[^-]+-'),
+ 'setuptools-X-'),
(re.compile(r'eggs\\\\demo'), 'eggs/demo'),
(re.compile(r'[a-zA-Z]:\\\\foo\\\\bar'), '/foo/bar'),
+ # Distribute unzips eggs by default.
+ (re.compile('\- demoneeded'), 'd demoneeded'),
])
),
doctest.DocFileSuite(
@@ -64,7 +67,7 @@
(re.compile('__buildout_signature__ = '
'sample-\S+\s+'
'zc.recipe.egg-\S+\s+'
- 'setuptools-\S+\s+'
+ '(setuptools|distribute)-\S+\s+'
'zc.buildout-\S+\s*'
),
'__buildout_signature__ = sample- zc.recipe.egg-\n'),
@@ -104,14 +107,17 @@
zc.buildout.testing.normalize_path,
zc.buildout.testing.normalize_endings,
zc.buildout.testing.normalize_script,
- (re.compile('Got setuptools \S+'), 'Got setuptools V'),
- (re.compile('([d-] )?setuptools-\S+-py'),
+ (re.compile('Got (setuptools|distribute) \S+'),
+ 'Got setuptools V'),
+ (re.compile('([d-] )?(setuptools|distribute)-\S+-py'),
'setuptools-V-py'),
(re.compile('-py2[.][0-35-9][.]'), 'py2.5.'),
(re.compile('zc.buildout-\S+[.]egg'),
'zc.buildout.egg'),
(re.compile('zc.buildout[.]egg-link'),
'zc.buildout.egg'),
+ # Distribute unzips eggs by default.
+ (re.compile('\- demoneeded'), 'd demoneeded'),
]),
),
)
More information about the checkins
mailing list