[Checkins] SVN: zc.buildout/branches/gary-support-system-python-2/ merge in svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/gary-support-system-python to a copy of trunk. merge conflicts and merge-related test failures are addressed in this branch.
Gary Poster
gary.poster at canonical.com
Tue Aug 11 15:22:31 EDT 2009
Log message for revision 102683:
merge in svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/gary-support-system-python to a copy of trunk. merge conflicts and merge-related test failures are addressed in this branch.
Changed:
U zc.buildout/branches/gary-support-system-python-2/CHANGES.txt
U zc.buildout/branches/gary-support-system-python-2/bootstrap/bootstrap.py
U zc.buildout/branches/gary-support-system-python-2/buildout.cfg
U zc.buildout/branches/gary-support-system-python-2/dev.py
U zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/bootstrap.txt
U zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/buildout.py
U zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/buildout.txt
U zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/easy_install.py
U zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/easy_install.txt
U zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/setup.txt
U zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/testing.py
U zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/tests.py
U zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/testselectingpython.py
U zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/unzip.txt
U zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/update.txt
U zc.buildout/branches/gary-support-system-python-2/zc.recipe.egg_/setup.py
U zc.buildout/branches/gary-support-system-python-2/zc.recipe.egg_/src/zc/recipe/egg/README.txt
U zc.buildout/branches/gary-support-system-python-2/zc.recipe.egg_/src/zc/recipe/egg/egg.py
-=-
Modified: zc.buildout/branches/gary-support-system-python-2/CHANGES.txt
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/CHANGES.txt 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/CHANGES.txt 2009-08-11 19:22:31 UTC (rev 102683)
@@ -13,6 +13,81 @@
- Used the download API to allow caching of base configurations (specified by
the buildout section's 'extends' option).
+- Support and bugfixes for using a system Python with buildout.
+
+ In all of these descriptions, "site-packages" is an imprecise term for a
+ precise definition: the packages that are added by the Python's site.py.
+ Practically, this is the difference of the set of paths normally used by
+ Python minus those used when Python is started with the -S flag.
+
+ * A new boolean option, 'include-site-packages', includes or excludes site
+ packages from finding requirements, and from generated scripts.
+ zc.buildout's own buildout.cfg dogfoods this option. This defaults
+ to 'true', which is very similar to buildout's previous behavior.
+
+ * A new boolean option, 'include-site-packages-for-buildout', does the
+ same thing but only for the bin/buildout script. This can be important
+ for getting recipes and their dependencies without conflicts. This
+ defaults to 'false', which is different from buildout's previous behavior.
+
+ * Another new option, 'allowed-eggs-from-site-packages', lets you specify
+ a whitelist of project names of eggs that are allowed to come from
+ your Python's site-packages. This lets you more tightly control your use
+ of site-packages.
+
+ This option interacts with the ``include-site-packages`` option in the
+ following ways.
+
+ If ``include-site-packages`` is true, then
+ ``allowed-eggs-from-site-packages`` filters what eggs from site-packages
+ may be chosen. Therefore, if ``allowed-eggs-from-site-packages`` is an
+ empty list, then no eggs from site-packages are chosen, but site-packages
+ will still be included at the end of path lists.
+
+ If ``include-site-packages`` is false, the value of
+ ``allowed-eggs-from-site-packages`` is irrelevant.
+
+ The same pattern holds true for the ``include-site-packages-for-buildout``
+ option, except only the bin/buildout script is affected by that
+ interaction.
+
+ * Script generation pushes dependency paths that are in site-packages to
+ the end of the dependency paths in sys.path (but, as before, these are
+ still before extra paths, the standard library, and the rest of the
+ site-package paths).
+
+ * Running "bin/buildout -v" will include output alerting you to selected eggs
+ that came from site-packages, using the pattern
+ "Egg from site-packages: ...".
+
+ * Fix an error when at least two dependencies were in a shared location like
+ site-packages, and the first one met the "versions" setting. The first
+ dependency would be added, but subsequent dependencies from the same
+ location (e.g., site-packages) would use the version of the package found
+ in the shared location, ignoring the version setting.
+
+- The generated interpreter scripts now have a few more similarities to a
+ real Python interpreter.
+
+ * __file__ is set correctly for executed files.
+
+ * -c incorporates all subsequent arguments as part of the command.
+
+ * -V returns the version.
+
+ * -S causes the script to not modify the standard path (for tests)
+
+- Improve bootstrap.
+
+ * New options let you specify where to find ez_setup.py and where to find
+ a download cache. These options can keep bootstrap from going over the
+ network.
+
+ * Another new option lets you specify where to put generated eggs.
+
+ * The buildout script generated by bootstrap honors more of the settings
+ in the designated configuration file (e.g., buildout.cfg).
+
1.3.0 (2009-06-22)
==================
Modified: zc.buildout/branches/gary-support-system-python-2/bootstrap/bootstrap.py
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/bootstrap/bootstrap.py 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/bootstrap/bootstrap.py 2009-08-11 19:22:31 UTC (rev 102683)
@@ -20,21 +20,108 @@
$Id$
"""
-import os, shutil, sys, tempfile, urllib2
+import os, re, shutil, sys, tempfile, textwrap, urllib, urllib2
-tmpeggs = tempfile.mkdtemp()
+# We have to manually parse our options rather than using one of the stdlib
+# tools because we want to pass the ones we don't recognize along to
+# zc.buildout.buildout.main.
-is_jython = sys.platform.startswith('java')
+configuration = {
+ '--ez_setup-source': 'http://peak.telecommunity.com/dist/ez_setup.py',
+ '--version': '',
+ '--download-base': None,
+ '--eggs': None}
+helpstring = __doc__ + textwrap.dedent('''
+ This script recognizes the following options itself. The first option it
+ encounters that is not one of these will cause the script to stop parsing
+ options and pass the rest on to buildout. Therefore, if you want to use
+ any of the following options *and* buildout command-line options like
+ -c, first use the following options, and then use the buildout options.
+
+ Options:
+ --version=ZC_BUILDOUT_VERSION
+ Specify a version number of the zc.buildout to use
+ --ez_setup-source=URL_OR_FILE
+ Specify a URL or file location for the ez_setup file.
+ Defaults to
+ %(--ez_setup-source)s
+ --download-base=URL_OR_DIRECTORY
+ Specify a URL or directory for downloading setuptools and
+ zc.buildout. Defaults to PyPI.
+ --eggs=DIRECTORY
+ Specify a directory for storing eggs. Defaults to a temporary
+ directory that is deleted when the bootstrap script completes.
+
+ By using --ez_setup-source and --download-base to point to local resources,
+ you can keep this script from going over the network.
+ ''' % configuration)
+match_equals = re.compile(r'(%s)=(.*)' % ('|'.join(configuration),)).match
+args = sys.argv[1:]
+if args == ['--help']:
+ print helpstring
+ sys.exit(0)
+
+# If we end up using a temporary directory for storing our eggs, this will
+# hold the path of that directory. On the other hand, if an explicit directory
+# is specified in the argv, this will remain None.
+tmpeggs = None
+
+while args:
+ val = args[0]
+ if val in configuration:
+ del args[0]
+ if not args or args[0].startswith('-'):
+ print "ERROR: %s requires an argument."
+ print helpstring
+ sys.exit(1)
+ configuration[val] = args[0]
+ else:
+ match = match_equals(val)
+ if match and match.group(1) in configuration:
+ configuration[match.group(1)] = match.group(2)
+ else:
+ break
+ del args[0]
+
+for name in ('--ez_setup-source', '--download-base'):
+ val = configuration[name]
+ if val is not None and '://' not in val: # We're being lazy.
+ configuration[name] = 'file://%s' % (
+ urllib.pathname2url(os.path.abspath(os.path.expanduser(val))),)
+
+if (configuration['--download-base'] and
+ not configuration['--download-base'].endswith('/')):
+ # Download base needs a trailing slash to make the world happy.
+ configuration['--download-base'] += '/'
+
+if not configuration['--eggs']:
+ configuration['--eggs'] = tmpeggs = tempfile.mkdtemp()
+else:
+ configuration['--eggs'] = os.path.abspath(
+ os.path.expanduser(configuration['--eggs']))
+
+# The requirement is what we will pass to setuptools to specify zc.buildout.
+requirement = 'zc.buildout'
+if configuration['--version']:
+ requirement += '==' + configuration['--version']
+
try:
+ import setuptools # A flag. Sometimes pkg_resources is installed alone.
import pkg_resources
except ImportError:
ez = {}
- exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py'
- ).read() in ez
- ez['use_setuptools'](to_dir=tmpeggs, download_delay=0)
-
+ exec urllib2.urlopen(configuration['--ez_setup-source']).read() in ez
+ setuptools_args = dict(to_dir=configuration['--eggs'], download_delay=0)
+ if configuration['--download-base']:
+ setuptools_args['download_base'] = configuration['--download-base']
+ ez['use_setuptools'](**setuptools_args)
import pkg_resources
+ # This does not (always?) update the default working set. We will
+ # do it.
+ for path in sys.path:
+ if path not in pkg_resources.working_set.entries:
+ pkg_resources.working_set.add_entry(path)
if sys.platform == 'win32':
def quote(c):
@@ -45,40 +132,40 @@
else:
def quote (c):
return c
+cmd = [quote(sys.executable),
+ '-c',
+ quote('from setuptools.command.easy_install import main; main()'),
+ '-mqNxd',
+ quote(configuration['--eggs'])]
-cmd = 'from setuptools.command.easy_install import main; main()'
-ws = pkg_resources.working_set
+if configuration['--download-base']:
+ cmd.extend(['-f', quote(configuration['--download-base'])])
-if len(sys.argv) > 2 and sys.argv[1] == '--version':
- VERSION = '==%s' % sys.argv[2]
- args = sys.argv[3:] + ['bootstrap']
-else:
- VERSION = ''
- args = sys.argv[1:] + ['bootstrap']
+cmd.append(requirement)
+ws = pkg_resources.working_set
+env = dict(
+ os.environ,
+ PYTHONPATH=ws.find(pkg_resources.Requirement.parse('setuptools')).location)
+
+is_jython = sys.platform.startswith('java')
if is_jython:
import subprocess
+ exitcode = subprocess.Popen(cmd, env=env).wait()
+else: # Windows needs this, apparently; otherwise we would prefer subprocess
+ exitcode = os.spawnle(*([os.P_WAIT, sys.executable] + cmd + [env]))
+if exitcode != 0:
+ sys.stdout.flush()
+ sys.stderr.flush()
+ print ("An error occured when trying to install zc.buildout. "
+ "Look above this message for any errors that "
+ "were output by easy_install.")
+ sys.exit(exitcode)
- assert subprocess.Popen([sys.executable] + ['-c', quote(cmd), '-mqNxd',
- quote(tmpeggs), 'zc.buildout' + VERSION],
- env=dict(os.environ,
- PYTHONPATH=
- ws.find(pkg_resources.Requirement.parse('setuptools')).location
- ),
- ).wait() == 0
-
-else:
- assert os.spawnle(
- os.P_WAIT, sys.executable, quote (sys.executable),
- '-c', quote (cmd), '-mqNxd', quote (tmpeggs), 'zc.buildout' + VERSION,
- dict(os.environ,
- PYTHONPATH=
- ws.find(pkg_resources.Requirement.parse('setuptools')).location
- ),
- ) == 0
-
-ws.add_entry(tmpeggs)
-ws.require('zc.buildout' + VERSION)
+ws.add_entry(configuration['--eggs'])
+ws.require(requirement)
import zc.buildout.buildout
+args.append('bootstrap')
zc.buildout.buildout.main(args)
-shutil.rmtree(tmpeggs)
+if tmpeggs is not None:
+ shutil.rmtree(tmpeggs)
Modified: zc.buildout/branches/gary-support-system-python-2/buildout.cfg
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/buildout.cfg 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/buildout.cfg 2009-08-11 19:22:31 UTC (rev 102683)
@@ -1,6 +1,7 @@
[buildout]
develop = zc.recipe.egg_ .
parts = test oltest py
+include-site-packages = false
[py]
recipe = zc.recipe.egg
Modified: zc.buildout/branches/gary-support-system-python-2/dev.py
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/dev.py 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/dev.py 2009-08-11 19:22:31 UTC (rev 102683)
@@ -47,9 +47,8 @@
pkg_resources.working_set.add_entry('src')
-import zc.buildout.easy_install
-zc.buildout.easy_install.scripts(
- ['zc.buildout'], pkg_resources.working_set , sys.executable, 'bin')
+import zc.buildout.buildout
+zc.buildout.buildout.main(['bootstrap'])
bin_buildout = os.path.join('bin', 'buildout')
Modified: zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/bootstrap.txt
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/bootstrap.txt 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/bootstrap.txt 2009-08-11 19:22:31 UTC (rev 102683)
@@ -57,7 +57,7 @@
... 'bootstrap.py --version UNKNOWN'); print 'X' # doctest: +ELLIPSIS
...
X
- No local packages or download links found for zc.buildout==UNKNOWN
+ No local packages or download links found for zc.buildout==UNKNOWN...
error: Could not find suitable distribution for Requirement.parse('zc.buildout==UNKNOWN')
...
@@ -120,4 +120,71 @@
zc.buildout.buildout.main()
<BLANKLINE>
+You can specify a location of ez_setup.py, so you can rely on a local or remote
+location. We'll write our own ez_setup.py that we will also use to test some
+other bootstrap options.
+ >>> write('ez_setup.py', '''\
+ ... def use_setuptools(**kwargs):
+ ... import sys, pprint
+ ... pprint.pprint(kwargs, width=40)
+ ... sys.exit()
+ ... ''')
+ >>> print system(
+ ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
+ ... 'bootstrap.py --ez_setup-source=./ez_setup.py')
+ ... # doctest: +ELLIPSIS
+ {'download_delay': 0,
+ 'to_dir': '...'}
+ <BLANKLINE>
+
+You can also pass a download-cache, and a place in which eggs should be stored
+(they are normally stored in a temporary directory).
+
+ >>> print system(
+ ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
+ ... 'bootstrap.py --ez_setup-source=./ez_setup.py '+
+ ... '--download-base=./download-cache --eggs=eggs')
+ ... # doctest: +ELLIPSIS
+ {'download_base': '/sample/download-cache/',
+ 'download_delay': 0,
+ 'to_dir': '/sample/eggs'}
+ <BLANKLINE>
+
+Here's the entire help text.
+
+ >>> print system(
+ ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
+ ... 'bootstrap.py --help'),
+ ... # doctest: +ELLIPSIS
+ Bootstrap a buildout-based project
+ <BLANKLINE>
+ Simply run this script in a directory containing a buildout.cfg.
+ The script accepts buildout command-line options, so you can
+ use the -c option to specify an alternate configuration file.
+ <BLANKLINE>
+ ...
+ <BLANKLINE>
+ This script recognizes the following options itself. The first option it
+ encounters that is not one of these will cause the script to stop parsing
+ options and pass the rest on to buildout. Therefore, if you want to use
+ any of the following options *and* buildout command-line options like
+ -c, first use the following options, and then use the buildout options.
+ <BLANKLINE>
+ Options:
+ --version=ZC_BUILDOUT_VERSION
+ Specify a version number of the zc.buildout to use
+ --ez_setup-source=URL_OR_FILE
+ Specify a URL or file location for the ez_setup file.
+ Defaults to
+ http://peak.telecommunity.com/dist/ez_setup.py
+ --download-base=URL_OR_DIRECTORY
+ Specify a URL or directory for downloading setuptools and
+ zc.buildout. Defaults to PyPI.
+ --eggs=DIRECTORY
+ Specify a directory for storing eggs. Defaults to a temporary
+ directory that is deleted when the bootstrap script completes.
+ <BLANKLINE>
+ By using --ez_setup-source and --download-base to point to local resources,
+ you can keep this script from going over the network.
+ <BLANKLINE>
Modified: zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/buildout.py
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/buildout.py 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/buildout.py 2009-08-11 19:22:31 UTC (rev 102683)
@@ -61,7 +61,7 @@
"""
class MissingSection(zc.buildout.UserError, KeyError):
- """A required section is missinh
+ """A required section is missing
"""
def __str__(self):
@@ -117,15 +117,27 @@
return data
_buildout_default_options = _annotate_section({
+ 'allow-hosts': '*',
+ 'allow-picked-versions': 'true',
+ 'allowed-eggs-from-site-packages': '*',
+ 'bin-directory': 'bin',
+ 'develop-eggs-directory': 'develop-eggs',
'eggs-directory': 'eggs',
- 'develop-eggs-directory': 'develop-eggs',
- 'bin-directory': 'bin',
+ 'executable': sys.executable,
+ 'find-links': '',
+ 'include-site-packages-for-buildout': 'false',
+ 'include-site-packages': 'true',
+ 'install-from-cache': 'false',
+ 'installed': '.installed.cfg',
+ 'log-format': '',
+ 'log-level': 'INFO',
+ 'newest': 'true',
+ 'offline': 'false',
'parts-directory': 'parts',
- 'installed': '.installed.cfg',
+ 'prefer-final': 'false',
'python': 'buildout',
- 'executable': sys.executable,
- 'log-level': 'INFO',
- 'log-format': '',
+ 'unzip': 'false',
+ 'use-dependency-links': 'true',
}, 'DEFAULT_VALUE')
@@ -150,7 +162,7 @@
print 'Creating %r.' % config_file
open(config_file, 'w').write('[buildout]\nparts = \n')
elif command == 'setup':
- # Sigh. this model of a buildout nstance
+ # Sigh. this model of a buildout instance
# with methods is breaking down :(
config_file = None
data['buildout']['directory'] = ('.', 'COMPUTED_VALUE')
@@ -195,7 +207,7 @@
# provide some defaults before options are parsed
# because while parsing options those attributes might be
# used already (Gottfried Ganssauge)
- buildout_section = data.get('buildout')
+ buildout_section = data['buildout']
# Try to make sure we have absolute paths for standard
# directories. We do this before doing substitutions, in case
@@ -208,22 +220,29 @@
d = self._buildout_path(buildout_section[name+'-directory'])
buildout_section[name+'-directory'] = d
- links = buildout_section and buildout_section.get('find-links', '')
+ # Arguably, these attributes shouldn't be used by recipes. Using
+ # attributes on the buildout causes bugs when a buildout is extended,
+ # potentially overriding these values; and yet the option is read from
+ # this attribute during the initialization of the recipes (see
+ # ``options = self['buildout']`` below) before the full processing
+ # is completed. These attributes are left behind for legacy support
+ # but recipe authors should beware of using them. A better practice is
+ # for a recipe to read the buildout['buildout'] options.
+ links = buildout_section['find-links']
self._links = links and links.split() or ()
-
- allow_hosts = buildout_section and buildout_section.get(
- 'allow-hosts', '*').split('\n')
+ allow_hosts = buildout_section['allow-hosts'].split('\n')
self._allow_hosts = tuple([host.strip() for host in allow_hosts
if host.strip() != ''])
-
self._logger = logging.getLogger('zc.buildout')
- self.offline = False
- self.newest = True
+ self.offline = buildout_section['offline'] == 'true'
+ self.newest = buildout_section['newest'] == 'true'
##################################################################
## WARNING!!!
## ALL ATTRIBUTES MUST HAVE REASONABLE DEFAULTS AT THIS POINT
- ## OTHERWISE ATTRIBUTEERRORS MIGHT HAPPEN ANY TIME
+ ## OTHERWISE ATTRIBUTEERRORS MIGHT HAPPEN ANY TIME FROM RECIPES.
+ ## RECIPES SHOULD GENERALLY USE buildout['buildout'] OPTIONS, NOT
+ ## BUILDOUT ATTRIBUTES.
##################################################################
# initialize some attrs and buildout directories.
options = self['buildout']
@@ -232,7 +251,7 @@
links = options.get('find-links', '')
self._links = links and links.split() or ()
- allow_hosts = options.get('allow-hosts', '*').split('\n')
+ allow_hosts = options.setdefault('allow-hosts', '*').split('\n')
self._allow_hosts = tuple([host.strip() for host in allow_hosts
if host.strip() != ''])
@@ -250,39 +269,60 @@
self._setup_logging()
- offline = options.get('offline', 'false')
+ offline = options['offline']
if offline not in ('true', 'false'):
self._error('Invalid value for offline option: %s', offline)
- options['offline'] = offline
- self.offline = offline == 'true'
+ self.offline = (offline == 'true')
if self.offline:
newest = options['newest'] = 'false'
else:
- newest = options.get('newest', 'true')
+ newest = options['newest']
if newest not in ('true', 'false'):
self._error('Invalid value for newest option: %s', newest)
- options['newest'] = newest
- self.newest = newest == 'true'
+ self.newest = (newest == 'true')
versions = options.get('versions')
if versions:
zc.buildout.easy_install.default_versions(dict(self[versions]))
- prefer_final = options.get('prefer-final', 'false')
+ prefer_final = options['prefer-final']
if prefer_final not in ('true', 'false'):
self._error('Invalid value for prefer-final option: %s',
prefer_final)
zc.buildout.easy_install.prefer_final(prefer_final=='true')
- use_dependency_links = options.get('use-dependency-links', 'true')
+ include_site_packages = options['include-site-packages']
+ if include_site_packages not in ('true', 'false'):
+ self._error('Invalid value for include-site-packages option: %s',
+ include_site_packages)
+ zc.buildout.easy_install.include_site_packages(
+ include_site_packages=='true')
+
+ allowed_eggs_from_site_packages = tuple(
+ name.strip() for name in
+ options['allowed-eggs-from-site-packages'].split('\n'))
+ zc.buildout.easy_install.allowed_eggs_from_site_packages(
+ allowed_eggs_from_site_packages)
+
+ include_site_packages_for_buildout = options[
+ 'include-site-packages-for-buildout']
+ if include_site_packages_for_buildout not in ('true', 'false'):
+ self._error(
+ 'Invalid value for include-site-packages-for-buildout option: '
+ '%s',
+ include_site_packages_for_buildout)
+ self.include_site_packages_for_buildout = (
+ include_site_packages_for_buildout=='true')
+
+ use_dependency_links = options['use-dependency-links']
if use_dependency_links not in ('true', 'false'):
self._error('Invalid value for use-dependency-links option: %s',
use_dependency_links)
zc.buildout.easy_install.use_dependency_links(
- use_dependency_links == 'true')
+ use_dependency_links=='true')
- allow_picked_versions = options.get('allow-picked-versions', 'true')
+ allow_picked_versions = options['allow-picked-versions']
if allow_picked_versions not in ('true', 'false'):
self._error('Invalid value for allow-picked-versions option: %s',
allow_picked_versions)
@@ -304,23 +344,19 @@
zc.buildout.easy_install.download_cache(download_cache)
- install_from_cache = options.get('install-from-cache')
- if install_from_cache:
- if install_from_cache not in ('true', 'false'):
- self._error('Invalid value for install-from-cache option: %s',
- install_from_cache)
- if install_from_cache == 'true':
- zc.buildout.easy_install.install_from_cache(True)
+ install_from_cache = options['install-from-cache']
+ if install_from_cache not in ('true', 'false'):
+ self._error('Invalid value for install-from-cache option: %s',
+ install_from_cache)
+ zc.buildout.easy_install.install_from_cache(
+ install_from_cache=='true')
+ always_unzip = options['unzip']
+ if always_unzip not in ('true', 'false'):
+ self._error('Invalid value for unzip option: %s',
+ always_unzip)
+ zc.buildout.easy_install.always_unzip(always_unzip=='true')
- always_unzip = options.get('unzip')
- if always_unzip:
- if always_unzip not in ('true', 'false'):
- self._error('Invalid value for unzip option: %s',
- always_unzip)
- if always_unzip == 'true':
- zc.buildout.easy_install.always_unzip(True)
-
# "Use" each of the defaults so they aren't reported as unused options.
for name in _buildout_default_options:
options[name]
@@ -338,15 +374,38 @@
return os.path.join(self._buildout_dir, name)
def bootstrap(self, args):
- __doing__ = 'Bootstraping.'
+ __doing__ = 'Bootstrapping.'
self._setup_directories()
+ options = self['buildout']
+
+ # Get a base working set for our distributions that corresponds to the
+ # stated desires in the configuration.
+ distributions = ['setuptools', 'zc.buildout']
+ if options.get('offline') == 'true':
+ ws = zc.buildout.easy_install.working_set(
+ distributions, options['executable'],
+ [options['develop-eggs-directory'], options['eggs-directory']],
+ include_site_packages=self.include_site_packages_for_buildout
+ )
+ else:
+ ws = zc.buildout.easy_install.install(
+ distributions, options['eggs-directory'],
+ links=self._links,
+ index=options.get('index'),
+ executable=options['executable'],
+ path=[options['develop-eggs-directory']],
+ newest=self.newest,
+ allow_hosts=self._allow_hosts,
+ include_site_packages=self.include_site_packages_for_buildout
+ )
+
# Now copy buildout and setuptools eggs, and record destination eggs:
entries = []
for name in 'setuptools', 'zc.buildout':
r = pkg_resources.Requirement.parse(name)
- dist = pkg_resources.working_set.find(r)
+ dist = ws.find(r)
if dist.precedence == pkg_resources.DEVELOP_DIST:
dest = os.path.join(self['buildout']['develop-eggs-directory'],
name+'.egg-link')
@@ -366,8 +425,9 @@
ws = pkg_resources.WorkingSet(entries)
ws.require('zc.buildout')
zc.buildout.easy_install.scripts(
- ['zc.buildout'], ws, sys.executable,
- self['buildout']['bin-directory'])
+ ['zc.buildout'], ws, options['executable'],
+ options['bin-directory'],
+ include_site_packages=self.include_site_packages_for_buildout)
init = bootstrap
@@ -908,7 +968,7 @@
if not args:
raise zc.buildout.UserError(
"The setup command requires the path to a setup script or \n"
- "directory containing a setup script, and it's arguments."
+ "directory containing a setup script, and its arguments."
)
setup = args.pop(0)
if os.path.isdir(setup):
@@ -920,7 +980,7 @@
fd, tsetup = tempfile.mkstemp()
try:
os.write(fd, zc.buildout.easy_install.runsetup_template % dict(
- setuptools=pkg_resources_loc,
+ sys_path=',\n '.join(repr(p) for p in sys.path),
setupdir=os.path.dirname(setup),
setup=setup,
__file__ = setup,
Modified: zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/buildout.txt
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/buildout.txt 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/buildout.txt 2009-08-11 19:22:31 UTC (rev 102683)
@@ -710,18 +710,25 @@
Annotated sections
------------------
-When used with the `annotate` command, buildout displays annotated sections.
+When used with the `annotate` command, buildout displays annotated sections.
All sections are displayed, sorted alphabetically. For each section,
-all key-value pairs are displayed, sorted alphabetically, along with
-the origin of the value (file name or COMPUTED_VALUE, DEFAULT_VALUE,
+all key-value pairs are displayed, sorted alphabetically, along with
+the origin of the value (file name or COMPUTED_VALUE, DEFAULT_VALUE,
COMMAND_LINE_VALUE).
- >>> print system(buildout+ ' annotate'), # doctest: +ELLIPSIS
+ >>> print system(buildout+ ' annotate'),
+ ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
<BLANKLINE>
Annotated sections
==================
<BLANKLINE>
[buildout]
+ allow-hosts= *
+ DEFAULT_VALUE
+ allow-picked-versions= true
+ DEFAULT_VALUE
+ allowed-eggs-from-site-packages= *
+ DEFAULT_VALUE
bin-directory= bin
DEFAULT_VALUE
develop= recipes
@@ -734,18 +741,36 @@
DEFAULT_VALUE
executable= ...
DEFAULT_VALUE
+ find-links=
+ DEFAULT_VALUE
+ include-site-packages= true
+ DEFAULT_VALUE
+ include-site-packages-for-buildout= false
+ DEFAULT_VALUE
+ install-from-cache= false
+ DEFAULT_VALUE
installed= .installed.cfg
DEFAULT_VALUE
- log-format=
+ log-format=
DEFAULT_VALUE
log-level= INFO
DEFAULT_VALUE
+ newest= true
+ DEFAULT_VALUE
+ offline= false
+ DEFAULT_VALUE
parts= data-dir
/sample-buildout/buildout.cfg
parts-directory= parts
DEFAULT_VALUE
+ prefer-final= false
+ DEFAULT_VALUE
python= buildout
DEFAULT_VALUE
+ unzip= false
+ DEFAULT_VALUE
+ use-dependency-links= true
+ DEFAULT_VALUE
<BLANKLINE>
[data-dir]
path= foo bins
@@ -1075,7 +1100,8 @@
Annotated sections output shows which files are responsible for which
operations.
- >>> print system(os.path.join('bin', 'buildout') + ' annotate'), # doctest: +ELLIPSIS
+ >>> print system(os.path.join('bin', 'buildout') + ' annotate'),
+ ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
<BLANKLINE>
Annotated sections
==================
@@ -1088,7 +1114,7 @@
/sample-buildout/base.cfg
+= /sample-buildout/extension1.cfg
+= /sample-buildout/extension2.cfg
- recipe=
+ recipe=
/sample-buildout/base.cfg
<BLANKLINE>
[part2]
@@ -1096,7 +1122,7 @@
/sample-buildout/base.cfg
-= /sample-buildout/extension1.cfg
-= /sample-buildout/extension2.cfg
- recipe=
+ recipe=
/sample-buildout/base.cfg
<BLANKLINE>
[part3]
@@ -1104,13 +1130,12 @@
c3 c4 c5
/sample-buildout/base.cfg
+= /sample-buildout/extension1.cfg
- recipe=
+ recipe=
/sample-buildout/base.cfg
<BLANKLINE>
[part4]
option= h1 h2
/sample-buildout/extension1.cfg
- ...
Cleanup.
@@ -1820,13 +1845,13 @@
d parts
d recipes
- >>> cat(sample_buildout, '.installed.cfg')
+ >>> cat(sample_buildout, '.installed.cfg') # doctest: +NORMALIZE_WHITESPACE
[buildout]
installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link
parts = debug d1 d2 d3
<BLANKLINE>
[debug]
- __buildout_installed__ =
+ __buildout_installed__ =
__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==
recipe = recipes:debug
<BLANKLINE>
@@ -1905,13 +1930,13 @@
The .installed.cfg is only updated for the recipes that ran:
- >>> cat(sample_buildout, '.installed.cfg')
+ >>> cat(sample_buildout, '.installed.cfg') # doctest: +NORMALIZE_WHITESPACE
[buildout]
installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link
parts = debug d1 d2 d3 d4
<BLANKLINE>
[debug]
- __buildout_installed__ =
+ __buildout_installed__ =
__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==
recipe = recipes:debug
<BLANKLINE>
@@ -2108,27 +2133,37 @@
... parts =
... """)
- >>> print system(buildout+' -vv'),
+ >>> print system(buildout+' -vv'), # doctest: +NORMALIZE_WHITESPACE
Installing 'zc.buildout', 'setuptools'.
- We have a develop egg: zc.buildout 1.0.0.
+ We have a develop egg: zc.buildout X.X.
We have the best distribution that satisfies 'setuptools'.
- Picked: setuptools = 0.6
+ Picked: setuptools = V.V
<BLANKLINE>
Configuration data:
[buildout]
+ allow-hosts = *
+ allow-picked-versions = true
+ allowed-eggs-from-site-packages = *
bin-directory = /sample-buildout/bin
develop-eggs-directory = /sample-buildout/develop-eggs
directory = /sample-buildout
eggs-directory = /sample-buildout/eggs
- executable = /usr/local/bin/python2.3
+ executable = python
+ find-links =
+ include-site-packages = true
+ include-site-packages-for-buildout = false
+ install-from-cache = false
installed = /sample-buildout/.installed.cfg
- log-format =
+ log-format =
log-level = INFO
newest = true
offline = false
- parts =
+ parts =
parts-directory = /sample-buildout/parts
+ prefer-final = false
python = buildout
+ unzip = false
+ use-dependency-links = true
verbosity = 20
<BLANKLINE>
@@ -2136,6 +2171,70 @@
command-line assignments. We've discussed most of these options
already, but let's review them and touch on some we haven't discussed:
+allow-hosts
+ On some environments the links visited by `zc.buildout` can be forbidden by
+ paranoid firewalls. These URLs might be in the chain of links visited by
+ `zc.buildout` as defined by buildout's `find-links` option, or as defined
+ by various eggs in their `url`, `download_url`, `dependency_links` metadata.
+
+ The fact that package_index works like a spider and might visit links and
+ go to other locations makes this even harder.
+
+ The `allow-hosts` option provides a way to prevent this, and
+ works exactly like the one provided in `easy_install`.
+
+ You can provide a list of allowed host, together with wildcards::
+
+ [buildout]
+ ...
+
+ allow-hosts =
+ *.python.org
+ example.com
+
+ All URLs that does not match these hosts will not be visited.
+
+allow-picked-versions
+ By default, the buildout will choose the best match for a given requirement
+ if the requirement is not specified precisely (for instance, using the
+ "versions" option. This behavior corresponds to the
+ "allow-picked-versions" being set to its default value, "true". If
+ "allow-picked-versions" is "false," instead of picking the best match,
+ buildout will raise an error. This helps enforce repeatability.
+
+allowed-eggs-from-site-packages
+ Sometimes you need or want to control what eggs from site-packages are
+ used. The allowed-eggs-from-site-packages option allows you to specify a
+ whitelist of project names that may be included from site-packages. You
+ can use globs to specify the value. It defaults to a single value of '*',
+ indicating that any package may come from site-packages.
+
+ Here's a usage example::
+
+ [buildout]
+ ...
+
+ allowed-eggs-from-site-packages =
+ demo
+ bigdemo
+ zope.*
+
+ This option interacts with the ``include-site-packages`` option in the
+ following ways.
+
+ If ``include-site-packages`` is true, then
+ ``allowed-eggs-from-site-packages`` filters what eggs from site-packages
+ may be chosen. Therefore, if ``allowed-eggs-from-site-packages`` is an
+ empty list, then no eggs from site-packages are chosen, but site-packages
+ will still be included at the end of path lists.
+
+ If ``include-site-packages`` is false, the value of
+ ``allowed-eggs-from-site-packages`` is irrelevant.
+
+ The same pattern holds true for the ``include-site-packages-for-buildout``
+ option, except only the bin/buildout script is affected by that
+ interaction.
+
bin-directory
The directory path where scripts are written. This can be a
relative path, which is interpreted relative to the directory
@@ -2160,6 +2259,77 @@
The Python executable used to run the buildout. See the python
option below.
+find-links
+ You can specify more locations to search for distributions using the
+ `find-links` option. All locations specified will be searched for
+ distributions along with the package index as described before.
+
+ Locations can be urls::
+
+ [buildout]
+ ...
+ find-links = http://download.zope.org/distribution/
+
+ They can also be directories on disk::
+
+ [buildout]
+ ...
+ find-links = /some/path
+
+ Finally, they can also be direct paths to distributions::
+
+ [buildout]
+ ...
+ find-links = /some/path/someegg-1.0.0-py2.3.egg
+
+ Any number of locations can be specified in the `find-links` option::
+
+ [buildout]
+ ...
+ find-links =
+ http://download.zope.org/distribution/
+ /some/otherpath
+ /some/path/someegg-1.0.0-py2.3.egg
+
+include-site-packages
+ By default, buildout will look for dependencies in the system's
+ site-packages. For this purpose, paths outside of Python's standard
+ library--or more precisely, those that are not included when Python is
+ started with the -S argument--are loosely referred to as "site-packages"
+ here. The include-site-packages buildout option can be used to override
+ the default behavior of using site packages
+ ("include-site-packages = false").
+
+include-site-packages-for-buildout
+ When buildout gets a recipe egg (as opposed to runs a recipe), it starts
+ with the current Python working set--the one that the bin/buildout script
+ uses itself. If this working set includes site-packages, and site-packages
+ includes an egg for a package that the recipe needs, *and* the recipe
+ specifies a newer version of that package, this can generate a version
+ conflict.
+
+ One solution to this is to not use site-packages
+ (``include-site-packages = false``). However, if you want your scripts to
+ use site-packages, then you have to specify 'include-site-packages = true'
+ for all of them.
+
+ To make this use case easier to handle, you can instead specify
+ ``include-site-packages-with-buildout = false``, which indicates that the
+ bin/buildout should *not* use site-packages; and
+ ``include-site-packages = true``, which indicates that the rest of the
+ scripts should use site-packages.
+
+ This is the default configuration.
+
+install-from-cache
+ A download cache can be used as the basis of application source releases.
+ In an application source release, we want to distribute an application that
+ can be built without making any network accesses. In this case, we
+ distribute a buildout with download cache and tell the buildout to install
+ from the download cache only, without making network accesses. The
+ buildout install-from-cache option can be used to signal that packages
+ should be installed only from the download cache.
+
installed
The file path where information about the results of the previous
buildout run is written. This can be a relative path, which is
@@ -2173,12 +2343,51 @@
log-level
The log level before verbosity adjustment
+newest
+ By default buildout and recipes will try to find the newest versions of
+ distributions needed to satisfy requirements. This can be very time
+ consuming, especially when incrementally working on setting up a buildout
+ or working on a recipe. The buildout "newest" option can be used to to
+ suppress this. If the "newest" option is set to false, then new
+ distributions won't be sought if an installed distribution meets
+ requirements. The "newest" option can also be set to false using the -N
+ command-line option. See also the "offline" option.
+
+offline
+ The "offline" option goes a bit further than the "newest" option. If the
+ buildout "offline" option is given a value of "true", the buildout and
+ recipes that are aware of the option will avoid doing network access. This
+ is handy when running the buildout when not connected to the internet. It
+ also makes buildouts run much faster. This option is typically set using
+ the buildout -o option.
+
parts
A white space separated list of parts to be installed.
parts-directory
A working directory that parts can used to store data.
+prefer-final
+ Currently, when searching for new releases, the newest available
+ release is used. This isn't usually ideal, as you may get a
+ development release or alpha releases not ready to be widely used.
+ You can request that final releases be preferred using the prefer
+ final option in the buildout section::
+
+ [buildout]
+ ...
+ prefer-final = true
+
+ When the prefer-final option is set to true, then when searching for
+ new releases, final releases are preferred. If there are final
+ releases that satisfy distribution requirements, then those releases
+ are used even if newer non-final releases are available. The buildout
+ prefer-final option can be used to override this behavior.
+
+ In buildout version 2, final releases will be preferred by default.
+ You will then need to use a false value for prefer-final to get the
+ newest releases.
+
python
The name of a section containing information about the default
Python interpreter. Recipes that need a installation
@@ -2189,11 +2398,30 @@
Python executable. By default, the buildout section defines the
default Python as the Python used to run the buildout.
+unzip
+ By default, zc.buildout doesn't unzip zip-safe eggs ("unzip = false").
+ This follows the policy followed by setuptools itself. Experience shows
+ this policy to to be inconvenient. Zipped eggs make debugging more
+ difficult and often import more slowly. You can include an unzip option in
+ the buildout section to change the default unzipping policy ("unzip =
+ true").
+
+use-dependency-links
+ By default buildout will obey the setuptools dependency_links metadata
+ when it looks for dependencies. This behavior can be controlled with
+ the use-dependency-links buildout option::
+
+ [buildout]
+ ...
+ use-dependency-links = false
+
+ The option defaults to true. If you set it to false, then dependency
+ links are only looked for in the locations specified by find-links.
+
verbosity
A log-level adjustment. Typically, this is set via the -q and -v
command-line options.
-
Creating new buildouts and bootstrapping
----------------------------------------
@@ -2205,6 +2433,7 @@
>>> print system(buildout
... +' -c'+os.path.join(sample_bootstrapped, 'setup.cfg')
+ ... +' buildout:include-site-packages-for-buildout=true'
... +' init'),
Creating '/sample-bootstrapped/setup.cfg'.
Creating directory '/sample-bootstrapped/bin'.
@@ -2242,23 +2471,25 @@
normally use the bootstrap command instead of init. It will complain
if there isn't a configuration file:
- >>> sample_bootstrapped2 = tmpdir('sample-bootstrapped2')
+ >>> sample_bootstrapped2 = tmpdir('sample-bootstrapped2')
- >>> print system(buildout
- ... +' -c'+os.path.join(sample_bootstrapped2, 'setup.cfg')
- ... +' bootstrap'),
- While:
- Initializing.
- Error: Couldn't open /sample-bootstrapped2/setup.cfg
+ >>> print system(buildout
+ ... +' -c'+os.path.join(sample_bootstrapped2, 'setup.cfg')
+ ... +' buildout:include-site-packages-for-buildout=true'
+ ... +' bootstrap'),
+ While:
+ Initializing.
+ Error: Couldn't open /sample-bootstrapped2/setup.cfg
- >>> write(sample_bootstrapped2, 'setup.cfg',
- ... """
- ... [buildout]
- ... parts =
- ... """)
+ >>> write(sample_bootstrapped2, 'setup.cfg',
+ ... """
+ ... [buildout]
+ ... parts =
+ ... """)
>>> print system(buildout
... +' -c'+os.path.join(sample_bootstrapped2, 'setup.cfg')
+ ... +' buildout:include-site-packages-for-buildout=true'
... +' bootstrap'),
Creating directory '/sample-bootstrapped2/bin'.
Creating directory '/sample-bootstrapped2/parts'.
@@ -2266,49 +2497,6 @@
Creating directory '/sample-bootstrapped2/develop-eggs'.
Generated script '/sample-bootstrapped2/bin/buildout'.
-
-Newest and Offline Modes
-------------------------
-
-By default buildout and recipes will try to find the newest versions
-of distributions needed to satisfy requirements. This can be very
-time consuming, especially when incrementally working on setting up a
-buildout or working on a recipe. The buildout newest option can be
-used to to suppress this. If the newest option is set to false, then
-new distributions won't be sought if an installed distribution meets
-requirements. The newest option can be set to false using the -N
-command-line option.
-
-The offline option goes a bit further. If the buildout offline option
-is given a value of "true", the buildout and recipes that are aware of
-the option will avoid doing network access. This is handy when
-running the buildout when not connected to the internet. It also
-makes buildouts run much faster. This option is typically set using
-the buildout -o option.
-
-Preferring Final Releases
--------------------------
-
-Currently, when searching for new releases, the newest available
-release is used. This isn't usually ideal, as you may get a
-development release or alpha releases not ready to be widely used.
-You can request that final releases be preferred using the prefer
-final option in the buildout section::
-
- [buildout]
- ...
- prefer-final = true
-
-When the prefer-final option is set to true, then when searching for
-new releases, final releases are preferred. If there are final
-releases that satisfy distribution requirements, then those releases
-are used even if newer non-final releases are available. The buildout
-prefer-final option can be used to override this behavior.
-
-In buildout version 2, final releases will be preferred by default.
-You will then need to use a false value for prefer-final to get the
-newest releases.
-
Finding distributions
---------------------
@@ -2327,50 +2515,9 @@
requirements of the buildout will always be used.
You can also specify more locations to search for distributions using
-the `find-links` option. All locations specified will be searched for
-distributions along with the package index as described before.
+the `find-links` option. See its description above.
-Locations can be urls::
- [buildout]
- ...
- find-links = http://download.zope.org/distribution/
-
-They can also be directories on disk::
-
- [buildout]
- ...
- find-links = /some/path
-
-Finally, they can also be direct paths to distributions::
-
- [buildout]
- ...
- find-links = /some/path/someegg-1.0.0-py2.3.egg
-
-Any number of locations can be specified in the `find-links` option::
-
- [buildout]
- ...
- find-links =
- http://download.zope.org/distribution/
- /some/otherpath
- /some/path/someegg-1.0.0-py2.3.egg
-
-Dependency links
-----------------
-
-By default buildout will obey the setuptools dependency_links metadata
-when it looks for dependencies. This behavior can be controlled with
-the use-dependency-links buildout option::
-
- [buildout]
- ...
- use-dependency-links = false
-
-The option defaults to true. If you set it to false, then dependency
-links are only looked for in the locations specified by find-links.
-
Controlling the installation database
-------------------------------------
@@ -2532,32 +2679,6 @@
Develop: '/sample-bootstrapped/demo'
unload ['buildout']
-Allow hosts
------------
-
-On some environments the links visited by `zc.buildout` can be forbidden
-by paranoiac firewalls. These URL might be on the chain of links
-visited by `zc.buildout` wheter they are defined in the `find-links` option,
-wheter they are defined by various eggs in their `url`, `download_url`,
-`dependency_links` metadata.
-
-It is even harder to track that package_index works like a spider and
-might visit links and go to other location.
-
-The `allow-hosts` option provides a way to prevent this, and
-works exactly like the one provided in `easy_install`.
-
-You can provide a list of allowed host, together with wildcards::
-
- [buildout]
- ...
-
- allow-hosts =
- *.python.org
- example.com
-
-All urls that does not match these hosts will not be visited.
-
.. [#future_recipe_methods] In the future, additional methods may be
added. Older recipes with fewer methods will still be
supported.
Modified: zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/easy_install.py
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/easy_install.py 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/easy_install.py 2009-08-11 19:22:31 UTC (rev 102683)
@@ -21,6 +21,7 @@
"""
import distutils.errors
+import fnmatch
import glob
import logging
import os
@@ -64,13 +65,55 @@
pkg_resources.Requirement.parse('setuptools')
).location
-# Include buildout and setuptools eggs in paths
-buildout_and_setuptools_path = [
- setuptools_loc,
- pkg_resources.working_set.find(
- pkg_resources.Requirement.parse('zc.buildout')).location,
- ]
+# Include buildout and setuptools eggs in paths. We prevent dupes just to
+# keep from duplicating any log messages about them.
+buildout_loc = pkg_resources.working_set.find(
+ pkg_resources.Requirement.parse('zc.buildout')).location
+buildout_and_setuptools_path = [setuptools_loc]
+if os.path.normpath(setuptools_loc) != os.path.normpath(buildout_loc):
+ buildout_and_setuptools_path.append(buildout_loc)
+def _get_system_packages(executable):
+ """return a pair of the standard lib and site packages for the executable.
+ """
+ # We want to get a list of the site packages, which is not easy. The
+ # canonical way to do this is to use distutils.sysconfig.get_python_lib(),
+ # but that only returns a single path, which does not reflect reality for
+ # many system Pythons, which have multiple additions. Instead, we start
+ # Python with -S, which does not import site.py and set up the extra paths
+ # like site-packages or (Ubuntu/Debian) dist-packages and python-support.
+ # We then compare that sys.path with the normal one. The set of the normal
+ # one minus the set of the ones in ``python -S`` is the set of packages
+ # that are effectively site-packages.
+ def get_sys_path(clean=False):
+ cmd = [executable, "-c",
+ "import sys, os;"
+ "print repr([os.path.normpath(p) for p in sys.path])"]
+ if clean:
+ cmd.insert(1, '-S')
+ _proc = subprocess.Popen(
+ cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ stdout, stderr = _proc.communicate();
+ if _proc.returncode:
+ raise RuntimeError(
+ 'error trying to get system packages:\n%s' % (stderr,))
+ res = eval(stdout)
+ try:
+ res.remove('.')
+ except ValueError:
+ pass
+ return res
+ stdlib = get_sys_path(clean=True)
+ # The given executable might not be the current executable, so it is
+ # appropriate to do another subprocess to figure out what the additional
+ # site-package paths are. Moreover, even if this executable *is* the
+ # current executable, this code might be run in the context of code that
+ # has manipulated the sys.path--for instance, to add local zc.buildout or
+ # setuptools eggs.
+ site_packages = [p for p in get_sys_path() if p not in stdlib]
+ return (stdlib, site_packages)
+
+
class IncompatibleVersionError(zc.buildout.UserError):
"""A specified version is incompatible with a given requirement.
"""
@@ -99,6 +142,7 @@
FILE_SCHEME = re.compile('file://', re.I).match
+
class AllowHostsPackageIndex(setuptools.package_index.PackageIndex):
"""Will allow urls that are local to the system.
@@ -111,7 +155,12 @@
_indexes = {}
-def _get_index(executable, index_url, find_links, allow_hosts=('*',)):
+def _get_index(executable, index_url, find_links, allow_hosts=('*',),
+ path=None):
+ # If path is None, the index will use sys.path. If you provide an empty
+ # path ([]), it will complain uselessly about missing index pages for
+ # packages found in the paths that you expect to use. Therefore, this path
+ # is always the same as the _env path in the Installer.
key = executable, index_url, tuple(find_links)
index = _indexes.get(key)
if index is not None:
@@ -120,7 +169,8 @@
if index_url is None:
index_url = default_index_url
index = AllowHostsPackageIndex(
- index_url, hosts=allow_hosts, python=_get_version(executable)
+ index_url, hosts=allow_hosts, search_path=path,
+ python=_get_version(executable)
)
if find_links:
@@ -152,6 +202,8 @@
_use_dependency_links = True
_allow_picked_versions = True
_always_unzip = False
+ _include_site_packages = True
+ _allowed_eggs_from_site_packages = ('*',)
def __init__(self,
dest=None,
@@ -163,6 +215,8 @@
newest=True,
versions=None,
use_dependency_links=None,
+ include_site_packages=None,
+ allowed_eggs_from_site_packages=None,
allow_hosts=('*',)
):
self._dest = dest
@@ -185,7 +239,19 @@
self._executable = executable
if always_unzip is not None:
self._always_unzip = always_unzip
- path = (path and path[:] or []) + buildout_and_setuptools_path
+ path = (path and path[:] or [])
+ if include_site_packages is not None:
+ self._include_site_packages = include_site_packages
+ if allowed_eggs_from_site_packages is not None:
+ self._allowed_eggs_from_site_packages = tuple(
+ allowed_eggs_from_site_packages)
+ stdlib, self._site_packages = _get_system_packages(executable)
+ if self._include_site_packages:
+ path.extend(buildout_and_setuptools_path)
+ path.extend(self._site_packages)
+ # else we could try to still include the buildout_and_setuptools_path
+ # if the elements are not in site_packages, but we're not bothering
+ # with this optimization for now, in the name of code simplicity.
if dest is not None and dest not in path:
path.insert(0, dest)
self._path = path
@@ -194,13 +260,42 @@
self._newest = newest
self._env = pkg_resources.Environment(path,
python=_get_version(executable))
- self._index = _get_index(executable, index, links, self._allow_hosts)
+ self._index = _get_index(executable, index, links, self._allow_hosts,
+ self._path)
if versions is not None:
self._versions = versions
+ _allowed_eggs_from_site_packages_regex = None
+ def allow_site_package_egg(self, name):
+ if (not self._include_site_packages or
+ not self._allowed_eggs_from_site_packages):
+ # If the answer is a blanket "no," perform a shortcut.
+ return False
+ if self._allowed_eggs_from_site_packages_regex is None:
+ pattern = '(%s)' % (
+ '|'.join(
+ fnmatch.translate(name)
+ for name in self._allowed_eggs_from_site_packages),
+ )
+ self._allowed_eggs_from_site_packages_regex = re.compile(pattern)
+ return bool(self._allowed_eggs_from_site_packages_regex.match(name))
+
def _satisfied(self, req, source=None):
- dists = [dist for dist in self._env[req.project_name] if dist in req]
+ # We get all distributions that match the given requirement. If we are
+ # not supposed to include site-packages for the given egg, we also
+ # filter those out. Even if include_site_packages is False and so we
+ # have excluded site packages from the _env's paths (see
+ # Installer.__init__), we need to do the filtering here because an
+ # .egg-link, such as one for setuptools or zc.buildout installed by
+ # zc.buildout.buildout.Buildout.bootstrap, can indirectly include a
+ # path in our _site_packages.
+ dists = [dist for dist in self._env[req.project_name] if (
+ dist in req and (
+ dist.location not in self._site_packages or
+ self.allow_site_package_egg(dist.project_name))
+ )
+ ]
if not dists:
logger.debug('We have no distributions for %s that satisfies %r.',
req.project_name, str(req))
@@ -346,8 +441,8 @@
if exit_code:
logger.error(
- "An error occured when trying to install %s."
- "Look above this message for any errors that"
+ "An error occured when trying to install %s. "
+ "Look above this message for any errors that "
"were output by easy_install.",
dist)
@@ -404,10 +499,20 @@
# Nothing is available.
return None
- # Filter the available dists for the requirement and source flag
+ # Filter the available dists for the requirement and source flag. If
+ # we are not supposed to include site-packages for the given egg, we
+ # also filter those out. Even if include_site_packages is False and so
+ # we have excluded site packages from the _env's paths (see
+ # Installer.__init__), we need to do the filtering here because an
+ # .egg-link, such as one for setuptools or zc.buildout installed by
+ # zc.buildout.buildout.Buildout.bootstrap, can indirectly include a
+ # path in our _site_packages.
dists = [dist for dist in index[requirement.project_name]
if ((dist in requirement)
and
+ (dist.location not in self._site_packages or
+ self.allow_site_package_egg(dist.project_name))
+ and
((not source) or
(dist.precedence == pkg_resources.SOURCE_DIST)
)
@@ -571,7 +676,7 @@
self._links.append(link)
self._index = _get_index(self._executable,
self._index_url, self._links,
- self._allow_hosts)
+ self._allow_hosts, self._path)
for dist in dists:
# Check whether we picked a version and, if we did, report it:
@@ -630,9 +735,9 @@
logger.debug('Installing %s.', repr(specs)[1:-1])
path = self._path
- dest = self._dest
- if dest is not None and dest not in path:
- path.insert(0, dest)
+ destination = self._dest
+ if destination is not None and destination not in path:
+ path.insert(0, destination)
requirements = [self._constrain(pkg_resources.Requirement.parse(spec))
for spec in specs]
@@ -650,35 +755,55 @@
self._maybe_add_setuptools(ws, dist)
# OK, we have the requested distributions and they're in the working
- # set, but they may have unmet requirements. We'll simply keep
- # trying to resolve requirements, adding missing requirements as they
- # are reported.
+ # set, but they may have unmet requirements. We'll resolve these
+ # requirements. This is code modified from
+ # pkg_resources.WorkingSet.resolve. We can't reuse that code directly
+ # because we have to constrain our requirements (see
+ # versions_section_ignored_for_dependency_in_favor_of_site_packages in
+ # zc.buildout.tests).
#
- # Note that we don't pass in the environment, because we want
+ requirements.reverse() # Set up the stack.
+ processed = {} # This is a set of processed requirements.
+ best = {} # This is a mapping of key -> dist.
+ #
+ # Note that we don't use the existing environment, because we want
# to look for new eggs unless what we have is the best that
# matches the requirement.
- while 1:
- try:
- ws.resolve(requirements)
- except pkg_resources.DistributionNotFound, err:
- [requirement] = err
- requirement = self._constrain(requirement)
- if dest:
- logger.debug('Getting required %r', str(requirement))
- else:
- logger.debug('Adding required %r', str(requirement))
- _log_requirement(ws, requirement)
+ env = pkg_resources.Environment(ws.entries)
+ while requirements:
+ # Process dependencies breadth-first.
+ req = self._constrain(requirements.pop(0))
+ if req in processed:
+ # Ignore cyclic or redundant dependencies.
+ continue
+ dist = best.get(req.key)
+ if dist is None:
+ # Find the best distribution and add it to the map.
+ dist = ws.by_key.get(req.key)
+ if dist is None:
+ try:
+ dist = best[req.key] = env.best_match(req, ws)
+ except pkg_resources.VersionConflict, err:
+ raise VersionConflict(err, ws)
+ if dist is None:
+ if destination:
+ logger.debug('Getting required %r', str(req))
+ else:
+ logger.debug('Adding required %r', str(req))
+ _log_requirement(ws, req)
+ for dist in self._get_dist(req,
+ ws, self._always_unzip):
+ ws.add(dist)
+ self._maybe_add_setuptools(ws, dist)
+ if dist not in req:
+ # Oops, the "best" so far conflicts with a dependency.
+ raise VersionConflict(
+ pkg_resources.VersionConflict(dist, req), ws)
+ requirements.extend(dist.requires(req.extras)[::-1])
+ processed[req] = True
+ if dist.location in self._site_packages:
+ logger.debug('Egg from site-packages: %s', dist)
- for dist in self._get_dist(requirement, ws, self._always_unzip
- ):
-
- ws.add(dist)
- self._maybe_add_setuptools(ws, dist)
- except pkg_resources.VersionConflict, err:
- raise VersionConflict(err, ws)
- else:
- break
-
return ws
def build(self, spec, build_ext):
@@ -773,6 +898,18 @@
Installer._prefer_final = bool(setting)
return old
+def include_site_packages(setting=None):
+ old = Installer._include_site_packages
+ if setting is not None:
+ Installer._include_site_packages = bool(setting)
+ return old
+
+def allowed_eggs_from_site_packages(setting=None):
+ old = Installer._allowed_eggs_from_site_packages
+ if setting is not None:
+ Installer._allowed_eggs_from_site_packages = tuple(setting)
+ return old
+
def use_dependency_links(setting=None):
old = Installer._use_dependency_links
if setting is not None:
@@ -795,9 +932,12 @@
links=(), index=None,
executable=sys.executable, always_unzip=None,
path=None, working_set=None, newest=True, versions=None,
- use_dependency_links=None, allow_hosts=('*',)):
+ use_dependency_links=None, include_site_packages=None,
+ allowed_eggs_from_site_packages=None, allow_hosts=('*',)):
installer = Installer(dest, links, index, executable, always_unzip, path,
newest, versions, use_dependency_links,
+ include_site_packages,
+ allowed_eggs_from_site_packages,
allow_hosts=allow_hosts)
return installer.install(specs, working_set)
@@ -805,9 +945,12 @@
def build(spec, dest, build_ext,
links=(), index=None,
executable=sys.executable,
- path=None, newest=True, versions=None, allow_hosts=('*',)):
+ path=None, newest=True, versions=None, include_site_packages=None,
+ allowed_eggs_from_site_packages=None, allow_hosts=('*',)):
installer = Installer(dest, links, index, executable, True, path, newest,
- versions, allow_hosts=allow_hosts)
+ versions, include_site_packages,
+ allowed_eggs_from_site_packages,
+ allow_hosts=allow_hosts)
return installer.build(spec, build_ext)
@@ -866,7 +1009,7 @@
undo.append(lambda: os.close(fd))
os.write(fd, runsetup_template % dict(
- setuptools=setuptools_loc,
+ sys_path=',\n '.join(repr(p) for p in sys.path),
setupdir=directory,
setup=setup,
__file__ = setup,
@@ -903,22 +1046,65 @@
[f() for f in undo]
-def working_set(specs, executable, path):
- return install(specs, None, executable=executable, path=path)
+def working_set(specs, executable, path, include_site_packages=None,
+ allowed_eggs_from_site_packages=None):
+ return install(
+ specs, None, executable=executable, path=path,
+ include_site_packages=include_site_packages,
+ allowed_eggs_from_site_packages=allowed_eggs_from_site_packages)
+def get_path(working_set, executable, extra_paths=(),
+ include_site_packages=True):
+ """Given working set and path to executable, return value for sys.path.
+
+ Distribution locations from the working set come first in the list. Within
+ that collection, this function pushes site-packages-based distribution
+ locations to the end of the list, so that they don't mask eggs.
+
+ This expects that the working_set has already been created to honor a
+ include_site_packages setting. That is, if include_site_packages is False,
+ this function does *not* verify that the working_set's distributions are
+ not in site packages.
+
+ However, it does explicitly include site packages if include_site_packages
+ is True.
+
+ The standard library (defined as what the given Python executable has on
+ the path before its site.py is run) is always included.
+ """
+ stdlib, site_packages = _get_system_packages(executable)
+ postponed = []
+ path = []
+ for dist in working_set:
+ location = os.path.normpath(dist.location)
+ if location in path:
+ path.remove(location)
+ postponed.append(location)
+ elif location in site_packages:
+ postponed.append(location)
+ site_packages.remove(location)
+ elif location not in postponed:
+ path.append(location)
+ path.extend(postponed)
+ path.extend(extra_paths)
+ # now we add in all paths
+ if include_site_packages:
+ path.extend(site_packages) # these are the remaining site_packages
+ path.extend(stdlib)
+ path = map(realpath, path)
+ return path
+
def scripts(reqs, working_set, executable, dest,
scripts=None,
extra_paths=(),
arguments='',
interpreter=None,
initialization='',
- relative_paths=False,
+ include_site_packages=True,
+ relative_paths=False
):
-
- path = [dist.location for dist in working_set]
- path.extend(extra_paths)
- path = map(realpath, path)
-
+ path = get_path(
+ working_set, executable, extra_paths, include_site_packages)
generated = []
if isinstance(reqs, str):
@@ -969,7 +1155,7 @@
if relative_paths:
relative_paths = os.path.normcase(relative_paths)
sname = os.path.normcase(os.path.abspath(sname))
- spath = ',\n '.join(
+ spath = ',\n '.join(
[_relativitize(os.path.normcase(path_item), sname, relative_paths)
for path_item in path]
)
@@ -977,7 +1163,7 @@
for i in range(_relative_depth(relative_paths, sname)):
rpsetup += "base = os.path.dirname(base)\n"
else:
- spath = repr(path)[1:-1].replace(', ', ',\n ')
+ spath = repr(path)[1:-1].replace(', ', ',\n ')
rpsetup = ''
return spath, rpsetup
@@ -1074,10 +1260,11 @@
script_template = script_header + '''\
%(relative_paths_setup)s
-import sys
-sys.path[0:0] = [
- %(path)s,
- ]
+import sys, os
+pythonpath = filter(None, os.environ.get('PYTHONPATH', '').split(':'))
+sys.path[:] = pythonpath + [
+ %(path)s,
+ ]
%(initialization)s
import %(module_name)s
@@ -1120,37 +1307,70 @@
py_script_template = script_header + '''\
+globs = globals().copy() # get a clean copy early
+
%(relative_paths_setup)s
-import sys
+import sys, os
-sys.path[0:0] = [
- %(path)s,
- ]
+_set_path = _interactive = True
+_force_interactive = False
-_interactive = True
-if len(sys.argv) > 1:
- import getopt
- _options, _args = getopt.getopt(sys.argv[1:], 'ic:')
+_command = None
+_args = sys.argv[1:]
+
+while _args:
+ if _args[0].startswith('-'):
+ _arg = _args.pop(0)
+ for _ix, _opt in enumerate(_arg[1:]):
+ if _opt == 'i':
+ _force_interactive = True
+ elif _opt == 'c':
+ _interactive = False
+ _command = _args.pop(0) # Argument expected for the -c option
+ _args.insert(0, '-c')
+ break
+ elif _opt == 'S':
+ # We'll approximate this. It is mostly convenient for tests.
+ _set_path = False
+ elif _opt == 'V':
+ print 'Python ' + sys.version.split()[0]
+ _interactive = False
+ break
+ else:
+ continue
+ break
+ else:
+ break
+
+if _set_path:
+ sys.path[:] = [
+ %(path)s,
+ ]
+pythonpath = filter(None, os.environ.get('PYTHONPATH', '').split(':'))
+sys.path[0:0] = pythonpath
+sys.path.insert(0, '.')
+
+sys.argv[:] = _args
+
+if _command:
+ exec _command
+elif _args:
_interactive = False
- for (_opt, _val) in _options:
- if _opt == '-i':
- _interactive = True
- elif _opt == '-c':
- exec _val
+ globs['__file__'] = sys.argv[0]
+ execfile(sys.argv[0], globs)
- if _args:
- sys.argv[:] = _args
- execfile(sys.argv[0])
-
-if _interactive:
+if _interactive or _force_interactive:
import code
- code.interact(banner="", local=globals())
+ del globs['__file__']
+ code.interact(banner="", local=globs)
'''
runsetup_template = """
import sys
-sys.path.insert(0, %(setupdir)r)
-sys.path.insert(0, %(setuptools)r)
+sys.path[:] = [
+ %(setupdir)r,
+ %(sys_path)s
+ ]
import os, setuptools
__file__ = %(__file__)r
@@ -1246,4 +1466,3 @@
subprocess.call([sys.executable, args])
else:
os.spawnv(os.P_WAIT, sys.executable, args)
-
Modified: zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/easy_install.txt
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/easy_install.txt 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/easy_install.txt 2009-08-11 19:22:31 UTC (rev 102683)
@@ -89,6 +89,14 @@
for using dependency_links in preference to other
locations. Defaults to true.
+include_site_packages
+ A flag indicating whether Python's non-standard-library packages should
+ be available for finding dependencies. Defaults to true.
+
+ Paths outside of Python's standard library--or more precisely, those that
+ are not included when Python is started with the -S argument--are loosely
+ referred to as "site-packages" here.
+
relative_paths
Adjust egg paths so they are relative to the script path. This
allows scripts to work when scripts and eggs are moved, as long as
@@ -399,6 +407,65 @@
>>> [d.version for d in ws]
['0.3', '1.1']
+Dependencies in Site Packages
+-----------------------------
+
+Paths outside of Python's standard library--or more precisely, those that are
+not included when Python is started with the -S argument--are loosely referred
+to as "site-packages" here. These site-packages are searched by default for
+distributions. This can be disabled, so that, for instance, a system Python
+can be used with buildout, cleaned of any packages installed by a user or
+system package manager.
+
+The default behavior can be controlled and introspected using
+zc.buildout.easy_install.include_site_packages.
+
+ >>> zc.buildout.easy_install.include_site_packages()
+ True
+
+Here's an example of using a Python executable that includes our dependencies.
+
+Our "primed_executable" has the "demoneeded," "other," and "setuptools"
+packages available. We'll simply be asking for "other" here.
+
+ >>> primed_executable = get_executable_with_site_packages()
+
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['other'], example_dest, links=[], executable=primed_executable,
+ ... index=None)
+ >>> [dist.project_name for dist in workingset]
+ ['other']
+
+That worked fine. Let's try again with site packages not allowed. We'll
+change the policy by changing the default. Notice that the function for
+changing the default value returns the previous value.
+
+ >>> zc.buildout.easy_install.include_site_packages(False)
+ True
+
+ >>> zc.buildout.easy_install.include_site_packages()
+ False
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['other'], example_dest, links=[], executable=primed_executable,
+ ... index=None)
+ Traceback (most recent call last):
+ ...
+ MissingDistribution: Couldn't find a distribution for 'other'.
+ >>> zc.buildout.easy_install.clear_index_cache()
+
+Now we'll reset the default.
+
+ >>> zc.buildout.easy_install.include_site_packages(True)
+ False
+
+ >>> zc.buildout.easy_install.include_site_packages()
+ True
+
Dependency links
----------------
@@ -580,14 +647,16 @@
The demo script run the entry point defined in the demo egg:
- >>> cat(bin, 'demo') # doctest: +NORMALIZE_WHITESPACE
+ >>> cat(bin, 'demo') # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
#!/usr/local/bin/python2.4
<BLANKLINE>
- import sys
- sys.path[0:0] = [
- '/sample-install/demo-0.3-py2.4.egg',
- '/sample-install/demoneeded-1.1-py2.4.egg',
- ]
+ import sys, os
+ pythonpath = filter(None, os.environ.get('PYTHONPATH', '').split(':'))
+ sys.path[:] = pythonpath + [
+ '/sample-install/demo-0.3-py2.4.egg',
+ '/sample-install/demoneeded-1.1-py2.4.egg',
+ ...
+ ]
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
@@ -596,7 +665,8 @@
Some things to note:
-- The demo and demoneeded eggs are added to the beginning of sys.path.
+- The demo and demoneeded eggs are at the beginning of sys.path. The script
+ controls the entire path.
- The module for the script entry point is imported and the entry
point, in this case, 'main', is run.
@@ -617,14 +687,16 @@
... [('demo', 'eggrecipedemo', 'main')],
... ws, sys.executable, bin)
- >>> cat(bin, 'demo') # doctest: +NORMALIZE_WHITESPACE
+ >>> cat(bin, 'demo') # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
#!/usr/local/bin/python2.4
<BLANKLINE>
- import sys
- sys.path[0:0] = [
- '/sample-install/demo-0.3-py2.4.egg',
- '/sample-install/demoneeded-1.1-py2.4.egg',
- ]
+ import sys, os
+ pythonpath = filter(None, os.environ.get('PYTHONPATH', '').split(':'))
+ sys.path[:] = pythonpath + [
+ '/sample-install/demo-0.3-py2.4.egg',
+ '/sample-install/demoneeded-1.1-py2.4.egg',
+ ...
+ ]
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
@@ -661,33 +733,65 @@
The py script simply runs the Python interactive interpreter with
the path set:
- >>> cat(bin, 'py') # doctest: +NORMALIZE_WHITESPACE
+ >>> cat(bin, 'py') # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
#!/usr/local/bin/python2.4
- import sys
+ globs = globals().copy() # get a clean copy early
<BLANKLINE>
- sys.path[0:0] = [
- '/sample-install/demo-0.3-py2.4.egg',
- '/sample-install/demoneeded-1.1-py2.4.egg',
- ]
+ import sys, os
<BLANKLINE>
- _interactive = True
- if len(sys.argv) > 1:
- import getopt
- _options, _args = getopt.getopt(sys.argv[1:], 'ic:')
+ _set_path = _interactive = True
+ _force_interactive = False
+ <BLANKLINE>
+ _command = None
+ _args = sys.argv[1:]
+ <BLANKLINE>
+ while _args:
+ if _args[0].startswith('-'):
+ _arg = _args.pop(0)
+ for _ix, _opt in enumerate(_arg[1:]):
+ if _opt == 'i':
+ _force_interactive = True
+ elif _opt == 'c':
+ _interactive = False
+ _command = _args.pop(0) # Argument expected for the -c option
+ _args.insert(0, '-c')
+ break
+ elif _opt == 'S':
+ # We'll approximate this. It is mostly convenient for tests.
+ _set_path = False
+ elif _opt == 'V':
+ print 'Python ' + sys.version.split()[0]
+ _interactive = False
+ break
+ else:
+ continue
+ break
+ else:
+ break
+ <BLANKLINE>
+ if _set_path:
+ sys.path[:] = [
+ '/sample-install/demo-0.3-pyN.N.egg',
+ '/sample-install/demoneeded-1.1-pyN.N.egg',
+ ...
+ ]
+ pythonpath = filter(None, os.environ.get('PYTHONPATH', '').split(':'))
+ sys.path[0:0] = pythonpath
+ sys.path.insert(0, '.')
+ <BLANKLINE>
+ sys.argv[:] = _args
+ <BLANKLINE>
+ if _command:
+ exec _command
+ elif _args:
_interactive = False
- for (_opt, _val) in _options:
- if _opt == '-i':
- _interactive = True
- elif _opt == '-c':
- exec _val
+ globs['__file__'] = sys.argv[0]
+ execfile(sys.argv[0], globs)
<BLANKLINE>
- if _args:
- sys.argv[:] = _args
- execfile(sys.argv[0])
- <BLANKLINE>
- if _interactive:
+ if _interactive or _force_interactive:
import code
- code.interact(banner="", local=globals())
+ del globs['__file__']
+ code.interact(banner="", local=globs)
If invoked with a script name and arguments, it will run that script, instead.
@@ -722,14 +826,16 @@
... ['demo'], ws, sys.executable, bin, dict(demo='run'),
... extra_paths=[foo])
- >>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE
+ >>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
#!/usr/local/bin/python2.4
<BLANKLINE>
- import sys
- sys.path[0:0] = [
+ import sys, os
+ pythonpath = filter(None, os.environ.get('PYTHONPATH', '').split(':'))
+ sys.path[:] = pythonpath + [
'/sample-install/demo-0.3-py2.4.egg',
'/sample-install/demoneeded-1.1-py2.4.egg',
'/foo',
+ ...
]
<BLANKLINE>
import eggrecipedemo
@@ -737,6 +843,68 @@
if __name__ == '__main__':
eggrecipedemo.main()
+Ordering paths
+--------------
+
+We have already seen that we have a precise definition for a loose term:
+"site-packages". Paths outside of Python's standard library--or more
+precisely, those that are not included when Python is started with the -S
+argument--are loosely referred to as "site-packages" here.
+
+When generating scripts, paths that come from the site-packages are ordered
+after the other specific dependencies generated from the working set. This
+is so that directories such as "site-packages" that can contain multiple
+dependencies come after the more specific distributions found by setuptools,
+reducing the chance of the distributions being masked by the system folders.
+
+This is controlled by the ``get_path`` function, which is available for
+other script recipes to use.
+
+As a demonstration, we will have create a working set that has dependencies
+on "bigdemo" and "other". In our first case, we will use a clean Python
+without any of these dependencies installed.
+
+ >>> dest1 = tmpdir('path-install-1')
+ >>> ws1 = zc.buildout.easy_install.install(
+ ... ['other', 'bigdemo'], dest1,
+ ... links=[link_server], index=link_server+'index/')
+ >>> path1 = zc.buildout.easy_install.get_path(ws1, sys.executable)
+
+ >>> import pprint
+ >>> pprint.pprint(path1) # doctest: +ELLIPSIS
+ ['.../path-install-1/other-1.0-py...egg',
+ '.../path-install-1/bigdemo-0.1-py...egg',
+ '.../path-install-1/demo-0.3-py...egg',
+ '.../path-install-1/demoneeded-1.1-py...egg',
+ ...]
+
+We will now compare the results using a Python that has bigdemo's indirect
+dependency available, "demoneeded," and "other," but not "demo" or "bigdemo".
+
+ >>> dest2 = tmpdir('path-install-2')
+ >>> ws2 = zc.buildout.easy_install.install(
+ ... ['other', 'bigdemo'], dest2,
+ ... links=[link_server], index=link_server+'index/',
+ ... executable=primed_executable)
+ >>> path2 = zc.buildout.easy_install.get_path(ws2, primed_executable)
+ >>> pprint.pprint(path2) # doctest: +ELLIPSIS
+ ['.../path-install-2/bigdemo-0.1-py...egg',
+ '.../path-install-2/demo-0.3-py...egg',
+ '.../executable/eggs/other-1.0-py...egg',
+ '.../executable/eggs/demoneeded-1.1-py...egg',
+ '.../executable/eggs/setuptools-0.6c9-py...egg',
+ '.../executable/bin',
+ ...]
+ >>> zc.buildout.easy_install.clear_index_cache() # clean up
+
+Notice that the paths from the executable come after the ones for this
+buildout. This is most evident in the change of order for the "other" egg.
+
+In fact, this ordering is not important in this example, because the
+executable's paths all are individual packages; but if a path were a directory
+that shared many packages, like a classic "site-packages" directory, its shared
+packages would not mask those selected by the buildout.
+
Providing script arguments
--------------------------
@@ -748,12 +916,14 @@
... ['demo'], ws, sys.executable, bin, dict(demo='run'),
... arguments='1, 2')
- >>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE
+ >>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
#!/usr/local/bin/python2.4
- import sys
- sys.path[0:0] = [
+ import sys, os
+ pythonpath = filter(None, os.environ.get('PYTHONPATH', '').split(':'))
+ sys.path[:] = pythonpath + [
'/sample-install/demo-0.3-py2.4.egg',
'/sample-install/demoneeded-1.1-py2.4.egg',
+ ...
]
<BLANKLINE>
import eggrecipedemo
@@ -771,13 +941,15 @@
... arguments='1, 2',
... initialization='import os\nos.chdir("foo")')
- >>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE
+ >>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
#!/usr/local/bin/python2.4
- import sys
- sys.path[0:0] = [
- '/sample-install/demo-0.3-py2.4.egg',
- '/sample-install/demoneeded-1.1-py2.4.egg',
- ]
+ import sys, os
+ pythonpath = filter(None, os.environ.get('PYTHONPATH', '').split(':'))
+ sys.path[:] = pythonpath + [
+ '/sample-install/demo-0.3-py2.4.egg',
+ '/sample-install/demoneeded-1.1-py2.4.egg',
+ ...
+ ]
<BLANKLINE>
import os
os.chdir("foo")
@@ -811,7 +983,7 @@
... interpreter='py',
... relative_paths=bo)
- >>> cat(bo, 'bin', 'run')
+ >>> cat(bo, 'bin', 'run') # doctest: +ELLIPSIS
#!/usr/local/bin/python2.4
<BLANKLINE>
import os
@@ -820,13 +992,15 @@
base = os.path.dirname(os.path.abspath(__file__))
base = os.path.dirname(base)
<BLANKLINE>
- import sys
- sys.path[0:0] = [
- join(base, 'eggs/demo-0.3-pyN.N.egg'),
- join(base, 'eggs/demoneeded-1.1-pyN.N.egg'),
- '/ba',
- join(base, 'bar'),
- ]
+ import sys, os
+ pythonpath = filter(None, os.environ.get('PYTHONPATH', '').split(':'))
+ sys.path[:] = pythonpath + [
+ join(base, 'eggs/demo-0.3-pyN.N.egg'),
+ join(base, 'eggs/demoneeded-1.1-pyN.N.egg'),
+ '/ba',
+ join(base, 'bar'),
+ ...
+ ]
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
@@ -843,44 +1017,75 @@
We specified an interpreter and its paths are adjusted too:
- >>> cat(bo, 'bin', 'py')
+ >>> cat(bo, 'bin', 'py') # doctest: +ELLIPSIS
#!/usr/local/bin/python2.4
+ globs = globals().copy() # get a clean copy early
<BLANKLINE>
+ <BLANKLINE>
import os
<BLANKLINE>
join = os.path.join
base = os.path.dirname(os.path.abspath(__file__))
base = os.path.dirname(base)
<BLANKLINE>
- import sys
+ import sys, os
<BLANKLINE>
- sys.path[0:0] = [
- join(base, 'eggs/demo-0.3-pyN.N.egg'),
- join(base, 'eggs/demoneeded-1.1-pyN.N.egg'),
- '/ba',
- join(base, 'bar'),
- ]
+ _set_path = _interactive = True
+ _force_interactive = False
<BLANKLINE>
- _interactive = True
- if len(sys.argv) > 1:
- import getopt
- _options, _args = getopt.getopt(sys.argv[1:], 'ic:')
+ _command = None
+ _args = sys.argv[1:]
+ <BLANKLINE>
+ while _args:
+ if _args[0].startswith('-'):
+ _arg = _args.pop(0)
+ for _ix, _opt in enumerate(_arg[1:]):
+ if _opt == 'i':
+ _force_interactive = True
+ elif _opt == 'c':
+ _interactive = False
+ _command = _args.pop(0) # Argument expected for the -c option
+ _args.insert(0, '-c')
+ break
+ elif _opt == 'S':
+ # We'll approximate this. It is mostly convenient for tests.
+ _set_path = False
+ elif _opt == 'V':
+ print 'Python ' + sys.version.split()[0]
+ _interactive = False
+ break
+ else:
+ continue
+ break
+ else:
+ break
+ <BLANKLINE>
+ if _set_path:
+ sys.path[:] = [
+ join(base, 'eggs/demo-0.3-pyN.N.egg'),
+ join(base, 'eggs/demoneeded-1.1-pyN.N.egg'),
+ '/ba',
+ join(base, 'bar'),
+ ...,
+ ]
+ pythonpath = filter(None, os.environ.get('PYTHONPATH', '').split(':'))
+ sys.path[0:0] = pythonpath
+ sys.path.insert(0, '.')
+ <BLANKLINE>
+ sys.argv[:] = _args
+ <BLANKLINE>
+ if _command:
+ exec _command
+ elif _args:
_interactive = False
- for (_opt, _val) in _options:
- if _opt == '-i':
- _interactive = True
- elif _opt == '-c':
- exec _val
+ globs['__file__'] = sys.argv[0]
+ execfile(sys.argv[0], globs)
<BLANKLINE>
- if _args:
- sys.argv[:] = _args
- execfile(sys.argv[0])
- <BLANKLINE>
- if _interactive:
+ if _interactive or _force_interactive:
import code
- code.interact(banner="", local=globals())
+ del globs['__file__']
+ code.interact(banner="", local=globs)
-
Handling custom build options for extensions provided in source distributions
-----------------------------------------------------------------------------
Modified: zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/setup.txt
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/setup.txt 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/setup.txt 2009-08-11 19:22:31 UTC (rev 102683)
@@ -30,7 +30,7 @@
>>> print system(buildout+' setup'),
Error: The setup command requires the path to a setup script or
- directory containing a setup script, and it's arguments.
+ directory containing a setup script, and its arguments.
Oops, we forgot to give the name of the setup script:
Modified: zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/testing.py
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/testing.py 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/testing.py 2009-08-11 19:22:31 UTC (rev 102683)
@@ -116,12 +116,19 @@
args = [zc.buildout.easy_install._safe_arg(arg)
for arg in args]
args.insert(0, '-q')
- args.append(dict(os.environ, PYTHONPATH=setuptools_location))
+ env = dict(os.environ)
+ if executable == sys.executable:
+ env['PYTHONPATH'] = setuptools_location
+ # else pass an executable that has setuptools! See testselectingpython.py.
+ args.append(env)
+
here = os.getcwd()
try:
os.chdir(d)
- os.spawnle(os.P_WAIT, executable, zc.buildout.easy_install._safe_arg (executable), setup, *args)
+ os.spawnle(os.P_WAIT, executable,
+ zc.buildout.easy_install._safe_arg(executable),
+ setup, *args)
if os.path.exists('build'):
rmtree('build')
finally:
@@ -133,6 +140,11 @@
def bdist_egg(setup, executable, dest):
_runsetup(setup, executable, 'bdist_egg', '-d', dest)
+def sys_install(setup, dest):
+ _runsetup(setup, sys.executable, 'install', '--home', dest,
+ '--single-version-externally-managed',
+ '--record', os.path.join(dest, 'added'))
+
def find_python(version):
e = os.environ.get('PYTHON%s' % version)
if e is not None:
@@ -266,13 +278,15 @@
# trick bootstrap into putting the buildout develop egg
# in the eggs dir.
('buildout', 'develop-eggs-directory', 'eggs'),
+ # we need to have setuptools around.
+ ('buildout', 'include-site-packages-for-buildout', 'true'),
]
).bootstrap([])
# Create the develop-eggs dir, which didn't get created the usual
- # way due to thr trick above:
+ # way due to the trick above:
os.mkdir('develop-eggs')
def start_server(path):
Modified: zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/tests.py
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/tests.py 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/tests.py 2009-08-11 19:22:31 UTC (rev 102683)
@@ -11,12 +11,12 @@
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
-"""XXX short summary goes here.
+"""Tests, test set up, and hook-up for stand-alone doctests.
$Id$
"""
-import os, re, shutil, sys, tempfile, unittest, zipfile
+import os, re, shutil, sys, tempfile, textwrap, unittest, zipfile
from zope.testing import doctest, renormalizing
import pkg_resources
import zc.buildout.testing, zc.buildout.easy_install
@@ -258,18 +258,17 @@
def show_who_requires_when_there_is_a_conflict():
"""
- It's a pain when we require eggs that have requirements that are
- incompatible. We want the error we get to tell us what is missing.
+It's a pain when we require eggs that have requirements that are incompatible.
+We want the error we get to tell us what is missing.
- Let's make a few develop distros, some of which have incompatible
- requirements.
+Let's make a few develop distros, some of which have incompatible requirements.
>>> make_dist_that_requires(sample_buildout, 'sampley',
... ['demoneeded ==1.0'])
>>> make_dist_that_requires(sample_buildout, 'samplez',
... ['demoneeded ==1.1'])
- Now, let's create a buildout that requires y and z:
+Now, let's create a buildout that requires y and z:
>>> write('buildout.cfg',
... '''
@@ -315,7 +314,7 @@
... samplez
... ''' % globals())
-If we use the verbose switch, we can see where requirements are comning from:
+If we use the verbose switch, we can see where requirements are coming from:
>>> print system(buildout+' -v'), # doctest: +ELLIPSIS
Installing 'zc.buildout', 'setuptools'.
@@ -348,12 +347,10 @@
def show_who_requires_missing_distributions():
"""
+When working with a lot of eggs, which require eggs recursively, it can be hard
+to tell why we're requireing things we can't find. Fortunately, buildout will
+tell us who's asking for something that we can't find.
- When working with a lot of eggs, which require eggs recursively,
- it can be hard to tell why we're requireing things we can't find.
- Fortunately, buildout will tell us who's asking for something that
- we can't find.
-
>>> make_dist_that_requires(sample_buildout, 'sampley', ['demoneeded'])
>>> make_dist_that_requires(sample_buildout, 'samplea', ['sampleb'])
>>> make_dist_that_requires(sample_buildout, 'sampleb',
@@ -382,11 +379,75 @@
Error: Couldn't find a distribution for 'demoneeded'.
"""
+def show_eggs_from_site_packages():
+ """
+Sometimes you want to know what eggs are coming from site-packages. This
+might be for a diagnostic, or so that you can get a starting value for the
+allowed-eggs-from-site-packages option. The -v flag will also include this
+information.
+Our "primed_executable" has the "demoneeded," "other," and "setuptools"
+packages available. We'll ask for "other" and "bigdemo".
+
+Here's our set up.
+
+ >>> primed_executable = get_executable_with_site_packages()
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... parts = eggs
+ ... prefer-final = true
+ ... find-links = %(link_server)s
+ ...
+ ... [primed_python]
+ ... executable = %(primed_executable)s
+ ...
+ ... [eggs]
+ ... recipe = zc.recipe.egg:eggs
+ ... python = primed_python
+ ... eggs = other
+ ... bigdemo
+ ... ''' % globals())
+
+Now here is the output. The lines that begin with "Egg from site-packages:"
+indicate the eggs from site-packages that have been selected. You'll see
+we have two: other 1.0 and demoneeded 1.1.
+
+ >>> print system(primed_executable+" "+buildout+" -v")
+ Installing 'zc.buildout', 'setuptools'.
+ We have a develop egg: zc.buildout V
+ We have the best distribution that satisfies 'setuptools'.
+ Picked: setuptools = V
+ Installing 'zc.recipe.egg'.
+ We have a develop egg: zc.recipe.egg V
+ Installing eggs.
+ Installing 'other', 'bigdemo'.
+ We have the best distribution that satisfies 'other'.
+ Picked: other = 1.0
+ We have no distributions for bigdemo that satisfies 'bigdemo'.
+ Getting distribution for 'bigdemo'.
+ Got bigdemo 0.1.
+ Picked: bigdemo = 0.1
+ Egg from site-packages: other 1.0
+ Getting required 'demo'
+ required by bigdemo 0.1.
+ We have no distributions for demo that satisfies 'demo'.
+ Getting distribution for 'demo'.
+ Got demo 0.3.
+ Picked: demo = 0.3
+ Getting required 'demoneeded'
+ required by demo 0.3.
+ We have the best distribution that satisfies 'demoneeded'.
+ Picked: demoneeded = 1.1
+ Egg from site-packages: demoneeded 1.1
+ <BLANKLINE>
+
+ """
+
def test_comparing_saved_options_with_funny_characters():
"""
- If an option has newlines, extra/odd spaces or a %, we need to make
- sure the comparison with the saved value works correctly.
+If an option has newlines, extra/odd spaces or a %, we need to make sure the
+comparison with the saved value works correctly.
>>> mkdir(sample_buildout, 'recipes')
>>> write(sample_buildout, 'recipes', 'debug.py',
@@ -604,6 +665,7 @@
>>> os.chdir(d)
>>> print system(os.path.join(sample_buildout, 'bin', 'buildout')
+ ... + ' buildout:include-site-packages-for-buildout=true'
... + ' bootstrap'),
Creating directory '/sample-bootstrap/bin'.
Creating directory '/sample-bootstrap/parts'.
@@ -631,6 +693,7 @@
>>> os.chdir(d)
>>> print system(os.path.join(sample_buildout, 'bin', 'buildout')
+ ... + ' buildout:include-site-packages-for-buildout=true'
... + ' bootstrap'),
Creating directory '/sample-bootstrap/bin'.
Creating directory '/sample-bootstrap/parts'.
@@ -1768,6 +1831,93 @@
1 2
"""
+def versions_section_ignored_for_dependency_in_favor_of_site_packages():
+ r"""
+This is a test for a bugfix.
+
+The error showed itself when at least two dependencies were in a shared
+location like site-packages, and the first one met the "versions" setting. The
+first dependency would be added, but subsequent dependencies from the same
+location (e.g., site-packages) would use the version of the package found in
+the shared location, ignoring the version setting.
+
+We begin with a Python that has demoneeded version 1.1 installed and a demo
+version 0.3, all in a site-packages-like shared directory. We need to create
+this.
+
+ >>> executable_buildout = tmpdir('executable_buildout')
+ >>> old_wd = os.getcwd()
+ >>> os.chdir(executable_buildout)
+ >>> import textwrap
+ >>> write('buildout.cfg', textwrap.dedent(
+ ... '''
+ ... [buildout]
+ ... parts = interpreter
+ ...
+ ... [interpreter]
+ ... recipe = zc.recipe.egg
+ ... scripts = py
+ ... interpreter = py
+ ... extra-paths = %(site-packages)s
+ ... include-site-packages = false
+ ... eggs = setuptools
+ ... ''') % {'site-packages': join(site_packages, 'lib', 'python')})
+ >>> zc.buildout.buildout.Buildout(
+ ... 'buildout.cfg',
+ ... [('buildout', 'log-level', 'WARNING'),
+ ... # trick bootstrap into putting the buildout develop egg
+ ... # in the eggs dir.
+ ... ('buildout', 'develop-eggs-directory', 'eggs'),
+ ... # we need to have setuptools around.
+ ... ('buildout', 'include-site-packages-for-buildout', 'true'),
+ ... ]
+ ... ).bootstrap([])
+ >>> os.mkdir('develop-eggs')
+ >>> zc.buildout.testing.install_develop(
+ ... 'zc.recipe.egg',
+ ... os.path.join(executable_buildout, 'develop-eggs'))
+ >>> print system(
+ ... os.path.join(executable_buildout, 'bin', 'buildout'))
+ Installing interpreter.
+ Generated interpreter '/executable_buildout/bin/py'.
+ <BLANKLINE>
+ >>> os.chdir(old_wd)
+ >>> primed_executable = os.path.join(executable_buildout, 'bin', 'py')
+
+``eggrecipedemo.main()`` shows the number after the dot (that is, ``X`` in
+``1.X``), for the demo package and the demoneeded package, so this demonstrates
+that our Python does in fact have demo version 0.3 and demoneeded version 1.1.
+
+ >>> print system(primed_executable+" -c "+
+ ... "'import eggrecipedemo; eggrecipedemo.main()'")
+ 3 1
+ <BLANKLINE>
+
+Now we will install bigdemo, specifying different versions of demo
+and demoneeded in a versions section. Before the bugfix, the demo version
+would be honored, but not the demoneeded.
+
+Now here's a setup that would expose the bug, using the
+zc.buildout.easy_install API.
+
+ >>> example_dest = tmpdir('example_dest')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['bigdemo'], example_dest, links=[sample_eggs],
+ ... executable=primed_executable,
+ ... index=None, include_site_packages=True,
+ ... versions={'demoneeded': '1.2c1', 'demo': '0.3'})
+ >>> for dist in workingset:
+ ... print dist
+ bigdemo 0.1
+ demo 0.3
+ demoneeded 1.2c1
+
+Before the bugfix, the demoneeded distribution was not included in the working
+set, and the demoneeded in site-packages (of the wrong number) would have been
+used.
+
+ """
+
if sys.version_info > (2, 4):
def test_exit_codes():
"""
@@ -2156,7 +2306,7 @@
"""
This test tests several permutations:
-Using different version numbers to work around zip impporter cache problems. :(
+Using different version numbers to work around zip importer cache problems. :(
- With prefer final:
@@ -2338,6 +2488,508 @@
"""
+def isolated_include_site_packages():
+ """
+
+This is an isolated test of the include_site_packages functionality, passing
+the argument directly to install, overriding a default.
+
+Our "primed_executable" has the "demoneeded," "other," and "setuptools"
+packages available. We'll simply be asking for "other" here.
+
+ >>> primed_executable = get_executable_with_site_packages()
+ >>> zc.buildout.easy_install.include_site_packages(False)
+ True
+
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['other'], example_dest, links=[], executable=primed_executable,
+ ... index=None, include_site_packages=True)
+ >>> [dist.project_name for dist in workingset]
+ ['other']
+
+That worked fine. Let's try again with site packages not allowed (and
+reversing the default).
+
+ >>> zc.buildout.easy_install.include_site_packages(True)
+ False
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['other'], example_dest, links=[], executable=primed_executable,
+ ... index=None, include_site_packages=False)
+ Traceback (most recent call last):
+ ...
+ MissingDistribution: Couldn't find a distribution for 'other'.
+
+That's a failure, as expected.
+
+Now we explore an important edge case.
+
+Some system Pythons include setuptools (and other Python packages) in their
+site-packages (or equivalent) using a .egg-info directory. The pkg_resources
+module (from setuptools) considers a package installed using .egg-info to be a
+develop egg.
+
+zc.buildout.buildout.Buildout.bootstrap will make setuptools and zc.buildout
+available to the buildout via the eggs directory, for normal eggs; or the
+develop-eggs directory, for develop-eggs.
+
+If setuptools or zc.buildout is found in site-packages and considered by
+pkg_resources to be a develop egg, then the bootstrap code will use a .egg-link
+in the local develop-eggs, pointing to site-packages, in its entirety. Because
+develop-eggs must always be available for searching for distributions, this
+indirectly brings site-packages back into the search path for distributions.
+
+Because of this, we have to take special care that we still exclude
+site-packages even in this case. See the comments about site packages in the
+Installer._satisfied and Installer._obtain methods for the implementation
+(as of this writing).
+
+In this demonstration, we insert a link to the "other" distribution in our
+develop-eggs, which would bring the package back in, except for the special
+care we have taken to exclude it.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> mkdir(example_dest, 'develop-eggs')
+ >>> stdlib, site_packages = (
+ ... zc.buildout.easy_install._get_system_packages(primed_executable))
+ >>> path_to_other = [p for p in site_packages if 'other' in p][0]
+ >>> write(example_dest, 'develop-eggs', 'other.egg-link', path_to_other)
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['other'], example_dest, links=[],
+ ... path=[join(example_dest, 'develop-eggs')],
+ ... executable=primed_executable,
+ ... index=None, include_site_packages=False)
+ Traceback (most recent call last):
+ ...
+ MissingDistribution: Couldn't find a distribution for 'other'.
+
+The MissingDistribution error shows that buildout correctly excluded the
+"site-packages" source even though it was indirectly included in the path
+via a .egg-link file.
+
+ """
+
+def buildout_include_site_packages_option():
+ """
+The include-site-packages buildout option can be used to override the default
+behavior of using site packages.
+
+The default is include-site-packages = true. As a demonstration, notice we do
+not set find-links, but the eggs are still found because they are in the
+executable's path.
+
+Our "primed_executable" has the "demoneeded," "other," and "setuptools"
+packages available. We'll simply be asking for "other" here.
+
+ >>> primed_executable = get_executable_with_site_packages()
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... parts = eggs
+ ... find-links =
+ ...
+ ... [primed_python]
+ ... executable = %(primed_executable)s
+ ...
+ ... [eggs]
+ ... recipe = zc.recipe.egg:eggs
+ ... python = primed_python
+ ... eggs = other
+ ... ''' % globals())
+
+ >>> print system(primed_executable+" "+buildout)
+ Installing eggs.
+ <BLANKLINE>
+
+However, if we set include-site-packages to false, we get an error, because
+the packages are not available in any links, and they are not allowed to be
+obtained from the executable's site packages.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... parts = eggs
+ ... find-links =
+ ... include-site-packages = false
+ ...
+ ... [primed_python]
+ ... executable = %(primed_executable)s
+ ...
+ ... [eggs]
+ ... recipe = zc.recipe.egg:eggs
+ ... eggs = other
+ ... ''' % globals())
+ >>> print system(primed_executable+" "+buildout)
+ Uninstalling eggs.
+ Installing eggs.
+ Couldn't find index page for 'other' (maybe misspelled?)
+ Getting distribution for 'other'.
+ While:
+ Installing eggs.
+ Getting distribution for 'other'.
+ Error: Couldn't find a distribution for 'other'.
+ <BLANKLINE>
+
+We get an error if we specify anything but true or false:
+
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... parts = eggs
+ ... find-links = %(link_server)s
+ ... include-site-packages = no
+ ...
+ ... [eggs]
+ ... recipe = zc.recipe.egg:eggs
+ ... eggs = other
+ ... ''' % globals())
+
+ >>> print system(primed_executable+" "+buildout)
+ While:
+ Initializing.
+ Error: Invalid value for include-site-packages option: no
+ <BLANKLINE>
+
+ """
+
+def include_site_packages_with_buildout():
+ """
+When buildout gets a recipe egg (as opposed to runs a recipe), it starts with
+the current Python working set--the one that the bin/buildout script uses
+itself. If this working set includes site-packages, and site-packages includes
+an egg for package that the recipe needs, and the recipe specifies a newer
+version of that package, this can generate a version conflict.
+
+One solution to this is to not use site-packages
+('include-site-packages = false'). However, if you want your scripts to use
+site-packages, then you have to specify 'include-site-packages = true' for
+all of them.
+
+To make this use case easier to handle, you can instead specify
+``include-site-packages-with-buildout = false``, which indicates that
+the bin/buildout should *not* use site-packages; and
+``include-site-packages = true``, which indicates that the rest of the scripts
+should use site-packages.
+
+This is the default configuration.
+
+ >>> from zc.buildout.buildout import Buildout
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... find-links = %(link_server)s
+ ... ''' % globals())
+ >>> buildout = Buildout('buildout.cfg', ())
+ >>> buildout['buildout']['include-site-packages-for-buildout']
+ 'false'
+ >>> buildout['buildout']['include-site-packages']
+ 'true'
+ >>> buildout.include_site_packages_for_buildout
+ False
+ >>> zc.buildout.easy_install.include_site_packages()
+ True
+
+This means that, when the buildout script is created by buildout, it explicitly
+specifies that site-packages should not be used. We'll monkeypatch the
+zc.buildout.easy_install install function so we can see this happens. (We test
+that this argument actually does what we want in other tests.)
+
+ >>> original_install = zc.buildout.easy_install.install
+ >>> def install(*args, **kwargs):
+ ... print 'include_site_packages =', kwargs['include_site_packages']
+ ... return original_install(*args, **kwargs)
+ ...
+ >>> zc.buildout.easy_install.install = install
+ >>> buildout.bootstrap(()) # doctest: +ELLIPSIS
+ include_site_packages = False...
+
+Now we'll do the reverse settings to show that the value will be honored in
+that case.
+
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... include-site-packages-for-buildout = true
+ ... include-site-packages = false
+ ... find-links = %(link_server)s
+ ... ''' % globals())
+ >>> buildout = Buildout('buildout.cfg', ())
+ >>> buildout['buildout']['include-site-packages-for-buildout']
+ 'true'
+ >>> buildout['buildout']['include-site-packages']
+ 'false'
+ >>> buildout.include_site_packages_for_buildout
+ True
+ >>> zc.buildout.easy_install.include_site_packages()
+ False
+ >>> buildout.bootstrap(()) # doctest: +ELLIPSIS
+ include_site_packages = True...
+ >>> zc.buildout.easy_install.install = original_install
+
+Now we'll show that a value that is not 'true' or 'false' generates an error.
+
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... include-site-packages-for-buildout = shazbot
+ ... ''')
+ >>> buildout = Buildout('buildout.cfg', ())
+ Traceback (most recent call last):
+ ...
+ UserError: Invalid value for include-site-packages-for-buildout option: shazbot
+
+ """
+
+def allowed_eggs_from_site_packages():
+ """
+Sometimes you need or want to control what eggs from site-packages are used.
+The allowed-eggs-from-site-packages option allows you to specify a whitelist of
+project names that may be included from site-packages. You can use globs to
+specify the value. It defaults to a single value of '*', indicating that any
+package may come from site-packages.
+
+This option interacts with include-site-packages in the following ways.
+
+If include-site-packages is true, then allowed-eggs-from-site-packages filters
+what eggs from site-packages may be chosen. If allowed-eggs-from-site-packages
+is an empty list, then no eggs from site-packages are chosen, but site-packages
+will still be included at the end of path lists.
+
+If include-site-packages is false, allowed-eggs-from-site-packages is
+irrelevant.
+
+This test shows the interaction with the zc.buildout.easy_install API. Another
+test below (allow_site_package_eggs_option) shows using it with a buildout.cfg.
+
+Our "primed_executable" has the "demoneeded," "other," and "setuptools"
+packages available. We'll simply be asking for "other" here.
+
+ >>> primed_executable = get_executable_with_site_packages()
+
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['other'], example_dest, links=[], executable=primed_executable,
+ ... index=None,
+ ... allowed_eggs_from_site_packages=['demoneeded', 'other'])
+ >>> [dist.project_name for dist in workingset]
+ ['other']
+
+That worked fine. It would work fine for a glob too.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['other'], example_dest, links=[], executable=primed_executable,
+ ... index=None,
+ ... allowed_eggs_from_site_packages=['demoneeded', '?th*'])
+ >>> [dist.project_name for dist in workingset]
+ ['other']
+
+But now let's try again with 'other' not allowed.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['other'], example_dest, links=[], executable=primed_executable,
+ ... index=None,
+ ... allowed_eggs_from_site_packages=['demoneeded'])
+ Traceback (most recent call last):
+ ...
+ MissingDistribution: Couldn't find a distribution for 'other'.
+
+Here's the same, but with an empty list.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['other'], example_dest, links=[], executable=primed_executable,
+ ... index=None,
+ ... allowed_eggs_from_site_packages=[])
+ Traceback (most recent call last):
+ ...
+ MissingDistribution: Couldn't find a distribution for 'other'.
+
+Of course, this doesn't stop us from getting a package from elsewhere. Here,
+we add a link server.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['other'], example_dest, executable=primed_executable,
+ ... links=[link_server], index=link_server+'index/',
+ ... allowed_eggs_from_site_packages=['demoneeded'])
+ >>> [dist.project_name for dist in workingset]
+ ['other']
+ >>> [dist.location for dist in workingset]
+ ['/site-packages-example-install/other-1.0-py2.6.egg']
+
+Finally, here's an example of an interaction we described above: we say that it
+is OK to allow the "other" egg to come from site-packages, but we don't
+include-site-packages.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['other'], example_dest, links=[], executable=primed_executable,
+ ... index=None, include_site_packages=False,
+ ... allowed_eggs_from_site_packages=['other'])
+ Traceback (most recent call last):
+ ...
+ MissingDistribution: Couldn't find a distribution for 'other'.
+
+ """
+
+def allowed_eggs_from_site_packages_option():
+ """
+As introduced in the previous test, the allowed-eggs-from-site-packages option
+allows you to specify a whitelist of project names that may be included from
+site-packages.
+
+This test shows the option being used in a buildout. We try to limit these
+tests to those that test additional parts of the code beyond those tested in
+the test above.
+
+The buildout defaults to a whitelist of ('*',), or any project name. The
+buildout configuration option defaults are managed separately from the
+zc.buildout.easy_install API defaults, so we show this here.
+
+ >>> from zc.buildout.buildout import Buildout
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... ''')
+ >>> buildout_instance = Buildout('buildout.cfg', ())
+ >>> buildout_instance['buildout']['allowed-eggs-from-site-packages']
+ '*'
+ >>> zc.buildout.easy_install.allowed_eggs_from_site_packages()
+ ('*',)
+
+In the test below, our "primed_executable" has the "demoneeded," "other," and "setuptools"
+packages available. We'll simply be asking for "other" here. The default
+value of '*' will allow it. This confirms behaviorally what we saw above.
+
+ >>> primed_executable = get_executable_with_site_packages()
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... parts = eggs
+ ... find-links =
+ ...
+ ... [primed_python]
+ ... executable = %(primed_executable)s
+ ...
+ ... [eggs]
+ ... recipe = zc.recipe.egg:eggs
+ ... python = primed_python
+ ... eggs = other
+ ... ''' % globals())
+
+ >>> print system(primed_executable+" "+buildout)
+ Installing eggs.
+ <BLANKLINE>
+
+Here we explicitly use a "*" for the same result. This also shows that we
+correctly parse a single-line value.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... parts = eggs
+ ... find-links =
+ ... allowed-eggs-from-site-packages = *
+ ...
+ ... [primed_python]
+ ... executable = %(primed_executable)s
+ ...
+ ... [eggs]
+ ... recipe = zc.recipe.egg:eggs
+ ... python = primed_python
+ ... eggs = other
+ ... ''' % globals())
+
+ >>> print system(primed_executable+" "+buildout)
+ Updating eggs.
+ <BLANKLINE>
+
+Specifying the egg exactly will work as well. This shows we correctly
+parse a multi-line value.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... parts = eggs
+ ... find-links =
+ ... allowed-eggs-from-site-packages = demoneeded
+ ... other
+ ...
+ ... [primed_python]
+ ... executable = %(primed_executable)s
+ ...
+ ... [eggs]
+ ... recipe = zc.recipe.egg:eggs
+ ... python = primed_python
+ ... eggs = other
+ ... ''' % globals())
+
+ >>> print system(primed_executable+" "+buildout)
+ Updating eggs.
+ <BLANKLINE>
+
+It will also work if we use a glob ("*" or "?"). (We won't show that here
+because we already tested it in the previous doctest.)
+
+However, if we do not include "other" in the "allowed-eggs-from-site-packages"
+key, we get an error, because the packages are not available in any links, and
+they are not allowed to come from the executable's site packages. (We won't
+show that here because we already tested it in the previous doctest.)
+
+Finally, here's a test with an empty value. It shows that we parse an empty
+value correctly, and verifies that we really are controlling what eggs are
+allowed, because we see that we were unable to get "other".
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... parts = eggs
+ ... find-links =
+ ... allowed-eggs-from-site-packages =
+ ...
+ ... [primed_python]
+ ... executable = %(primed_executable)s
+ ...
+ ... [eggs]
+ ... recipe = zc.recipe.egg:eggs
+ ... eggs = other
+ ... ''' % globals())
+ >>> print system(primed_executable+" "+buildout)
+ Uninstalling eggs.
+ Installing eggs.
+ Couldn't find index page for 'other' (maybe misspelled?)
+ Getting distribution for 'other'.
+ While:
+ Installing eggs.
+ Getting distribution for 'other'.
+ Error: Couldn't find a distribution for 'other'.
+ <BLANKLINE>
+
+ """
+
def develop_with_modules():
"""
Distribution setup scripts can import modules in the distribution directory:
@@ -2531,8 +3183,17 @@
######################################################################
def create_sample_eggs(test, executable=sys.executable):
+ """Creates sample eggs, source distributions, and a faux site-packages."
+
+ Unlike the faux site-packages created by
+ ``get_executable_with_site_packages``, this one has packages installed the
+ way distributions often install eggs in system Pythons (via
+ zc.buildout.testing.sys_install).
+ """
write = test.globs['write']
dest = test.globs['sample_eggs']
+ site_packages = test.globs['tmpdir']('site_packages')
+ test.globs['site_packages'] = site_packages
tmp = tempfile.mkdtemp()
try:
write(tmp, 'README.txt', '')
@@ -2549,6 +3210,8 @@
% (i, c1)
)
zc.buildout.testing.sdist(tmp, dest)
+ if i==1:
+ zc.buildout.testing.sys_install(tmp, site_packages)
write(
tmp, 'setup.py',
@@ -2578,6 +3241,8 @@
" zip_safe=True, version='0.%s%s')\n" % (i, c1)
)
zc.buildout.testing.bdist_egg(tmp, executable, dest)
+ if i==3:
+ zc.buildout.testing.sys_install(tmp, site_packages)
write(tmp, 'eggrecipebigdemo.py', 'import eggrecipedemo')
write(
@@ -2652,7 +3317,52 @@
test.globs['sample_eggs'])
test.globs['update_extdemo'] = lambda : add_source_dist(test, 1.5)
zc.buildout.testing.install_develop('zc.recipe.egg', test)
+ # Most tests don't need this set up, and it takes some time, so we just
+ # make it available as a convenience.
+ def get_executable_with_site_packages(requirements=None):
+ executable_buildout = test.globs['tmpdir']('executable')
+ old_wd = os.getcwd()
+ os.chdir(executable_buildout)
+ if requirements is None:
+ requirements = ['demoneeded', 'setuptools', 'other']
+ elif len([req for req in requirements
+ if req.startswith('setuptools')]) == 0:
+ requirements.append('setuptools') # you always need that.
+ requirements = '\n '.join(requirements)
+ test.globs['write']('buildout.cfg', textwrap.dedent(
+ '''
+ [buildout]
+ parts = interpreter
+ find-links = %(link_server)s
+ prefer-final = true
+ [interpreter]
+ recipe = zc.recipe.egg
+ interpreter = py
+ eggs = %(requirements)s
+ ''') % {'requirements': requirements,
+ 'link_server': test.globs['link_server']})
+ zc.buildout.buildout.Buildout(
+ 'buildout.cfg',
+ [('buildout', 'log-level', 'WARNING'),
+ # trick bootstrap into putting the buildout develop egg
+ # in the eggs dir.
+ ('buildout', 'develop-eggs-directory', 'eggs'),
+ # we need to have setuptools around.
+ ('buildout', 'include-site-packages-for-buildout', 'true'),
+ ]
+ ).bootstrap([])
+ os.mkdir('develop-eggs')
+ zc.buildout.testing.install_develop(
+ 'zc.recipe.egg',
+ os.path.join(executable_buildout, 'develop-eggs'))
+ test.globs['system'](
+ os.path.join(executable_buildout, 'bin', 'buildout'))
+ os.chdir(old_wd)
+ return os.path.join(executable_buildout, 'bin', 'py')
+ test.globs['get_executable_with_site_packages'] = (
+ get_executable_with_site_packages)
+
egg_parse = re.compile('([0-9a-zA-Z_.]+)-([0-9a-zA-Z_.]+)-py(\d[.]\d).egg$'
).match
def makeNewRelease(project, ws, dest):
@@ -2851,7 +3561,10 @@
'-q develop -mxN -d "/sample-buildout/develop-eggs'),
'-q develop -mxN -d /sample-buildout/develop-eggs'
),
- (re.compile(r'^[*]...'), '...'),
+ (re.compile(r'^[*]...'), '...'),
+ # for bug_92891_bootstrap_crashes_with_egg_recipe_in_buildout_section
+ (re.compile(r"Unused options for buildout: 'eggs' 'scripts'\."),
+ "Unused options for buildout: 'scripts' 'eggs'."),
]),
),
zc.buildout.testselectingpython.test_suite(),
Modified: zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/testselectingpython.py
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/testselectingpython.py 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/testselectingpython.py 2009-08-11 19:22:31 UTC (rev 102683)
@@ -11,8 +11,9 @@
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
-import os, re, sys, unittest
+import os, re, subprocess, sys, textwrap, unittest
from zope.testing import doctest, renormalizing
+import zc.buildout.easy_install
import zc.buildout.tests
import zc.buildout.testing
@@ -25,13 +26,14 @@
test_selecting_python_via_easy_install=
"""\
-We can specify an specific Python executable.
+We can specify a specific Python executable.
>>> dest = tmpdir('sample-install')
>>> ws = zc.buildout.easy_install.install(
... ['demo'], dest, links=[link_server],
... index='http://www.python.org/pypi/',
- ... always_unzip=True, executable= other_executable)
+ ... always_unzip=True, executable= other_executable,
+ ... include_site_packages=False)
>>> ls(dest)
d demo-0.3-py%(other_version)s.egg
@@ -43,6 +45,31 @@
def multi_python(test):
other_executable = zc.buildout.testing.find_python(other_version)
+ command = textwrap.dedent('''\
+ try:
+ import setuptools
+ except ImportError:
+ import sys
+ sys.exit(1)
+ ''')
+ if subprocess.call([other_executable, '-c', command],
+ env=os.environ):
+ # the other executable does not have setuptools. Get setuptools.
+ # We will do this using the same tools we are testing, for better or
+ # worse. Alternatively, we could try using bootstrap.
+ executable_dir = test.globs['tmpdir']('executable_dir')
+ ws = zc.buildout.easy_install.install(
+ ['setuptools'], executable_dir,
+ index='http://www.python.org/pypi/',
+ always_unzip=True, executable=other_executable)
+ zc.buildout.easy_install.scripts(
+ ['setuptools'], ws, other_executable, executable_dir,
+ interpreter='py')
+ original_executable = other_executable
+ other_executable = os.path.join(executable_dir, 'py')
+ assert not subprocess.call(
+ [other_executable, '-c', command], env=os.environ), (
+ 'test set up failed')
sample_eggs = test.globs['tmpdir']('sample_eggs')
os.mkdir(os.path.join(sample_eggs, 'index'))
test.globs['sample_eggs'] = sample_eggs
Modified: zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/unzip.txt
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/unzip.txt 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/unzip.txt 2009-08-11 19:22:31 UTC (rev 102683)
@@ -21,9 +21,8 @@
d setuptools-0.6c8-py2.4.egg
- zc.buildout.egg-link
-This follows the
-policy followed by setuptools itself. Experience shows this policy
-to to be inconvenient. Zipped eggs make debugging more difficult and
+This follows the policy followed by setuptools itself. Experience shows this
+policy to to be inconvenient. Zipped eggs make debugging more difficult and
often import more slowly.
You can include an unzip option in the buildout section to change the
Modified: zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/update.txt
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/update.txt 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/src/zc/buildout/update.txt 2009-08-11 19:22:31 UTC (rev 102683)
@@ -21,7 +21,7 @@
... index = %(new_releases)s
... parts = show-versions
... develop = showversions
- ...
+ ...
... [show-versions]
... recipe = showversions
... """ % dict(new_releases=new_releases))
@@ -31,7 +31,7 @@
>>> mkdir(sample_buildout, 'showversions')
- >>> write(sample_buildout, 'showversions', 'showversions.py',
+ >>> write(sample_buildout, 'showversions', 'showversions.py',
... """
... import pkg_resources
...
@@ -52,7 +52,7 @@
>>> write(sample_buildout, 'showversions', 'setup.py',
... """
... from setuptools import setup
- ...
+ ...
... setup(
... name = "showversions",
... entry_points = {'zc.buildout': ['default = showversions:Recipe']},
@@ -80,14 +80,16 @@
Our buildout script has been updated to use the new eggs:
- >>> cat(sample_buildout, 'bin', 'buildout')
+ >>> cat(sample_buildout, 'bin', 'buildout') # doctest: +ELLIPSIS
#!/usr/local/bin/python2.4
<BLANKLINE>
- import sys
- sys.path[0:0] = [
- '/sample-buildout/eggs/zc.buildout-99.99-py2.4.egg',
- '/sample-buildout/eggs/setuptools-99.99-py2.4.egg',
- ]
+ import sys, os
+ pythonpath = filter(None, os.environ.get('PYTHONPATH', '').split(':'))
+ sys.path[:] = pythonpath + [
+ '/sample-buildout/eggs/zc.buildout-99.99-py2.4.egg',
+ '/sample-buildout/eggs/setuptools-99.99-py2.4.egg',
+ ...
+ ]
<BLANKLINE>
import zc.buildout.buildout
<BLANKLINE>
@@ -107,7 +109,7 @@
... develop = showversions
... zc.buildout-version = < 99
... setuptools-version = < 99
- ...
+ ...
... [show-versions]
... recipe = showversions
... """ % dict(new_releases=new_releases))
@@ -126,7 +128,7 @@
setuptools 0.6
There are a number of cases, described below, in which the updates
-don't happen.
+don't happen.
We won't upgrade in offline mode:
@@ -137,7 +139,7 @@
... index = %(new_releases)s
... parts = show-versions
... develop = showversions
- ...
+ ...
... [show-versions]
... recipe = showversions
... """ % dict(new_releases=new_releases))
@@ -166,9 +168,9 @@
... [buildout]
... find-links = %(new_releases)s
... index = %(new_releases)s
- ... parts =
+ ... parts =
... """ % dict(new_releases=new_releases))
-
+
>>> cd(sample_buildout2)
>>> print system(buildout),
Creating directory '/sample_buildout2/bin'.
Modified: zc.buildout/branches/gary-support-system-python-2/zc.recipe.egg_/setup.py
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/zc.recipe.egg_/setup.py 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/zc.recipe.egg_/setup.py 2009-08-11 19:22:31 UTC (rev 102683)
@@ -16,7 +16,7 @@
$Id$
"""
-version = '0'
+version = '1.3.0dev'
import os
from setuptools import setup, find_packages
@@ -66,7 +66,7 @@
package_dir = {'':'src'},
namespace_packages = ['zc', 'zc.recipe'],
install_requires = [
- 'zc.buildout >=1.2.0',
+ 'zc.buildout >=1.4.0dev',
'setuptools'],
tests_require = ['zope.testing'],
test_suite = name+'.tests.test_suite',
Modified: zc.buildout/branches/gary-support-system-python-2/zc.recipe.egg_/src/zc/recipe/egg/README.txt
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/zc.recipe.egg_/src/zc/recipe/egg/README.txt 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/zc.recipe.egg_/src/zc/recipe/egg/README.txt 2009-08-11 19:22:31 UTC (rev 102683)
@@ -96,7 +96,7 @@
>>> ls(sample_buildout, 'bin')
- buildout
-If we want scripts provided by eggs to be installed, we should use the
+If we want scripts provided by eggs to be installed, we should use the
scripts recipe:
>>> write(sample_buildout, 'buildout.cfg',
@@ -135,7 +135,7 @@
This option is useful when working with distributions that don't
declare entry points, such as distributions not written to work
- with setuptools.
+ with setuptools.
Examples can be seen in the section "Specifying entry points" below.
@@ -271,7 +271,7 @@
- setuptools-0.6-py2.3.egg
- zc.buildout-1.0-py2.3.egg
-If we run the buildout on the default online and newest modes,
+If we run the buildout on the default online and newest modes,
we'll get an update for demo:
>>> print system(buildout),
@@ -372,16 +372,19 @@
Let's look at the script that was generated:
- >>> cat(sample_buildout, 'bin', 'foo') # doctest: +NORMALIZE_WHITESPACE
+ >>> cat(sample_buildout, 'bin', 'foo')
+ ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
#!/usr/local/bin/python2.4
<BLANKLINE>
- import sys
- sys.path[0:0] = [
- '/sample-buildout/eggs/demo-0.4c1-py2.4.egg',
- '/sample-buildout/eggs/demoneeded-1.2c1-py2.4.egg',
- '/foo/bar',
- '/sample-buildout/spam',
- ]
+ import sys, os
+ pythonpath = filter(None, os.environ.get('PYTHONPATH', '').split(':'))
+ sys.path[:] = pythonpath + [
+ '/sample-buildout/eggs/demo-0.4c1-py2.4.egg',
+ '/sample-buildout/eggs/demoneeded-1.2c1-py2.4.egg',
+ '/foo/bar',
+ '/sample-buildout/spam',
+ ...
+ ]
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
@@ -419,7 +422,8 @@
Let's look at the script that was generated:
- >>> cat(sample_buildout, 'bin', 'foo') # doctest: +NORMALIZE_WHITESPACE
+ >>> cat(sample_buildout, 'bin', 'foo')
+ ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
#!/usr/local/bin/python2.4
<BLANKLINE>
import os
@@ -428,13 +432,15 @@
base = os.path.dirname(os.path.abspath(__file__))
base = os.path.dirname(base)
<BLANKLINE>
- import sys
- sys.path[0:0] = [
- join(base, 'eggs/demo-0.4c1-pyN.N.egg'),
- join(base, 'eggs/demoneeded-1.2c1-pyN.N.egg'),
- '/foo/bar',
- join(base, 'spam'),
- ]
+ import sys, os
+ pythonpath = filter(None, os.environ.get('PYTHONPATH', '').split(':'))
+ sys.path[:] = pythonpath + [
+ join(base, 'eggs/demo-0.4c1-pyN.N.egg'),
+ join(base, 'eggs/demoneeded-1.2c1-pyN.N.egg'),
+ '/foo/bar',
+ join(base, 'spam'),
+ ...
+ ]
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
@@ -466,7 +472,8 @@
Installing demo.
Generated script '/sample-buildout/bin/foo'.
- >>> cat(sample_buildout, 'bin', 'foo') # doctest: +NORMALIZE_WHITESPACE
+ >>> cat(sample_buildout, 'bin', 'foo')
+ ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
#!/usr/local/bin/python2.4
<BLANKLINE>
import os
@@ -475,13 +482,15 @@
base = os.path.dirname(os.path.abspath(__file__))
base = os.path.dirname(base)
<BLANKLINE>
- import sys
- sys.path[0:0] = [
- join(base, 'eggs/demo-0.4c1-pyN.N.egg'),
- join(base, 'eggs/demoneeded-1.2c1-pyN.N.egg'),
- '/foo/bar',
- join(base, 'spam'),
- ]
+ import sys, os
+ pythonpath = filter(None, os.environ.get('PYTHONPATH', '').split(':'))
+ sys.path[:] = pythonpath + [
+ join(base, 'eggs/demo-0.4c1-pyN.N.egg'),
+ join(base, 'eggs/demoneeded-1.2c1-pyN.N.egg'),
+ '/foo/bar',
+ join(base, 'spam'),
+ ...
+ ]
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
@@ -519,16 +528,19 @@
Installing demo.
Generated script '/sample-buildout/bin/foo'.
- >>> cat(sample_buildout, 'bin', 'foo') # doctest: +NORMALIZE_WHITESPACE
+ >>> cat(sample_buildout, 'bin', 'foo')
+ ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
#!/usr/local/bin/python2.4
<BLANKLINE>
- import sys
- sys.path[0:0] = [
- '/sample-buildout/eggs/demo-0.4c1-py2.4.egg',
- '/sample-buildout/eggs/demoneeded-1.2c1-py2.4.egg',
- '/foo/bar',
- '/sample-buildout/spam',
- ]
+ import sys, os
+ pythonpath = filter(None, os.environ.get('PYTHONPATH', '').split(':'))
+ sys.path[:] = pythonpath + [
+ '/sample-buildout/eggs/demo-0.4c1-py2.4.egg',
+ '/sample-buildout/eggs/demoneeded-1.2c1-py2.4.egg',
+ '/foo/bar',
+ '/sample-buildout/spam',
+ ...
+ ]
<BLANKLINE>
a = (1, 2
3, 4)
@@ -577,16 +589,18 @@
- demo
- other
- >>> cat(sample_buildout, 'bin', 'other')
+ >>> cat(sample_buildout, 'bin', 'other') # doctest: +ELLIPSIS
#!/usr/local/bin/python2.4
<BLANKLINE>
- import sys
- sys.path[0:0] = [
- '/sample-buildout/eggs/demo-0.4c1-py2.4.egg',
- '/sample-buildout/eggs/demoneeded-1.2c1-py2.4.egg',
- '/foo/bar',
- '/sample-buildout/spam',
- ]
+ import sys, os
+ pythonpath = filter(None, os.environ.get('PYTHONPATH', '').split(':'))
+ sys.path[:] = pythonpath + [
+ '/sample-buildout/eggs/demo-0.4c1-py2.4.egg',
+ '/sample-buildout/eggs/demoneeded-1.2c1-py2.4.egg',
+ '/foo/bar',
+ '/sample-buildout/spam',
+ ...
+ ]
<BLANKLINE>
import foo.bar
<BLANKLINE>
Modified: zc.buildout/branches/gary-support-system-python-2/zc.recipe.egg_/src/zc/recipe/egg/egg.py
===================================================================
--- zc.buildout/branches/gary-support-system-python-2/zc.recipe.egg_/src/zc/recipe/egg/egg.py 2009-08-11 18:57:47 UTC (rev 102682)
+++ zc.buildout/branches/gary-support-system-python-2/zc.recipe.egg_/src/zc/recipe/egg/egg.py 2009-08-11 19:22:31 UTC (rev 102683)
@@ -25,8 +25,8 @@
self.buildout = buildout
self.name = name
self.options = options
- links = options.get('find-links',
- buildout['buildout'].get('find-links'))
+ b_options = buildout['buildout']
+ links = options.get('find-links', b_options['find-links'])
if links:
links = links.split()
options['find-links'] = '\n'.join(links)
@@ -34,33 +34,40 @@
links = ()
self.links = links
- index = options.get('index', buildout['buildout'].get('index'))
+ index = options.get('index', b_options.get('index'))
if index is not None:
options['index'] = index
self.index = index
- allow_hosts = buildout['buildout'].get('allow-hosts', '*')
+ allow_hosts = b_options['allow-hosts']
allow_hosts = tuple([host.strip() for host in allow_hosts.split('\n')
if host.strip()!=''])
- self.allow_hosts = allow_hosts
+ self.allow_hosts = allow_hosts
- options['eggs-directory'] = buildout['buildout']['eggs-directory']
+ options['eggs-directory'] = b_options['eggs-directory']
options['_e'] = options['eggs-directory'] # backward compat.
- options['develop-eggs-directory'
- ] = buildout['buildout']['develop-eggs-directory']
+ options['develop-eggs-directory'] = b_options['develop-eggs-directory']
options['_d'] = options['develop-eggs-directory'] # backward compat.
- assert options.get('unzip') in ('true', 'false', None)
+ # verify that these are None, 'true' or 'false'
+ get_bool(options, 'unzip')
+ get_bool(options, 'include-site-packages')
- python = options.get('python', buildout['buildout']['python'])
+ python = options.get('python', b_options['python'])
options['executable'] = buildout[python]['executable']
+ @property
+ def include_site_packages(self):
+ return get_bool(self.options, 'include-site-packages',
+ self.buildout['buildout']['include-site-packages'])
+
def working_set(self, extra=()):
"""Separate method to just get the working set
This is intended for reuse by similar recipes.
"""
options = self.options
+ b_options = self.buildout['buildout']
distributions = [
r.strip()
@@ -68,25 +75,31 @@
if r.strip()]
orig_distributions = distributions[:]
distributions.extend(extra)
-
+ kw = {
+ 'allowed_eggs_from_site_packages': tuple(
+ name.strip() for name in
+ options.get(
+ 'allowed-eggs-from-site-packages',
+ b_options['allowed-eggs-from-site-packages']).split('\n'))}
if self.buildout['buildout'].get('offline') == 'true':
ws = zc.buildout.easy_install.working_set(
distributions, options['executable'],
- [options['develop-eggs-directory'], options['eggs-directory']]
+ [options['develop-eggs-directory'], options['eggs-directory']],
+ include_site_packages = self.include_site_packages,
+ **kw
)
else:
- kw = {}
- if options.get('unzip'):
+ if 'unzip' in options:
kw['always_unzip'] = get_bool(options, 'unzip')
-
ws = zc.buildout.easy_install.install(
distributions, options['eggs-directory'],
- links = self.links,
- index = self.index,
- executable = options['executable'],
+ links=self.links,
+ index=self.index,
+ executable=options['executable'],
path=[options['develop-eggs-directory']],
newest=self.buildout['buildout'].get('newest') == 'true',
allow_hosts=self.allow_hosts,
+ include_site_packages = self.include_site_packages,
**kw)
return orig_distributions, ws
@@ -115,7 +128,7 @@
relative_paths = options.get(
- 'relative-paths',
+ 'relative-paths',
buildout['buildout'].get('relative-paths', 'false')
)
if relative_paths == 'true':
@@ -150,7 +163,7 @@
reqs.append(parsed.groups())
if get_bool(options, 'dependent-scripts'):
- # generate scripts for all packages in the working set,
+ # Generate scripts for all packages in the working set,
# except setuptools.
reqs = list(reqs)
for dist in ws:
@@ -167,6 +180,7 @@
initialization=options.get('initialization', ''),
arguments=options.get('arguments', ''),
relative_paths=self._relative_paths,
+ include_site_packages = self.include_site_packages,
)
return ()
@@ -183,6 +197,6 @@
return False
else:
raise zc.buildout.UserError(
- "Invalid value for %s: %s" % (name, value))
+ "Invalid value for %s option: %s" % (name, value))
Egg = Scripts
More information about the Checkins
mailing list