[Checkins] SVN: zc.buildout/branches/gary-4/ this is now the branch for my site-packages changes. It is merged with trunk. The zc.recipe.egg recipe returns to being fully backward compatible; if you want to use a system Python, use the z3c.recipe.scripts recipe for scripts and interpreter.
Gary Poster
gary.poster at canonical.com
Wed Feb 10 20:04:30 EST 2010
Log message for revision 108916:
this is now the branch for my site-packages changes. It is merged with trunk. The zc.recipe.egg recipe returns to being fully backward compatible; if you want to use a system Python, use the z3c.recipe.scripts recipe for scripts and interpreter.
Changed:
_U zc.buildout/branches/gary-4/
U zc.buildout/branches/gary-4/CHANGES.txt
U zc.buildout/branches/gary-4/README.txt
U zc.buildout/branches/gary-4/buildout.cfg
U zc.buildout/branches/gary-4/src/zc/buildout/easy_install.py
U zc.buildout/branches/gary-4/src/zc/buildout/easy_install.txt
U zc.buildout/branches/gary-4/src/zc/buildout/testing.py
U zc.buildout/branches/gary-4/src/zc/buildout/tests.py
U zc.buildout/branches/gary-4/src/zc/buildout/testselectingpython.py
U zc.buildout/branches/gary-4/src/zc/buildout/update.txt
A zc.buildout/branches/gary-4/z3c.recipe.scripts_/
U zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/README.txt
U zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/api.txt
U zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/custom.txt
U zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/egg.py
U zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/selecting-python.txt
-=-
Property changes on: zc.buildout/branches/gary-4
___________________________________________________________________
Modified: svn:ignore
- eggs
develop-eggs
parts
.installed.cfg
bin
dist
build
doc.txt
doc.html
+ eggs
*.egg-info
develop-eggs
parts
.installed.cfg
bin
dist
build
doc.txt
doc.html
Modified: zc.buildout/branches/gary-4/CHANGES.txt
===================================================================
--- zc.buildout/branches/gary-4/CHANGES.txt 2010-02-10 22:09:17 UTC (rev 108915)
+++ zc.buildout/branches/gary-4/CHANGES.txt 2010-02-11 01:04:30 UTC (rev 108916)
@@ -6,6 +6,25 @@
New Features:
+- Buildout can be safely used with a system Python, as long as you use the
+ new z3c.recipe.scripts recipe to generate scripts and interpreters, rather
+ than zc.recipe.egg (which is still a fully supported, and simpler, way of
+ generating scripts and interpreters if you are using a "clean" Python).
+
+ A hopefully slight limitation: in no cases are distributions in your
+ site-packages used to satisfy buildout dependencies. The
+ site-packages can be used in addition to the dependencies specified in
+ your buildout, and buildout dependencies can override code in your
+ site-packages, but even if your Python's site-packages has the same
+ exact version as specified in your buildout configuration, buildout
+ will still use its own copy.
+
+- Added new function, ``zc.buildout.easy_install.generate_scripts``, to
+ generate scripts and interpreter. It produces a full-featured
+ interpreter (all command-line options supported) and the ability to
+ safely let scripts include site packages. The ``z3c.recipe.scripts``
+ recipe uses this new function.
+
- Improve bootstrap.
* New options let you specify where to find ez_setup.py and where to find
@@ -23,6 +42,17 @@
This means, among other things, that ``bin/buildout -vv`` and
``bin/buildout annotate`` correctly list more of the options.
+- Installing a namespace package using a Python that already has a package
+ in the same namespace (e.g., in the Python's site-packages) failed in
+ some cases.
+
+- Another variation of this error showed itself when at least two
+ dependencies were in a shared location like site-packages, and the
+ first one met the "versions" setting. The first dependency would be
+ added, but subsequent dependencies from the same location (e.g.,
+ site-packages) would use the version of the package found in the
+ shared location, ignoring the version setting.
+
1.4.3 (2009-12-10)
==================
Modified: zc.buildout/branches/gary-4/README.txt
===================================================================
--- zc.buildout/branches/gary-4/README.txt 2010-02-10 22:09:17 UTC (rev 108915)
+++ zc.buildout/branches/gary-4/README.txt 2010-02-11 01:04:30 UTC (rev 108916)
@@ -37,6 +37,11 @@
dependencies. It installs their console-script entry points with
the needed eggs included in their paths.
+`z3c.recipe.scripts <http://pypi.python.org/pypi/z3c.recipe.scripts>`_
+ This scripts recipe builds interpreter scripts and entry point scripts
+ based on eggs. These scripts have more features and flexibility than the
+ ones offered by zc.recipe.egg.
+
`zc.recipe.testrunner <http://pypi.python.org/pypi/zc.recipe.testrunner>`_
The testrunner egg creates a test runner script for one or
more eggs.
Modified: zc.buildout/branches/gary-4/buildout.cfg
===================================================================
--- zc.buildout/branches/gary-4/buildout.cfg 2010-02-10 22:09:17 UTC (rev 108915)
+++ zc.buildout/branches/gary-4/buildout.cfg 2010-02-11 01:04:30 UTC (rev 108916)
@@ -1,5 +1,5 @@
[buildout]
-develop = zc.recipe.egg_ .
+develop = zc.recipe.egg_ z3c.recipe.scripts_ .
parts = test oltest py
[py]
@@ -13,6 +13,7 @@
eggs =
zc.buildout
zc.recipe.egg
+ z3c.recipe.scripts
# Tests that can be run wo a network
[oltest]
@@ -20,6 +21,7 @@
eggs =
zc.buildout
zc.recipe.egg
+ z3c.recipe.scripts
defaults =
[
'-t',
Modified: zc.buildout/branches/gary-4/src/zc/buildout/easy_install.py
===================================================================
--- zc.buildout/branches/gary-4/src/zc/buildout/easy_install.py 2010-02-10 22:09:17 UTC (rev 108915)
+++ zc.buildout/branches/gary-4/src/zc/buildout/easy_install.py 2010-02-11 01:04:30 UTC (rev 108916)
@@ -137,11 +137,52 @@
else:
_safe_arg = str
-_easy_install_cmd = _safe_arg(
- 'from setuptools.command.easy_install import main; main()'
- )
+# The following string is used to run easy_install in
+# Installer._call_easy_install. It is started with python -S (that is,
+# don't import site at start). That flag, and all of the code in this
+# snippet above the last two lines, exist to work around a relatively rare
+# problem. If
+#
+# - your buildout configuration is trying to install a package that is within
+# a namespace package, and
+#
+# - you use a Python that has a different version of this package
+# installed in in its site-packages using
+# --single-version-externally-managed (that is, using the mechanism
+# sometimes used by system packagers:
+# http://peak.telecommunity.com/DevCenter/setuptools#install-command ), and
+#
+# - the new package tries to do sys.path tricks in the setup.py to get a
+# __version__,
+#
+# then the older package will be loaded first, making the setup version
+# the wrong number. While very arguably packages simply shouldn't do
+# the sys.path tricks, some do, and we don't want buildout to fall over
+# when they do.
+#
+# The namespace packages installed in site-packages with
+# --single-version-externally-managed use a mechanism that cause them to
+# be processed when site.py is imported. Simply starting Python with -S
+# addresses the problem in Python 2.4 and 2.5, but Python 2.6's distutils
+# imports a value from the site module, so we unfortunately have to do more
+# drastic surgery in the _easy_install_cmd code below. The changes to
+# sys.modules specifically try to only remove namespace modules installed by
+# the --single-version-externally-managed code.
+_easy_install_cmd = _safe_arg('''\
+import sys; \
+p = sys.path[:]; \
+m = sys.modules.keys(); \
+import site; \
+sys.path[:] = p; \
+m_attrs = set(('__builtins__', '__file__', '__package__', '__path__')); \
+match = set(('__path__',)); \
+[sys.modules.pop(k) for k, v in sys.modules.items()\
+ if k not in m and v and m_attrs.intersection(dir(v)) == match]; \
+from setuptools.command.easy_install import main; \
+main()''')
+
class Installer:
_versions = {}
@@ -301,7 +342,7 @@
try:
path = setuptools_loc
- args = ('-c', _easy_install_cmd, '-mUNxd', _safe_arg(tmp))
+ args = ('-Sc', _easy_install_cmd, '-mUNxd', _safe_arg(tmp))
if self._always_unzip:
args += ('-Z', )
level = logger.getEffectiveLevel()
@@ -904,6 +945,9 @@
def working_set(specs, executable, path):
return install(specs, None, executable=executable, path=path)
+############################################################################
+# Script generation functions
+
def scripts(reqs, working_set, executable, dest,
scripts=None,
extra_paths=(),
@@ -912,20 +956,85 @@
initialization='',
relative_paths=False,
):
+ """Generate scripts and/or an interpreter.
+ See generate_scripts for a newer version with more options and a
+ different approach.
+ """
+ path = _get_path(working_set, extra_paths)
+ if initialization:
+ initialization = '\n'+initialization+'\n'
+ generated = _generate_scripts(
+ reqs, working_set, dest, path, scripts, relative_paths,
+ initialization, executable, arguments)
+ if interpreter:
+ sname = os.path.join(dest, interpreter)
+ spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths)
+ generated.extend(
+ _pyscript(spath, sname, executable, rpsetup))
+ return generated
+
+def generate_scripts(
+ dest, working_set, executable, site_py_dest,
+ reqs=(), scripts=None, interpreter=None, extra_paths=(),
+ initialization='', add_site_packages=False, exec_sitecustomize=False,
+ relative_paths=False, script_arguments='', script_initialization=''):
+ """Generate scripts and/or an interpreter.
+
+ This accomplishes the same job as the ``scripts`` function, above,
+ but it does so in an alternative way that allows safely including
+ Python site packages, if desired, and choosing to execute the Python's
+ sitecustomize.
+ """
+ generated = []
+ generated.append(_generate_sitecustomize(
+ site_py_dest, executable, initialization, exec_sitecustomize))
+ generated.append(_generate_site(
+ site_py_dest, working_set, executable, extra_paths,
+ add_site_packages, relative_paths))
+ script_initialization = (
+ '\nimport site # imports custom buildbot-generated site.py\n%s' % (
+ script_initialization,))
+ if not script_initialization.endswith('\n'):
+ script_initialization += '\n'
+ generated.extend(_generate_scripts(
+ reqs, working_set, dest, [site_py_dest], scripts, relative_paths,
+ script_initialization, executable, script_arguments, block_site=True))
+ if interpreter:
+ generated.extend(_generate_interpreter(
+ interpreter, dest, executable, site_py_dest, relative_paths))
+ return generated
+
+# Utilities for the script generation functions.
+
+# These are shared by both ``scripts`` and ``generate_scripts``
+
+def _get_path(working_set, extra_paths=()):
+ """Given working set and extra paths, return a normalized path list."""
path = [dist.location for dist in working_set]
path.extend(extra_paths)
- path = map(realpath, path)
+ return map(realpath, path)
- generated = []
+def _generate_scripts(reqs, working_set, dest, path, scripts, relative_paths,
+ initialization, executable, arguments,
+ block_site=False):
+ """Generate scripts for the given requirements.
+ - reqs is an iterable of string requirements or entry points.
+ - The requirements must be findable in the given working_set.
+ - The dest is the directory in which the scripts should be created.
+ - The path is a list of paths that should be added to sys.path.
+ - The scripts is an optional dictionary. If included, the keys should be
+ the names of the scripts that should be created, as identified in their
+ entry points; and the values should be the name the script should
+ actually be created with.
+ - relative_paths, if given, should be the path that is the root of the
+ buildout (the common path that should be the root of what is relative).
+ """
if isinstance(reqs, str):
raise TypeError('Expected iterable of requirements or entry points,'
' got string.')
-
- if initialization:
- initialization = '\n'+initialization+'\n'
-
+ generated = []
entry_points = []
for req in reqs:
if isinstance(req, str):
@@ -939,7 +1048,6 @@
)
else:
entry_points.append(req)
-
for name, module_name, attrs in entry_points:
if scripts is not None:
sname = scripts.get(name)
@@ -947,40 +1055,48 @@
continue
else:
sname = name
-
sname = os.path.join(dest, sname)
spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths)
-
generated.extend(
- _script(module_name, attrs, spath, sname, executable, arguments,
- initialization, rpsetup)
- )
+ _script(sname, executable, rpsetup, spath, initialization,
+ module_name, attrs, arguments, block_site=block_site))
+ return generated
- if interpreter:
- sname = os.path.join(dest, interpreter)
- spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths)
- generated.extend(_pyscript(spath, sname, executable, rpsetup))
+def _relative_path_and_setup(sname, path,
+ relative_paths=False, indent_level=1):
+ """Return a string of code of paths and of setup if appropriate.
- return generated
-
-def _relative_path_and_setup(sname, path, relative_paths):
+ - sname is the full path to the script name to be created.
+ - path is the list of paths to be added to sys.path.
+ - relative_paths, if given, should be the path that is the root of the
+ buildout (the common path that should be the root of what is relative).
+ - indent_level is the number of four-space indents that the path should
+ insert before each element of the path.
+ """
if relative_paths:
relative_paths = os.path.normcase(relative_paths)
sname = os.path.normcase(os.path.abspath(sname))
- spath = ',\n '.join(
+ spath = _format_paths(
[_relativitize(os.path.normcase(path_item), sname, relative_paths)
- for path_item in path]
- )
+ for path_item in path], indent_level=indent_level)
rpsetup = relative_paths_setup
for i in range(_relative_depth(relative_paths, sname)):
rpsetup += "base = os.path.dirname(base)\n"
else:
- spath = repr(path)[1:-1].replace(', ', ',\n ')
+ spath = _format_paths((repr(p) for p in path),
+ indent_level=indent_level)
rpsetup = ''
return spath, rpsetup
+def _relative_depth(common, path):
+ """Return number of dirs separating ``path`` from ancestor, ``common``.
-def _relative_depth(common, path):
+ For instance, if path is /foo/bar/baz/bing, and common is /foo, this will
+ return 2--in UNIX, the number of ".." to get from bing's directory
+ to foo.
+
+ This is a helper for _relative_path_and_setup.
+ """
n = 0
while 1:
dirname = os.path.dirname(path)
@@ -993,6 +1109,11 @@
return n
def _relative_path(common, path):
+ """Return the relative path from ``common`` to ``path``.
+
+ This is a helper for _relativitize, which is a helper to
+ _relative_path_and_setup.
+ """
r = []
while 1:
dirname, basename = os.path.split(path)
@@ -1006,6 +1127,11 @@
return os.path.join(*r)
def _relativitize(path, script, relative_paths):
+ """Return a code string for the given path.
+
+ Path is relative to the base path ``relative_paths``if the common prefix
+ between ``path`` and ``script`` starts with ``relative_paths``.
+ """
if path == script:
raise AssertionError("path == script")
common = os.path.dirname(os.path.commonprefix([path, script]))
@@ -1016,7 +1142,6 @@
else:
return repr(path)
-
relative_paths_setup = """
import os
@@ -1024,58 +1149,78 @@
base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
"""
-def _script(module_name, attrs, path, dest, executable, arguments,
- initialization, rsetup):
+def _write_script(full_name, contents, logged_type):
+ """Write contents of script in full_name, logging the action.
+
+ The only tricky bit in this function is that it supports Windows by
+ creating exe files using a pkg_resources helper.
+ """
generated = []
- script = dest
+ script_name = full_name
if is_win32:
- dest += '-script.py'
-
- contents = script_template % dict(
- python = _safe_arg(executable),
- path = path,
- module_name = module_name,
- attrs = attrs,
- arguments = arguments,
- initialization = initialization,
- relative_paths_setup = rsetup,
- )
- changed = not (os.path.exists(dest) and open(dest).read() == contents)
-
- if is_win32:
- # generate exe file and give the script a magic name:
- exe = script+'.exe'
+ script_name += '-script.py'
+ # Generate exe file and give the script a magic name.
+ exe = full_name + '.exe'
new_data = pkg_resources.resource_string('setuptools', 'cli.exe')
if not os.path.exists(exe) or (open(exe, 'rb').read() != new_data):
# Only write it if it's different.
open(exe, 'wb').write(new_data)
generated.append(exe)
-
+ changed = not (os.path.exists(script_name) and
+ open(script_name).read() == contents)
if changed:
- open(dest, 'w').write(contents)
- logger.info("Generated script %r.", script)
-
+ open(script_name, 'w').write(contents)
try:
- os.chmod(dest, 0755)
+ os.chmod(script_name, 0755)
except (AttributeError, os.error):
pass
-
- generated.append(dest)
+ logger.info("Generated %s %r.", logged_type, full_name)
+ generated.append(script_name)
return generated
+def _format_paths(paths, indent_level=1):
+ """Format paths for inclusion in a script."""
+ separator = ',\n' + indent_level * ' '
+ return separator.join(paths)
+
+def _script(dest, executable, relative_paths_setup, path, initialization,
+ module_name, attrs, arguments, block_site=False):
+ if block_site:
+ dash_S = ' -S'
+ else:
+ dash_S = ''
+ contents = script_template % dict(
+ python=_safe_arg(executable),
+ dash_S=dash_S,
+ path=path,
+ module_name=module_name,
+ attrs=attrs,
+ arguments=arguments,
+ initialization=initialization,
+ relative_paths_setup=relative_paths_setup,
+ )
+ return _write_script(dest, contents, 'script')
+
if is_jython and jython_os_name == 'linux':
- script_header = '#!/usr/bin/env %(python)s'
+ script_header = '#!/usr/bin/env %(python)s%(dash_S)s'
else:
- script_header = '#!%(python)s'
+ script_header = '#!%(python)s%(dash_S)s'
+sys_path_template = '''\
+import sys
+sys.path[0:0] = [
+ %s,
+ ]
+'''
script_template = script_header + '''\
%(relative_paths_setup)s
import sys
sys.path[0:0] = [
- %(path)s,
- ]
+ %(path)s,
+ ]
+
%(initialization)s
import %(module_name)s
@@ -1083,47 +1228,25 @@
%(module_name)s.%(attrs)s(%(arguments)s)
'''
+# These are used only by the older ``scripts`` function.
def _pyscript(path, dest, executable, rsetup):
- generated = []
- script = dest
- if is_win32:
- dest += '-script.py'
-
contents = py_script_template % dict(
- python = _safe_arg(executable),
- path = path,
- relative_paths_setup = rsetup,
+ python=_safe_arg(executable),
+ dash_S='',
+ path=path,
+ relative_paths_setup=rsetup,
)
- changed = not (os.path.exists(dest) and open(dest).read() == contents)
+ return _write_script(dest, contents, 'interpreter')
- if is_win32:
- # generate exe file and give the script a magic name:
- exe = script + '.exe'
- open(exe, 'wb').write(
- pkg_resources.resource_string('setuptools', 'cli.exe')
- )
- generated.append(exe)
-
- if changed:
- open(dest, 'w').write(contents)
- try:
- os.chmod(dest,0755)
- except (AttributeError, os.error):
- pass
- logger.info("Generated interpreter %r.", script)
-
- generated.append(dest)
- return generated
-
py_script_template = script_header + '''\
%(relative_paths_setup)s
import sys
sys.path[0:0] = [
- %(path)s,
- ]
+ %(path)s,
+ ]
_interactive = True
if len(sys.argv) > 1:
@@ -1151,6 +1274,313 @@
__import__("code").interact(banner="", local=globals())
'''
+# These are used only by the newer ``generate_scripts`` function.
+
+def _get_system_paths(executable):
+ """return lists of standard lib and site paths for executable.
+ """
+ # We want to get a list of the site packages, which is not easy.
+ # The canonical way to do this is to use
+ # distutils.sysconfig.get_python_lib(), but that only returns a
+ # single path, which does not reflect reality for many system
+ # Pythons, which have multiple additions. Instead, we start Python
+ # with -S, which does not import site.py and set up the extra paths
+ # like site-packages or (Ubuntu/Debian) dist-packages and
+ # python-support. We then compare that sys.path with the normal one
+ # (minus user packages if this is Python 2.6, because we don't
+ # support those (yet?). The set of the normal one minus the set of
+ # the ones in ``python -S`` is the set of packages that are
+ # effectively site-packages.
+ #
+ # The given executable might not be the current executable, so it is
+ # appropriate to do another subprocess to figure out what the
+ # additional site-package paths are. Moreover, even if this
+ # executable *is* the current executable, this code might be run in
+ # the context of code that has manipulated the sys.path--for
+ # instance, to add local zc.buildout or setuptools eggs.
+ def get_sys_path(*args, **kwargs):
+ cmd = [executable]
+ cmd.extend(args)
+ cmd.extend([
+ "-c", "import sys, os;"
+ "print repr([os.path.normpath(p) for p in sys.path if p])"])
+ # Windows needs some (as yet to be determined) part of the real env.
+ env = os.environ.copy()
+ env.update(kwargs)
+ _proc = subprocess.Popen(
+ cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
+ stdout, stderr = _proc.communicate();
+ if _proc.returncode:
+ raise RuntimeError(
+ 'error trying to get system packages:\n%s' % (stderr,))
+ res = eval(stdout.strip())
+ try:
+ res.remove('.')
+ except ValueError:
+ pass
+ return res
+ stdlib = get_sys_path('-S') # stdlib only
+ no_user_paths = get_sys_path(PYTHONNOUSERSITE='x')
+ site_paths = [p for p in no_user_paths if p not in stdlib]
+ return (stdlib, site_paths)
+
+def _get_module_file(executable, name):
+ """Return a module's file path.
+
+ - executable is a path to the desired Python executable.
+ - name is the name of the (pure, not C) Python module.
+ """
+ cmd = [executable, "-c",
+ "import imp; "
+ "fp, path, desc = imp.find_module(%r); "
+ "fp.close; "
+ "print path" % (name,)]
+ _proc = subprocess.Popen(
+ cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ stdout, stderr = _proc.communicate();
+ if _proc.returncode:
+ logger.info(
+ 'Could not find file for module %s:\n%s', name, stderr)
+ return None
+ # else: ...
+ res = stdout.strip()
+ if res.endswith('.pyc') or res.endswith('.pyo'):
+ raise RuntimeError('Cannot find uncompiled version of %s' % (name,))
+ if not os.path.exists(res):
+ raise RuntimeError(
+ 'File does not exist for module %s:\n%s' % (name, res))
+ return res
+
+def _generate_sitecustomize(dest, executable, initialization='',
+ exec_sitecustomize=False):
+ """Write a sitecustomize file with optional custom initialization.
+
+ The created script will execute the underlying Python's
+ sitecustomize if exec_sitecustomize is True.
+ """
+ sitecustomize_path = os.path.join(dest, 'sitecustomize.py')
+ sitecustomize = open(sitecustomize_path, 'w')
+ if initialization:
+ sitecustomize.write(initialization + '\n')
+ if exec_sitecustomize:
+ real_sitecustomize_path = _get_module_file(
+ executable, 'sitecustomize')
+ if real_sitecustomize_path:
+ real_sitecustomize = open(real_sitecustomize_path, 'r')
+ sitecustomize.write(
+ '\n# The following is from\n# %s\n' %
+ (real_sitecustomize_path,))
+ sitecustomize.write(real_sitecustomize.read())
+ real_sitecustomize.close()
+ sitecustomize.close()
+ return sitecustomize_path
+
+def _generate_site(dest, working_set, executable, extra_paths=(),
+ add_site_packages=False, relative_paths=False):
+ """Write a site.py file with eggs from working_set.
+
+ extra_paths will be added to the path. If add_site_packages is True,
+ paths from the underlying Python will be added.
+ """
+ path = _get_path(working_set, extra_paths)
+ site_path = os.path.join(dest, 'site.py')
+ path_string, rpsetup = _relative_path_and_setup(
+ site_path, path, relative_paths, indent_level=2)
+ if rpsetup:
+ rpsetup = '\n'.join(
+ [(line and ' %s' % (line,) or line)
+ for line in rpsetup.split('\n')])
+ real_site_path = _get_module_file(executable, 'site')
+ real_site = open(real_site_path, 'r')
+ site = open(site_path, 'w')
+ extra_path_snippet = add_site_packages_snippet[add_site_packages]
+ extra_path_snippet_followup = add_site_packages_snippet_followup[
+ add_site_packages]
+ if add_site_packages:
+ stdlib, site_paths = _get_system_paths(executable)
+ extra_path_snippet = extra_path_snippet % _format_paths(
+ (repr(p) for p in site_paths), 2)
+ addsitepackages_marker = 'def addsitepackages('
+ enableusersite_marker = 'ENABLE_USER_SITE = '
+ successful_rewrite = False
+ for line in real_site.readlines():
+ if line.startswith(enableusersite_marker):
+ site.write(enableusersite_marker)
+ site.write('False # buildout does not support user sites.\n')
+ elif line.startswith(addsitepackages_marker):
+ site.write(addsitepackages_script % (
+ extra_path_snippet, rpsetup, path_string,
+ extra_path_snippet_followup))
+ site.write(line[len(addsitepackages_marker):])
+ successful_rewrite = True
+ else:
+ site.write(line)
+ if not successful_rewrite:
+ raise RuntimeError('Buildout did not successfully rewrite site.py')
+ return site_path
+
+add_site_packages_snippet = ['''
+ paths = []''', '''
+ paths = [ # These are the underlying Python's site-packages.
+ %s]
+ sys.path[0:0] = paths
+ known_paths.update([os.path.normcase(os.path.abspath(p)) for p in paths])
+ try:
+ import pkg_resources
+ except ImportError:
+ # No namespace packages in sys.path; no fixup needed.
+ pkg_resources = None''']
+
+add_site_packages_snippet_followup = ['', '''
+ if pkg_resources is not None:
+ # There may be namespace packages in sys.path. This is much faster
+ # than importing pkg_resources after the sys.path has a large number
+ # of eggs.
+ for p in sys.path:
+ pkg_resources.fixup_namespace_packages(p)''']
+
+addsitepackages_script = '''\
+def addsitepackages(known_paths):%s
+%s paths[0:0] = [ # eggs
+ %s
+ ]
+ # Process all dirs. Look for .pth files. If they exist, defer
+ # processing "import" varieties.
+ dotpth = os.extsep + "pth"
+ deferred = []
+ for path in reversed(paths):
+ # Duplicating addsitedir.
+ sitedir, sitedircase = makepath(path)
+ if not sitedircase in known_paths and os.path.exists(sitedir):
+ sys.path.insert(0, sitedir)
+ known_paths.add(sitedircase)
+ try:
+ names = os.listdir(sitedir)
+ except os.error:
+ continue
+ names = [name for name in names if name.endswith(dotpth)]
+ names.sort()
+ for name in names:
+ # Duplicating addpackage.
+ fullname = os.path.join(sitedir, name)
+ try:
+ f = open(fullname, "rU")
+ except IOError:
+ continue
+ try:
+ for line in f:
+ if line.startswith("#"):
+ continue
+ if (line.startswith("import ") or
+ line.startswith("import\t")):
+ # This line is supposed to be executed. It
+ # might be a setuptools namespace package
+ # installed with a system package manager.
+ # Defer this so we can process egg namespace
+ # packages first, or else the eggs with the same
+ # namespace will be ignored.
+ deferred.append((sitedir, name, fullname, line))
+ continue
+ line = line.rstrip()
+ dir, dircase = makepath(sitedir, line)
+ if not dircase in known_paths and os.path.exists(dir):
+ sys.path.append(dir)
+ known_paths.add(dircase)
+ finally:
+ f.close()%s
+ # Process "import ..." .pth lines.
+ for sitedir, name, fullname, line in deferred:
+ # Note that some lines--such as the one setuptools writes for
+ # namespace packages--expect some or all of sitedir, name, and
+ # fullname to be present in the frame locals, as it is in
+ # ``addpackage``.
+ try:
+ exec line
+ except:
+ print "Error in %%s" %% (fullname,)
+ raise
+ global addsitepackages
+ addsitepackages = original_addsitepackages
+ return known_paths
+
+buildout_addsitepackages = addsitepackages
+
+def original_addsitepackages('''
+
+def _generate_interpreter(name, dest, executable, site_py_dest,
+ relative_paths=False):
+ """Write an interpreter script, using the site.py approach."""
+ full_name = os.path.join(dest, name)
+ site_py_dest_string, rpsetup = _relative_path_and_setup(
+ full_name, [site_py_dest], relative_paths)
+ if sys.platform == 'win32':
+ windows_import = '\nimport subprocess'
+ # os.exec* is a mess on Windows, particularly if the path
+ # to the executable has spaces and the Python is using MSVCRT.
+ # The standard fix is to surround the executable's path with quotes,
+ # but that has been unreliable in testing.
+ #
+ # Here's a demonstration of the problem. Given a Python
+ # compiled with a MSVCRT-based compiler, such as the free Visual
+ # C++ 2008 Express Edition, and an executable path with spaces
+ # in it such as the below, we see the following.
+ #
+ # >>> import os
+ # >>> p0 = 'C:\\Documents and Settings\\Administrator\\My Documents\\Downloads\\Python-2.6.4\\PCbuild\\python.exe'
+ # >>> os.path.exists(p0)
+ # True
+ # >>> os.execv(p0, [])
+ # Traceback (most recent call last):
+ # File "<stdin>", line 1, in <module>
+ # OSError: [Errno 22] Invalid argument
+ #
+ # That seems like a standard problem. The standard solution is
+ # to quote the path (see, for instance
+ # http://bugs.python.org/issue436259). However, this solution,
+ # and other variations, fail:
+ #
+ # >>> p1 = '"C:\\Documents and Settings\\Administrator\\My Documents\\Downloads\\Python-2.6.4\\PCbuild\\python.exe"'
+ # >>> os.execv(p1, [])
+ # Traceback (most recent call last):
+ # File "<stdin>", line 1, in <module>
+ # OSError: [Errno 22] Invalid argument
+ #
+ # We simply use subprocess instead, since it handles everything
+ # nicely, and the transparency of exec* (that is, not running,
+ # perhaps unexpectedly, in a subprocess) is arguably not a
+ # necessity, at least for many use cases.
+ execute = 'subprocess.call(argv, env=environ)'
+ else:
+ windows_import = ''
+ execute = 'os.execve(sys.executable, argv, environ)'
+ contents = interpreter_template % dict(
+ python=_safe_arg(executable),
+ dash_S=' -S',
+ site_dest=site_py_dest_string,
+ relative_paths_setup=rpsetup,
+ windows_import=windows_import,
+ execute=execute,
+ )
+ return _write_script(full_name, contents, 'interpreter')
+
+interpreter_template = script_header + '''\
+
+%(relative_paths_setup)s
+import os
+import sys%(windows_import)s
+
+argv = [sys.executable] + sys.argv[1:]
+environ = os.environ.copy()
+path = %(site_dest)s
+if environ.get('PYTHONPATH'):
+ path = os.pathsep.join([path, environ['PYTHONPATH']])
+environ['PYTHONPATH'] = path
+%(execute)s
+'''
+
+# End of script generation code.
+############################################################################
+
runsetup_template = """
import sys
sys.path.insert(0, %(setupdir)r)
Modified: zc.buildout/branches/gary-4/src/zc/buildout/easy_install.txt
===================================================================
--- zc.buildout/branches/gary-4/src/zc/buildout/easy_install.txt 2010-02-10 22:09:17 UTC (rev 108915)
+++ zc.buildout/branches/gary-4/src/zc/buildout/easy_install.txt 2010-02-11 01:04:30 UTC (rev 108916)
@@ -521,25 +521,38 @@
Script generation
-----------------
-The easy_install module provides support for creating scripts from
-eggs. It provides a function similar to setuptools except that it
-provides facilities for baking a script's path into the script. This
-has two advantages:
+The easy_install module provides support for creating scripts from eggs.
+It provides two competing functions. One, ``scripts``, is a
+well-established approach to generating reliable scripts with a "clean"
+Python--e.g., one that does not have any packages in its site-packages.
+The other, ``generate_scripts``, is newer, a bit trickier, and is
+designed to work with a Python that has code in its site-packages, such
+as a system Python.
+Both are similar to setuptools except that they provides facilities for
+baking a script's path into the script. This has two advantages:
+
- The eggs to be used by a script are not chosen at run time, making
startup faster and, more importantly, deterministic.
-- The script doesn't have to import pkg_resources because the logic
- that pkg_resources would execute at run time is executed at
- script-creation time.
+- The script doesn't have to import pkg_resources because the logic that
+ pkg_resources would execute at run time is executed at script-creation
+ time. (There is an exception in ``generate_scripts`` if you want to
+ have your Python's site packages available, as discussed below, but
+ even in that case pkg_resources is only partially activated, which can
+ be a significant time savings.)
-The scripts method can be used to generate scripts. Let's create a
-destination directory for it to place them in:
- >>> import tempfile
+The ``scripts`` function
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``scripts`` function is the first way to generate scripts that we'll
+examine. It is the earlier approach that the package offered. Let's
+create a destination directory for it to place them in:
+
>>> bin = tmpdir('bin')
-Now, we'll use the scripts method to generate scripts in this directory
+Now, we'll use the scripts function to generate scripts in this directory
from the demo egg:
>>> import sys
@@ -736,8 +749,8 @@
>>> print system(os.path.join(bin, 'run')),
3 1
-Including extra paths in scripts
---------------------------------
+The ``scripts`` function: Including extra paths in scripts
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We can pass a keyword argument, extra paths, to cause additional paths
to be included in the a generated script:
@@ -762,8 +775,8 @@
if __name__ == '__main__':
eggrecipedemo.main()
-Providing script arguments
---------------------------
+The ``scripts`` function: Providing script arguments
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
An "argument" keyword argument can be used to pass arguments to an
entry point. The value passed is a source string to be placed between the
@@ -786,8 +799,8 @@
if __name__ == '__main__':
eggrecipedemo.main(1, 2)
-Passing initialization code
----------------------------
+The ``scripts`` function: Passing initialization code
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can also pass script initialization code:
@@ -812,8 +825,8 @@
if __name__ == '__main__':
eggrecipedemo.main(1, 2)
-Relative paths
---------------
+The ``scripts`` function: Relative paths
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sometimes, you want to be able to move a buildout directory around and
have scripts still work without having to rebuild them. We can
@@ -836,7 +849,7 @@
... interpreter='py',
... relative_paths=bo)
- >>> cat(bo, 'bin', 'run')
+ >>> cat(bo, 'bin', 'run') # doctest: +NORMALIZE_WHITESPACE
#!/usr/local/bin/python2.4
<BLANKLINE>
import os
@@ -868,7 +881,7 @@
We specified an interpreter and its paths are adjusted too:
- >>> cat(bo, 'bin', 'py')
+ >>> cat(bo, 'bin', 'py') # doctest: +NORMALIZE_WHITESPACE
#!/usr/local/bin/python2.4
<BLANKLINE>
import os
@@ -911,7 +924,556 @@
del _interactive
__import__("code").interact(banner="", local=globals())
+The ``generate_scripts`` function
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The newer function for creating scripts is ``generate_scripts``. It has the
+same basic functionality as the ``scripts`` function: it can create scripts
+to run arbitrary entry points, and to run a Python interpreter. The
+following are the differences from a user's perspective.
+
+- It can be used safely with a Python that has packages installed itself,
+ such as a system-installed Python.
+
+- In contrast to the interpreter generated by the ``scripts`` method, which
+ supports only a small subset of the usual Python executable's options,
+ the interpreter generated by ``generate_scripts`` supports all of them.
+ This makes it possible to use as full Python replacement for scripts that
+ need the distributions specified in your buildout.
+
+- Both the interpreter and the entry point scripts allow you to include the
+ site packages, and/or the sitecustomize, of the Python executable, if
+ desired.
+
+It works by creating site.py and sitecustomize.py files that set up the
+desired paths and initialization. These must be placed within an otherwise
+empty directory. Typically this is in a recipe's parts directory.
+
+Here's the simplest example, building an interpreter script.
+
+ >>> interpreter_dir = tmpdir('interpreter')
+ >>> interpreter_parts_dir = os.path.join(
+ ... interpreter_dir, 'parts', 'interpreter')
+ >>> interpreter_bin_dir = os.path.join(interpreter_dir, 'bin')
+ >>> mkdir(interpreter_bin_dir)
+ >>> mkdir(interpreter_dir, 'eggs')
+ >>> mkdir(interpreter_dir, 'parts')
+ >>> mkdir(interpreter_parts_dir)
+
+ >>> ws = zc.buildout.easy_install.install(
+ ... ['demo'], join(interpreter_dir, 'eggs'), links=[link_server],
+ ... index=link_server+'index/')
+ >>> generated = zc.buildout.easy_install.generate_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... interpreter='py')
+
+Depending on whether the machine being used is running Windows or not, this
+produces either three or four files. In both cases, we have site.py and
+sitecustomize.py generated in the parts/interpreter directory. For Windows,
+we have py.exe and py-script.py; for other operating systems, we have py.
+
+ >>> sitecustomize_path = os.path.join(
+ ... interpreter_parts_dir, 'sitecustomize.py')
+ >>> site_path = os.path.join(interpreter_parts_dir, 'site.py')
+ >>> interpreter_path = os.path.join(interpreter_bin_dir, 'py')
+ >>> if sys.platform == 'win32':
+ ... py_path = os.path.join(interpreter_bin_dir, 'py-script.py')
+ ... expected = [sitecustomize_path,
+ ... site_path,
+ ... os.path.join(interpreter_bin_dir, 'py.exe'),
+ ... py_path]
+ ... else:
+ ... py_path = interpreter_path
+ ... expected = [sitecustomize_path, site_path, py_path]
+ ...
+ >>> assert generated == expected, repr((generated, expected))
+
+We didn't ask for any initialization, and we didn't ask to use the underlying
+sitecustomization, so sitecustomize.py is empty.
+
+ >>> cat(sitecustomize_path)
+
+The interpreter script is simple. It puts the directory with the
+site.py and sitecustomize.py on the PYTHONPATH and (re)starts Python.
+
+ >>> cat(py_path)
+ #!/usr/bin/python2.4 -S
+ <BLANKLINE>
+ import os
+ import sys
+ <BLANKLINE>
+ argv = [sys.executable] + sys.argv[1:]
+ environ = os.environ.copy()
+ path = '/interpreter/parts/interpreter'
+ if environ.get('PYTHONPATH'):
+ path = os.pathsep.join([path, environ['PYTHONPATH']])
+ environ['PYTHONPATH'] = path
+ os.execve(sys.executable, argv, environ)
+
+The site.py file is a modified version of the underlying Python's site.py.
+The most important modification is that it has a different version of the
+addsitepackages function. It has all of the trickier bits, and sets up the
+Python path, similarly to the behavior of the function it replaces. The
+following shows the part that buildout inserts.
+
+ >>> sys.stdout.write('#\n'); cat(site_path)
+ ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
+ #...
+ def addsitepackages(known_paths):
+ paths = []
+ paths[0:0] = [ # eggs
+ '/interpreter/eggs/demo-0.3-pyN.N.egg',
+ '/interpreter/eggs/demoneeded-1.1-pyN.N.egg'
+ ]
+ # Process all dirs. Look for .pth files. If they exist, defer
+ # processing "import" varieties.
+ dotpth = os.extsep + "pth"
+ deferred = []
+ for path in reversed(paths):
+ # Duplicating addsitedir.
+ sitedir, sitedircase = makepath(path)
+ if not sitedircase in known_paths and os.path.exists(sitedir):
+ sys.path.insert(0, sitedir)
+ known_paths.add(sitedircase)
+ try:
+ names = os.listdir(sitedir)
+ except os.error:
+ continue
+ names = [name for name in names if name.endswith(dotpth)]
+ names.sort()
+ for name in names:
+ # Duplicating addpackage.
+ fullname = os.path.join(sitedir, name)
+ try:
+ f = open(fullname, "rU")
+ except IOError:
+ continue
+ try:
+ for line in f:
+ if line.startswith("#"):
+ continue
+ if (line.startswith("import ") or
+ line.startswith("import ")):
+ # This line is supposed to be executed. It
+ # might be a setuptools namespace package
+ # installed with a system package manager.
+ # Defer this so we can process egg namespace
+ # packages first, or else the eggs with the same
+ # namespace will be ignored.
+ deferred.append((sitedir, name, fullname, line))
+ continue
+ line = line.rstrip()
+ dir, dircase = makepath(sitedir, line)
+ if not dircase in known_paths and os.path.exists(dir):
+ sys.path.append(dir)
+ known_paths.add(dircase)
+ finally:
+ f.close()
+ # Process "import ..." .pth lines.
+ for sitedir, name, fullname, line in deferred:
+ # Note that some lines--such as the one setuptools writes for
+ # namespace packages--expect some or all of sitedir, name, and
+ # fullname to be present in the frame locals, as it is in
+ # ``addpackage``.
+ try:
+ exec line
+ except:
+ print "Error in %s" % (fullname,)
+ raise
+ global addsitepackages
+ addsitepackages = original_addsitepackages
+ return known_paths
+ <BLANKLINE>
+ buildout_addsitepackages = addsitepackages
+ <BLANKLINE>
+ def original_addsitepackages(known_paths):...
+
+As you can see, it manipulates the path to insert the eggs and then processes
+any .pth files. The lines in the .pth files that use the "import" feature
+are deferred because it is a pattern we will need in a later example, when we
+show how we can add site packages, and handle competing namespace packages
+in both site packages and eggs.
+
+Here are some examples of the interpreter in use.
+
+ >>> print call_py(interpreter_path, "print 16+26")
+ 42
+ <BLANKLINE>
+ >>> res = call_py(interpreter_path, "import sys; print sys.path")
+ >>> print res # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
+ ['',
+ '/interpreter/eggs/demo-0.3-pyN.N.egg',
+ '/interpreter/eggs/demoneeded-1.1-pyN.N.egg',
+ '/interpreter/parts/interpreter',
+ ...]
+ <BLANKLINE>
+ >>> clean_paths = eval(res.strip()) # This is used later for comparison.
+
+If you provide initialization, it goes in sitecustomize.py.
+
+ >>> def reset_interpreter():
+ ... # This is necessary because, in our tests, the timestamps of the
+ ... # .pyc files are not outdated when we want them to be.
+ ... rmdir(interpreter_bin_dir)
+ ... mkdir(interpreter_bin_dir)
+ ... rmdir(interpreter_parts_dir)
+ ... mkdir(interpreter_parts_dir)
+ ...
+ >>> reset_interpreter()
+
+ >>> initialization_string = """\
+ ... import os
+ ... os.environ['FOO'] = 'bar baz bing shazam'"""
+ >>> generated = zc.buildout.easy_install.generate_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... interpreter='py', initialization=initialization_string)
+ >>> cat(sitecustomize_path)
+ import os
+ os.environ['FOO'] = 'bar baz bing shazam'
+ >>> print call_py(interpreter_path, "import os; print os.environ['FOO']")
+ bar baz bing shazam
+ <BLANKLINE>
+
+If you use relative paths, this affects the interpreter and site.py.
+
+ >>> reset_interpreter()
+ >>> generated = zc.buildout.easy_install.generate_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... interpreter='py', relative_paths=interpreter_dir)
+ >>> cat(py_path)
+ #!/usr/bin/python2.4 -S
+ <BLANKLINE>
+ import os
+ <BLANKLINE>
+ join = os.path.join
+ base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
+ base = os.path.dirname(base)
+ <BLANKLINE>
+ import os
+ import sys
+ <BLANKLINE>
+ argv = [sys.executable] + sys.argv[1:]
+ environ = os.environ.copy()
+ path = join(base, 'parts/interpreter')
+ if environ.get('PYTHONPATH'):
+ path = os.pathsep.join([path, environ['PYTHONPATH']])
+ environ['PYTHONPATH'] = path
+ os.execve(sys.executable, argv, environ)
+
+For site.py, we again show only the pertinent parts. Notice that the egg
+paths join a base to a path, as with the use of this argument in the
+``scripts`` function.
+
+ >>> sys.stdout.write('#\n'); cat(site_path) # doctest: +ELLIPSIS
+ #...
+ def addsitepackages(known_paths):
+ paths = []
+ <BLANKLINE>
+ import os
+ <BLANKLINE>
+ join = os.path.join
+ base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
+ base = os.path.dirname(base)
+ base = os.path.dirname(base)
+ paths[0:0] = [ # eggs
+ join(base, 'eggs/demo-0.3-pyN.N.egg'),
+ join(base, 'eggs/demoneeded-1.1-pyN.N.egg')
+ ]...
+
+The paths resolve in practice as you would expect.
+
+ >>> print call_py(interpreter_path,
+ ... "import sys, pprint; pprint.pprint(sys.path)")
+ ... # doctest: +ELLIPSIS
+ ['',
+ '/interpreter/eggs/demo-0.3-py2.4.egg',
+ '/interpreter/eggs/demoneeded-1.1-py2.4.egg',
+ '/interpreter/parts/interpreter',
+ ...]
+ <BLANKLINE>
+
+The ``extra_paths`` argument affects the path in site.py. Notice that
+/interpreter/other is added after the eggs.
+
+ >>> reset_interpreter()
+ >>> mkdir(interpreter_dir, 'other')
+ >>> generated = zc.buildout.easy_install.generate_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... interpreter='py', extra_paths=[join(interpreter_dir, 'other')])
+ >>> sys.stdout.write('#\n'); cat(site_path) # doctest: +ELLIPSIS
+ #...
+ def addsitepackages(known_paths):
+ paths = []
+ paths[0:0] = [ # eggs
+ '/interpreter/eggs/demo-0.3-pyN.N.egg',
+ '/interpreter/eggs/demoneeded-1.1-pyN.N.egg',
+ '/interpreter/other'
+ ]...
+
+ >>> print call_py(interpreter_path,
+ ... "import sys, pprint; pprint.pprint(sys.path)")
+ ... # doctest: +ELLIPSIS
+ ['',
+ '/interpreter/eggs/demo-0.3-pyN.N.egg',
+ '/interpreter/eggs/demoneeded-1.1-pyN.N.egg',
+ '/interpreter/other',
+ '/interpreter/parts/interpreter',
+ ...]
+ <BLANKLINE>
+
+The ``generate_scripts`` function: using site-packages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``generate_scripts`` function supports including site packages. This has
+some advantages and some serious dangers.
+
+A typical reason to include site-packages is that it is easier to
+install one or more dependencies in your Python than it is with
+buildbot. Some packages, such as lxml or Python PostgreSQL integration,
+have dependencies that can be much easier to build and/or install using
+other mechanisms, such as your operating system's package manager. By
+installing some core packages into your Python's site-packages, this can
+significantly simplify some application installations.
+
+However, doing this has a significant danger. One of the primary goals
+of buildout is to provide repeatability. Some packages (one of the
+better known Python openid packages, for instance) change their behavior
+depending on what packages are available. If Python curl bindings are
+available, these may be preferred by the library. If a certain XML
+package is installed, it may be preferred by the library. These hidden
+choices may cause small or large behavior differences. The fact that
+they can be rarely encountered can actually make it worse: you forget
+that this might be a problem, and debugging the differences can be
+difficult. If you allow site-packages to be included in your buildout,
+and the Python you use is not managed precisely by your application (for
+instance, it is a system Python), you open yourself up to these
+possibilities. Don't be unaware of the dangers.
+
+That explained, let's see how it works. Unfortunately, because of how
+setuptools namespace packages are implemented differently for operating
+system packages (debs or rpms) and normal installation, there's a tricky
+dance.
+
+ >>> reset_interpreter()
+ >>> generated = zc.buildout.easy_install.generate_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... interpreter='py', add_site_packages=True)
+ >>> sys.stdout.write('#\n'); cat(site_path)
+ ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
+ #...
+ def addsitepackages(known_paths):
+ paths = [ # These are the underlying Python's site-packages.
+ '...']
+ sys.path[0:0] = paths
+ known_paths.update([os.path.normcase(os.path.abspath(p)) for p in paths])
+ try:
+ import pkg_resources
+ except ImportError:
+ # No namespace packages in sys.path; no fixup needed.
+ pkg_resources = None
+ paths[0:0] = [ # eggs
+ '/interpreter/eggs/demo-0.3-pyN.N.egg',
+ '/interpreter/eggs/demoneeded-1.1-pyN.N.egg'
+ ]
+ # Process all dirs. Look for .pth files. If they exist, defer
+ # processing "import" varieties.
+ dotpth = os.extsep + "pth"
+ deferred = []
+ for path in reversed(paths):
+ # Duplicating addsitedir.
+ sitedir, sitedircase = makepath(path)
+ if not sitedircase in known_paths and os.path.exists(sitedir):
+ sys.path.insert(0, sitedir)
+ known_paths.add(sitedircase)
+ try:
+ names = os.listdir(sitedir)
+ except os.error:
+ continue
+ names = [name for name in names if name.endswith(dotpth)]
+ names.sort()
+ for name in names:
+ # Duplicating addpackage.
+ fullname = os.path.join(sitedir, name)
+ try:
+ f = open(fullname, "rU")
+ except IOError:
+ continue
+ try:
+ for line in f:
+ if line.startswith("#"):
+ continue
+ if (line.startswith("import ") or
+ line.startswith("import ")):
+ # This line is supposed to be executed. It
+ # might be a setuptools namespace package
+ # installed with a system package manager.
+ # Defer this so we can process egg namespace
+ # packages first, or else the eggs with the same
+ # namespace will be ignored.
+ deferred.append((sitedir, name, fullname, line))
+ continue
+ line = line.rstrip()
+ dir, dircase = makepath(sitedir, line)
+ if not dircase in known_paths and os.path.exists(dir):
+ sys.path.append(dir)
+ known_paths.add(dircase)
+ finally:
+ f.close()
+ if pkg_resources is not None:
+ # There may be namespace packages in sys.path. This is much faster
+ # than importing pkg_resources after the sys.path has a large number
+ # of eggs.
+ for p in sys.path:
+ pkg_resources.fixup_namespace_packages(p)
+ # Process "import ..." .pth lines.
+ for sitedir, name, fullname, line in deferred:
+ # Note that some lines--such as the one setuptools writes for
+ # namespace packages--expect some or all of sitedir, name, and
+ # fullname to be present in the frame locals, as it is in
+ # ``addpackage``.
+ try:
+ exec line
+ except:
+ print "Error in %s" % (fullname,)
+ raise
+ global addsitepackages
+ addsitepackages = original_addsitepackages
+ return known_paths
+ <BLANKLINE>
+ buildout_addsitepackages = addsitepackages
+ <BLANKLINE>
+ def original_addsitepackages(known_paths):...
+
+As you can see, the script now first tries to import pkg_resources. If it
+exists, then we need to process egg files specially to look for namespace
+packages there *before* we process process lines in .pth files that use the
+"import" feature--lines that might be part of the setuptools namespace
+package implementation for system packages, as mentioned above, and that
+must come after processing egg namespaces.
+
+Here's an example of the new script in use. Other documents and tests in
+this package give the feature a more thorough workout, but this should
+give you an idea of the feature.
+
+ >>> res = call_py(interpreter_path, "import sys; print sys.path")
+ >>> print res # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
+ ['',
+ '/interpreter/eggs/demo-0.3-py2.4.egg',
+ '/interpreter/eggs/demoneeded-1.1-py2.4.egg',
+ '...',
+ '/interpreter/parts/interpreter',
+ ...]
+ <BLANKLINE>
+
+The clean_paths gathered earlier is a subset of this full list of paths.
+
+ >>> full_paths = eval(res.strip())
+ >>> len(clean_paths) < len(full_paths)
+ True
+ >>> set(os.path.normpath(p) for p in clean_paths).issubset(
+ ... os.path.normpath(p) for p in full_paths)
+ True
+
+The ``exec_sitecustomize`` argument does the same thing for the
+sitecustomize module--it allows you to include the code from the
+sitecustomize module in the underlying Python if you set the argument to
+True. The z3c.recipe.scripts package sets up the full environment necessary
+to demonstrate this piece.
+
+The ``generate_scripts`` function: writing scripts for entry points
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+All of the examples so far for this function have been creating
+interpreters. The function can also write scripts for entry
+points. They are almost identical to the scripts that we saw for the
+``scripts`` function except that they ``import site`` after setting the
+sys.path to include our custom site.py and sitecustomize.py files. These
+files then initialize the Python environment as we have already seen. Let's
+see a simple example.
+
+ >>> reset_interpreter()
+ >>> generated = zc.buildout.easy_install.generate_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... reqs=['demo'])
+
+As before, in Windows, 2 files are generated for each script. A script
+file, ending in '-script.py', and an exe file that allows the script
+to be invoked directly without having to specify the Python
+interpreter and without having to provide a '.py' suffix. This is in addition
+to the site.py and sitecustomize.py files that are generated as with our
+interpreter examples above.
+
+ >>> if sys.platform == 'win32':
+ ... demo_path = os.path.join(interpreter_bin_dir, 'demo-script.py')
+ ... expected = [sitecustomize_path,
+ ... site_path,
+ ... os.path.join(interpreter_bin_dir, 'demo.exe'),
+ ... demo_path]
+ ... else:
+ ... demo_path = os.path.join(interpreter_bin_dir, 'demo')
+ ... expected = [sitecustomize_path, site_path, demo_path]
+ ...
+ >>> assert generated == expected, repr((generated, expected))
+
+The demo script runs the entry point defined in the demo egg:
+
+ >>> cat(demo_path) # doctest: +NORMALIZE_WHITESPACE
+ #!/usr/local/bin/python2.4 -S
+ <BLANKLINE>
+ import sys
+ sys.path[0:0] = [
+ '/interpreter/parts/interpreter',
+ ]
+ <BLANKLINE>
+ <BLANKLINE>
+ import site # imports custom buildbot-generated site.py
+ <BLANKLINE>
+ import eggrecipedemo
+ <BLANKLINE>
+ if __name__ == '__main__':
+ eggrecipedemo.main()
+
+ >>> demo_call = join(interpreter_bin_dir, 'demo')
+ >>> if sys.platform == 'win32':
+ ... demo_call = '"%s"' % demo_call
+ >>> print system(demo_call)
+ 3 1
+ <BLANKLINE>
+
+There are a few differences from the ``scripts`` function. First, the
+``reqs`` argument (an iterable of string requirements or entry point
+tuples) is a keyword argument here. We see that in the example above.
+Second, the ``arguments`` argument is now named ``script_arguments`` to
+try and clarify that it does not affect interpreters. While the
+``initialization`` argument continues to affect both the interpreters
+and the entry point scripts, if you have initialization that is only
+pertinent to the entry point scripts, you can use the
+``script_initialization`` argument.
+
+Let's see ``script_arguments`` and ``script_initialization`` in action.
+
+ >>> reset_interpreter()
+ >>> generated = zc.buildout.easy_install.generate_scripts(
+ ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
+ ... reqs=['demo'], script_arguments='1, 2',
+ ... script_initialization='import os\nos.chdir("foo")')
+
+ >>> cat(demo_path) # doctest: +NORMALIZE_WHITESPACE
+ #!/usr/local/bin/python2.4 -S
+ import sys
+ sys.path[0:0] = [
+ '/interpreter/parts/interpreter',
+ ]
+ <BLANKLINE>
+ import site # imports custom buildbot-generated site.py
+ import os
+ os.chdir("foo")
+ <BLANKLINE>
+ import eggrecipedemo
+ <BLANKLINE>
+ if __name__ == '__main__':
+ eggrecipedemo.main(1, 2)
+
Handling custom build options for extensions provided in source distributions
-----------------------------------------------------------------------------
Modified: zc.buildout/branches/gary-4/src/zc/buildout/testing.py
===================================================================
--- zc.buildout/branches/gary-4/src/zc/buildout/testing.py 2010-02-10 22:09:17 UTC (rev 108915)
+++ zc.buildout/branches/gary-4/src/zc/buildout/testing.py 2010-02-11 01:04:30 UTC (rev 108916)
@@ -28,6 +28,7 @@
import subprocess
import sys
import tempfile
+import textwrap
import threading
import time
import urllib2
@@ -105,6 +106,16 @@
e.close()
return result
+def call_py(interpreter, cmd, flags=None):
+ if sys.platform == 'win32':
+ args = ['"%s"' % arg for arg in (interpreter, flags, cmd) if arg]
+ args.insert(-1, '"-c"')
+ return system('"%s"' % ' '.join(args))
+ else:
+ cmd = repr(cmd)
+ return system(
+ ' '.join(arg for arg in (interpreter, flags, '-c', cmd) if arg))
+
def get(url):
return urllib2.urlopen(url).read()
@@ -116,7 +127,11 @@
args = [zc.buildout.easy_install._safe_arg(arg)
for arg in args]
args.insert(0, '-q')
- args.append(dict(os.environ, PYTHONPATH=setuptools_location))
+ env = dict(os.environ)
+ if executable == sys.executable:
+ env['PYTHONPATH'] = setuptools_location
+ # else pass an executable that has setuptools! See testselectingpython.py.
+ args.append(env)
here = os.getcwd()
try:
@@ -135,6 +150,11 @@
def bdist_egg(setup, executable, dest):
_runsetup(setup, executable, 'bdist_egg', '-d', dest)
+def sys_install(setup, dest):
+ _runsetup(setup, sys.executable, 'install', '--install-purelib', dest,
+ '--record', os.path.join(dest, '__added_files__'),
+ '--single-version-externally-managed')
+
def find_python(version):
e = os.environ.get('PYTHON%s' % version)
if e is not None:
@@ -202,6 +222,24 @@
time.sleep(0.01)
raise ValueError('Timed out waiting for: '+label)
+def make_buildout():
+ # Create a basic buildout.cfg to avoid a warning from buildout:
+ open('buildout.cfg', 'w').write(
+ "[buildout]\nparts =\n"
+ )
+ # Use the buildout bootstrap command to create a buildout
+ zc.buildout.buildout.Buildout(
+ 'buildout.cfg',
+ [('buildout', 'log-level', 'WARNING'),
+ # trick bootstrap into putting the buildout develop egg
+ # in the eggs dir.
+ ('buildout', 'develop-eggs-directory', 'eggs'),
+ ]
+ ).bootstrap([])
+ # Create the develop-eggs dir, which didn't get created the usual
+ # way due to the trick above:
+ os.mkdir('develop-eggs')
+
def buildoutSetUp(test):
test.globs['__tear_downs'] = __tear_downs = []
@@ -255,34 +293,48 @@
sample = tmpdir('sample-buildout')
os.chdir(sample)
+ make_buildout()
- # Create a basic buildout.cfg to avoid a warning from buildout:
- open('buildout.cfg', 'w').write(
- "[buildout]\nparts =\n"
- )
-
- # Use the buildout bootstrap command to create a buildout
- zc.buildout.buildout.Buildout(
- 'buildout.cfg',
- [('buildout', 'log-level', 'WARNING'),
- # trick bootstrap into putting the buildout develop egg
- # in the eggs dir.
- ('buildout', 'develop-eggs-directory', 'eggs'),
- ]
- ).bootstrap([])
-
-
-
- # Create the develop-eggs dir, which didn't get created the usual
- # way due to the trick above:
- os.mkdir('develop-eggs')
-
def start_server(path):
port, thread = _start_server(path, name=path)
url = 'http://localhost:%s/' % port
register_teardown(lambda: stop_server(url, thread))
return url
+ def make_py(initialization=''):
+ """Returns paths to new executable and to its site-packages.
+ """
+ buildout = tmpdir('executable_buildout')
+ site_packages_dir = os.path.join(buildout, 'site-packages')
+ mkdir(site_packages_dir)
+ old_wd = os.getcwd()
+ os.chdir(buildout)
+ make_buildout()
+ initialization = '\n'.join(
+ ' ' + line for line in initialization.split('\n'))
+ install_develop(
+ 'zc.recipe.egg', os.path.join(buildout, 'develop-eggs'))
+ install_develop(
+ 'z3c.recipe.scripts', os.path.join(buildout, 'develop-eggs'))
+ write('buildout.cfg', textwrap.dedent('''\
+ [buildout]
+ parts = py
+
+ [py]
+ recipe = z3c.recipe.scripts
+ interpreter = py
+ initialization =
+ %(initialization)s
+ extra-paths = %(site-packages)s
+ eggs = setuptools
+ ''') % {
+ 'initialization': initialization,
+ 'site-packages': site_packages_dir})
+ system(os.path.join(buildout, 'bin', 'buildout'))
+ os.chdir(old_wd)
+ return (
+ os.path.join(buildout, 'bin', 'py'), site_packages_dir)
+
test.globs.update(dict(
sample_buildout = sample,
ls = ls,
@@ -293,6 +345,7 @@
tmpdir = tmpdir,
write = write,
system = system,
+ call_py = call_py,
get = get,
cd = (lambda *path: os.chdir(os.path.join(*path))),
join = os.path.join,
@@ -301,6 +354,7 @@
start_server = start_server,
buildout = os.path.join(sample, 'bin', 'buildout'),
wait_until = wait_until,
+ make_py = make_py
))
zc.buildout.easy_install.prefer_final(prefer_final)
Modified: zc.buildout/branches/gary-4/src/zc/buildout/tests.py
===================================================================
--- zc.buildout/branches/gary-4/src/zc/buildout/tests.py 2010-02-10 22:09:17 UTC (rev 108915)
+++ zc.buildout/branches/gary-4/src/zc/buildout/tests.py 2010-02-11 01:04:30 UTC (rev 108916)
@@ -53,6 +53,7 @@
>>> ls('develop-eggs')
- foo.egg-link
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
"""
@@ -84,6 +85,7 @@
>>> ls('develop-eggs')
- foo.egg-link
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
>>> print system(join('bin', 'buildout')+' -vvv'), # doctest: +ELLIPSIS
@@ -668,6 +670,7 @@
>>> ls('develop-eggs')
- foox.egg-link
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
Create another:
@@ -692,6 +695,7 @@
>>> ls('develop-eggs')
- foox.egg-link
- fooy.egg-link
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
Remove one:
@@ -709,6 +713,7 @@
>>> ls('develop-eggs')
- fooy.egg-link
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
Remove the other:
@@ -723,6 +728,7 @@
All gone
>>> ls('develop-eggs')
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
'''
@@ -797,6 +803,7 @@
... + join(sample_buildout, 'eggs'))
>>> ls('develop-eggs')
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
>>> ls('eggs') # doctest: +ELLIPSIS
@@ -1769,6 +1776,233 @@
1 2
"""
+def versions_section_ignored_for_dependency_in_favor_of_site_packages():
+ r"""
+This is a test for a bugfix.
+
+The error showed itself when at least two dependencies were in a shared
+location like site-packages, and the first one met the "versions" setting. The
+first dependency would be added, but subsequent dependencies from the same
+location (e.g., site-packages) would use the version of the package found in
+the shared location, ignoring the version setting.
+
+We begin with a Python that has demoneeded version 1.1 installed and a
+demo version 0.3, all in a site-packages-like shared directory. We need
+to create this. ``eggrecipedemo.main()`` shows the number after the dot
+(that is, ``X`` in ``1.X``), for the demo package and the demoneeded
+package, so this demonstrates that our Python does in fact have demo
+version 0.3 and demoneeded version 1.1.
+
+ >>> py_path = make_py_with_system_install(make_py, sample_eggs)
+ >>> print call_py(
+ ... py_path,
+ ... "import tellmy.version; print tellmy.version.__version__"),
+ 1.1
+
+Now here's a setup that would expose the bug, using the
+zc.buildout.easy_install API.
+
+ >>> example_dest = tmpdir('example_dest')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['tellmy.version'], example_dest, links=[sample_eggs],
+ ... executable=py_path,
+ ... index=None,
+ ... versions={'tellmy.version': '1.0'})
+ >>> for dist in workingset:
+ ... res = str(dist)
+ ... if res.startswith('tellmy.version'):
+ ... print res
+ ... break
+ tellmy.version 1.0
+
+Before the bugfix, the desired tellmy.version distribution would have
+been blocked the one in site-packages.
+"""
+
+def handle_namespace_package_in_both_site_packages_and_buildout_eggs():
+ r"""
+If you have the same namespace package in both site-packages and in
+buildout, we need to be very careful that faux-Python-executables and
+scripts generated by easy_install.generate_scripts correctly combine the two.
+We show this with the local recipe that uses the function, z3c.recipe.scripts.
+
+To demonstrate this, we will create three packages: tellmy.version 1.0,
+tellmy.version 1.1, and tellmy.fortune 1.0. tellmy.version 1.1 is installed.
+
+ >>> py_path = make_py_with_system_install(make_py, sample_eggs)
+ >>> print call_py(
+ ... py_path,
+ ... "import tellmy.version; print tellmy.version.__version__")
+ 1.1
+ <BLANKLINE>
+
+Now we will create a buildout that creates a script and a faux-Python script.
+We want to see that both can successfully import the specified versions of
+tellmy.version and tellmy.fortune.
+
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... parts = eggs
+ ... find-links = %(link_server)s
+ ...
+ ... [primed_python]
+ ... executable = %(py_path)s
+ ...
+ ... [eggs]
+ ... recipe = z3c.recipe.scripts
+ ... python = primed_python
+ ... interpreter = py
+ ... add-site-packages = true
+ ... eggs = tellmy.version == 1.0
+ ... tellmy.fortune == 1.0
+ ... demo
+ ... script-initialization =
+ ... import tellmy.version
+ ... print tellmy.version.__version__
+ ... import tellmy.fortune
+ ... print tellmy.fortune.__version__
+ ... ''' % globals())
+
+ >>> print system(buildout)
+ Installing eggs.
+ Getting distribution for 'tellmy.version==1.0'.
+ Got tellmy.version 1.0.
+ Getting distribution for 'tellmy.fortune==1.0'.
+ Got tellmy.fortune 1.0.
+ Getting distribution for 'demo'.
+ Got demo 0.4c1.
+ Getting distribution for 'demoneeded'.
+ Got demoneeded 1.2c1.
+ Generated script '/sample-buildout/bin/demo'.
+ Generated interpreter '/sample-buildout/bin/py'.
+ <BLANKLINE>
+
+Finally, we are ready for the actual test. Prior to the bug fix that
+this tests, the results of both calls below was the following::
+
+ 1.1
+ Traceback (most recent call last):
+ ...
+ ImportError: No module named fortune
+ <BLANKLINE>
+
+In other words, we got the site-packages version of tellmy.version, and
+we could not import tellmy.fortune at all. The following are the correct
+results for the interpreter and for the script.
+
+ >>> print call_py(
+ ... join('bin', 'py'),
+ ... "import tellmy.version; " +
+ ... "print tellmy.version.__version__; " +
+ ... "import tellmy.fortune; " +
+ ... "print tellmy.fortune.__version__") # doctest: +ELLIPSIS
+ 1.0
+ 1.0...
+
+ >>> print system(join('bin', 'demo'))
+ 1.0
+ 1.0
+ 4 2
+ <BLANKLINE>
+ """
+
+def handle_sys_path_version_hack():
+ r"""
+This is a test for a bugfix.
+
+If you use a Python that has a different version of one of your
+dependencies, and the new package tries to do sys.path tricks in the
+setup.py to get a __version__, and it uses namespace packages, the older
+package will be loaded first, making the setup version the wrong number.
+While very arguably packages simply shouldn't do this, some do, and we
+don't want buildout to fall over when they do.
+
+To demonstrate this, we will need to create a distribution that has one of
+these unpleasant tricks, and a Python that has an older version installed.
+
+ >>> py_path, site_packages_path = make_py()
+ >>> for version in ('1.0', '1.1'):
+ ... tmp = tempfile.mkdtemp()
+ ... try:
+ ... write(tmp, 'README.txt', '')
+ ... mkdir(tmp, 'src')
+ ... mkdir(tmp, 'src', 'tellmy')
+ ... write(tmp, 'src', 'tellmy', '__init__.py',
+ ... "__import__("
+ ... "'pkg_resources').declare_namespace(__name__)\n")
+ ... mkdir(tmp, 'src', 'tellmy', 'version')
+ ... write(tmp, 'src', 'tellmy', 'version',
+ ... '__init__.py', '__version__=%r\n' % version)
+ ... write(
+ ... tmp, 'setup.py',
+ ... "from setuptools import setup\n"
+ ... "import sys\n"
+ ... "sys.path.insert(0, 'src')\n"
+ ... "from tellmy.version import __version__\n"
+ ... "setup(\n"
+ ... " name='tellmy.version',\n"
+ ... " package_dir = {'': 'src'},\n"
+ ... " packages = ['tellmy', 'tellmy.version'],\n"
+ ... " install_requires = ['setuptools'],\n"
+ ... " namespace_packages=['tellmy'],\n"
+ ... " zip_safe=True, version=__version__,\n"
+ ... " author='bob', url='bob', author_email='bob')\n"
+ ... )
+ ... zc.buildout.testing.sdist(tmp, sample_eggs)
+ ... if version == '1.0':
+ ... # We install the 1.0 version in site packages the way a
+ ... # system packaging system (debs, rpms) would do it.
+ ... zc.buildout.testing.sys_install(tmp, site_packages_path)
+ ... finally:
+ ... shutil.rmtree(tmp)
+ >>> print call_py(
+ ... py_path,
+ ... "import tellmy.version; print tellmy.version.__version__")
+ 1.0
+ <BLANKLINE>
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... parts = eggs
+ ... find-links = %(sample_eggs)s
+ ...
+ ... [primed_python]
+ ... executable = %(py_path)s
+ ...
+ ... [eggs]
+ ... recipe = zc.recipe.egg:eggs
+ ... python = primed_python
+ ... eggs = tellmy.version == 1.1
+ ... ''' % globals())
+
+Before the bugfix, running this buildout would generate this error:
+
+ Installing eggs.
+ Getting distribution for 'tellmy.version==1.1'.
+ Installing tellmy.version 1.1
+ Caused installation of a distribution:
+ tellmy.version 1.0
+ with a different version.
+ Got None.
+ While:
+ Installing eggs.
+ Error: There is a version conflict.
+ We already have: tellmy.version 1.0
+ <BLANKLINE>
+
+The bugfix was simply to add Python's "-S" option when calling
+easyinstall (see zc.buildout.easy_install.Installer._call_easy_install).
+Now the install works correctly, as seen here.
+
+ >>> print system(buildout)
+ Installing eggs.
+ Getting distribution for 'tellmy.version==1.1'.
+ Got tellmy.version 1.1.
+ <BLANKLINE>
+
+ """
+
if sys.version_info > (2, 4):
def test_exit_codes():
"""
@@ -2367,6 +2601,7 @@
>>> ls('develop-eggs')
- foo.egg-link
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
"""
@@ -2654,6 +2889,44 @@
######################################################################
+def make_py_with_system_install(make_py, sample_eggs):
+ from zc.buildout.testing import write, mkdir
+ py_path, site_packages_path = make_py()
+ for pkg, version in (('version', '1.0'), ('version', '1.1'),
+ ('fortune', '1.0')):
+ tmp = tempfile.mkdtemp()
+ try:
+ write(tmp, 'README.txt', '')
+ mkdir(tmp, 'src')
+ mkdir(tmp, 'src', 'tellmy')
+ write(tmp, 'src', 'tellmy', '__init__.py',
+ "__import__("
+ "'pkg_resources').declare_namespace(__name__)\n")
+ mkdir(tmp, 'src', 'tellmy', pkg)
+ write(tmp, 'src', 'tellmy', pkg,
+ '__init__.py', '__version__=%r\n' % version)
+ write(
+ tmp, 'setup.py',
+ "from setuptools import setup\n"
+ "setup(\n"
+ " name='tellmy.%(pkg)s',\n"
+ " package_dir = {'': 'src'},\n"
+ " packages = ['tellmy', 'tellmy.%(pkg)s'],\n"
+ " install_requires = ['setuptools'],\n"
+ " namespace_packages=['tellmy'],\n"
+ " zip_safe=True, version=%(version)r,\n"
+ " author='bob', url='bob', author_email='bob')\n"
+ % locals()
+ )
+ zc.buildout.testing.sdist(tmp, sample_eggs)
+ if pkg == 'version' and version == '1.1':
+ # We install the 1.1 version in site packages the way a
+ # system packaging system (debs, rpms) would do it.
+ zc.buildout.testing.sys_install(tmp, site_packages_path)
+ finally:
+ shutil.rmtree(tmp)
+ return py_path
+
def create_sample_eggs(test, executable=sys.executable):
write = test.globs['write']
dest = test.globs['sample_eggs']
@@ -2776,6 +3049,7 @@
test.globs['sample_eggs'])
test.globs['update_extdemo'] = lambda : add_source_dist(test, 1.5)
zc.buildout.testing.install_develop('zc.recipe.egg', test)
+ zc.buildout.testing.install_develop('z3c.recipe.scripts', test)
egg_parse = re.compile('([0-9a-zA-Z_.]+)-([0-9a-zA-Z_.]+)-py(\d[.]\d).egg$'
).match
@@ -2934,6 +3208,10 @@
(re.compile('[-d] setuptools-\S+[.]egg'), 'setuptools.egg'),
(re.compile(r'\\[\\]?'), '/'),
(re.compile(r'\#!\S+\bpython\S*'), '#!/usr/bin/python'),
+ # Normalize generate_script's Windows interpreter to UNIX:
+ (re.compile(r'\nimport subprocess\n'), '\n'),
+ (re.compile('subprocess\\.call\\(argv, env=environ\\)'),
+ 'os.execve(sys.executable, argv, environ)'),
]+(sys.version_info < (2, 5) and [
(re.compile('.*No module named runpy.*', re.S), ''),
(re.compile('.*usage: pdb.py scriptfile .*', re.S), ''),
Modified: zc.buildout/branches/gary-4/src/zc/buildout/testselectingpython.py
===================================================================
--- zc.buildout/branches/gary-4/src/zc/buildout/testselectingpython.py 2010-02-10 22:09:17 UTC (rev 108915)
+++ zc.buildout/branches/gary-4/src/zc/buildout/testselectingpython.py 2010-02-11 01:04:30 UTC (rev 108916)
@@ -11,7 +11,7 @@
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
-import os, re, sys, unittest
+import os, re, subprocess, sys, textwrap, unittest
from zope.testing import doctest, renormalizing
import zc.buildout.tests
import zc.buildout.testing
@@ -42,6 +42,33 @@
def multi_python(test):
other_executable = zc.buildout.testing.find_python(other_version)
+ command = textwrap.dedent('''\
+ try:
+ import setuptools
+ except ImportError:
+ import sys
+ sys.exit(1)
+ ''')
+ if subprocess.call([other_executable, '-c', command],
+ env=os.environ):
+ # the other executable does not have setuptools. Get setuptools.
+ # We will do this using the same tools we are testing, for better or
+ # worse. Alternatively, we could try using bootstrap.
+ executable_dir = test.globs['tmpdir']('executable_dir')
+ executable_parts = os.path.join(executable_dir, 'parts')
+ test.globs['mkdir'](executable_parts)
+ ws = zc.buildout.easy_install.install(
+ ['setuptools'], executable_dir,
+ index='http://www.python.org/pypi/',
+ always_unzip=True, executable=other_executable)
+ zc.buildout.easy_install.generate_scripts(
+ executable_dir, ws, other_executable, executable_parts,
+ reqs=['setuptools'], interpreter='py')
+ original_executable = other_executable
+ other_executable = os.path.join(executable_dir, 'py')
+ assert not subprocess.call(
+ [other_executable, '-c', command], env=os.environ), (
+ 'test set up failed')
sample_eggs = test.globs['tmpdir']('sample_eggs')
os.mkdir(os.path.join(sample_eggs, 'index'))
test.globs['sample_eggs'] = sample_eggs
Modified: zc.buildout/branches/gary-4/src/zc/buildout/update.txt
===================================================================
--- zc.buildout/branches/gary-4/src/zc/buildout/update.txt 2010-02-10 22:09:17 UTC (rev 108915)
+++ zc.buildout/branches/gary-4/src/zc/buildout/update.txt 2010-02-11 01:04:30 UTC (rev 108916)
@@ -81,6 +81,7 @@
Our buildout script has been updated to use the new eggs:
>>> cat(sample_buildout, 'bin', 'buildout')
+ ... # doctest: +NORMALIZE_WHITESPACE
#!/usr/local/bin/python2.4
<BLANKLINE>
import sys
Modified: zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/README.txt
===================================================================
--- zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/README.txt 2010-02-10 22:09:17 UTC (rev 108915)
+++ zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/README.txt 2010-02-11 01:04:30 UTC (rev 108916)
@@ -154,6 +154,8 @@
interpreter
The name of a script to generate that allows access to a Python
interpreter that has the path set based on the eggs installed.
+ (See the ``z3c.recipe.scripts`` recipe for a more full-featured
+ interpreter.)
extra-paths
Extra paths to include in a generated script.
@@ -577,7 +579,7 @@
- demo
- other
- >>> cat(sample_buildout, 'bin', 'other')
+ >>> cat(sample_buildout, 'bin', 'other') # doctest: +NORMALIZE_WHITESPACE
#!/usr/local/bin/python2.4
<BLANKLINE>
import sys
@@ -640,3 +642,4 @@
Uninstalling bigdemo.
Installing demo.
Generated script '/sample-buildout/bin/foo'.
+
Modified: zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/api.txt
===================================================================
--- zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/api.txt 2010-02-10 22:09:17 UTC (rev 108915)
+++ zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/api.txt 2010-02-11 01:04:30 UTC (rev 108916)
@@ -117,6 +117,7 @@
extras = other
find-links = http://localhost:27071/
index = http://localhost:27071/index
+ python = buildout
recipe = sample
If we use the extra-paths option:
Modified: zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/custom.txt
===================================================================
--- zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/custom.txt 2010-02-10 22:09:17 UTC (rev 108915)
+++ zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/custom.txt 2010-02-11 01:04:30 UTC (rev 108916)
@@ -150,6 +150,7 @@
>>> ls(sample_buildout, 'develop-eggs')
d extdemo-1.4-py2.4-unix-i686.egg
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
Note that no scripts or dependencies are installed. To install
@@ -231,6 +232,7 @@
>>> ls(sample_buildout, 'develop-eggs')
- demo.egg-link
d extdemo-1.4-py2.4-unix-i686.egg
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
But if we run the buildout in the default on-line and newest modes, we
@@ -248,6 +250,7 @@
- demo.egg-link
d extdemo-1.4-py2.4-linux-i686.egg
d extdemo-1.5-py2.4-linux-i686.egg
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
Controlling the version used
@@ -287,6 +290,7 @@
>>> ls(sample_buildout, 'develop-eggs')
- demo.egg-link
d extdemo-1.4-py2.4-linux-i686.egg
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
@@ -553,6 +557,7 @@
>>> ls('develop-eggs')
- demo.egg-link
- extdemo.egg-link
+ - z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link
and the extdemo now has a built extension:
Modified: zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/egg.py
===================================================================
--- zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/egg.py 2010-02-10 22:09:17 UTC (rev 108915)
+++ zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/egg.py 2010-02-11 01:04:30 UTC (rev 108916)
@@ -19,11 +19,12 @@
import logging, os, re, zipfile
import zc.buildout.easy_install
+
class Eggs(object):
def __init__(self, buildout, name, options):
self.buildout = buildout
- self.name = name
+ self.name = self.default_eggs = name
self.options = options
b_options = buildout['buildout']
links = options.get('find-links', b_options['find-links'])
@@ -52,7 +53,7 @@
# verify that this is None, 'true' or 'false'
get_bool(options, 'unzip')
- python = options.get('python', b_options['python'])
+ python = options.setdefault('python', b_options['python'])
options['executable'] = buildout[python]['executable']
def working_set(self, extra=()):
@@ -65,15 +66,16 @@
distributions = [
r.strip()
- for r in options.get('eggs', self.name).split('\n')
+ for r in options.get('eggs', self.default_eggs).split('\n')
if r.strip()]
orig_distributions = distributions[:]
distributions.extend(extra)
- if self.buildout['buildout'].get('offline') == 'true':
+ if b_options.get('offline') == 'true':
ws = zc.buildout.easy_install.working_set(
distributions, options['executable'],
- [options['develop-eggs-directory'], options['eggs-directory']]
+ [options['develop-eggs-directory'],
+ options['eggs-directory']],
)
else:
kw = {}
@@ -85,7 +87,7 @@
index=self.index,
executable=options['executable'],
path=[options['develop-eggs-directory']],
- newest=self.buildout['buildout'].get('newest') == 'true',
+ newest=b_options.get('newest') == 'true',
allow_hosts=self.allow_hosts,
**kw)
@@ -97,16 +99,19 @@
update = install
-class Scripts(Eggs):
+class ScriptBase(Eggs):
+
def __init__(self, buildout, name, options):
- super(Scripts, self).__init__(buildout, name, options)
+ super(ScriptBase, self).__init__(buildout, name, options)
- options['bin-directory'] = buildout['buildout']['bin-directory']
+ b_options = buildout['buildout']
+
+ options['bin-directory'] = b_options['bin-directory']
options['_b'] = options['bin-directory'] # backward compat.
self.extra_paths = [
- os.path.join(buildout['buildout']['directory'], p.strip())
+ os.path.join(b_options['directory'], p.strip())
for p in options.get('extra-paths', '').split('\n')
if p.strip()
]
@@ -115,11 +120,9 @@
relative_paths = options.get(
- 'relative-paths',
- buildout['buildout'].get('relative-paths', 'false')
- )
+ 'relative-paths', b_options.get('relative-paths', 'false'))
if relative_paths == 'true':
- options['buildout-directory'] = buildout['buildout']['directory']
+ options['buildout-directory'] = b_options['directory']
self._relative_paths = options['buildout-directory']
else:
self._relative_paths = ''
@@ -128,12 +131,13 @@
parse_entry_point = re.compile(
'([^=]+)=(\w+(?:[.]\w+)*):(\w+(?:[.]\w+)*)$'
).match
+
def install(self):
reqs, ws = self.working_set()
options = self.options
scripts = options.get('scripts')
- if scripts or scripts is None:
+ if scripts or scripts is None or options.get('interpreter'):
if scripts is not None:
scripts = scripts.split()
scripts = dict([
@@ -157,22 +161,32 @@
name = dist.project_name
if name != 'setuptools' and name not in reqs:
reqs.append(name)
-
- return zc.buildout.easy_install.scripts(
- reqs, ws, options['executable'],
- options['bin-directory'],
- scripts=scripts,
- extra_paths=self.extra_paths,
- interpreter=options.get('interpreter'),
- initialization=options.get('initialization', ''),
- arguments=options.get('arguments', ''),
- relative_paths=self._relative_paths,
- )
-
+ return self._install(reqs, ws, scripts)
return ()
update = install
+ def _install(self, reqs, ws, scripts):
+ # Subclasses implement this.
+ raise NotImplementedError()
+
+
+class Scripts(ScriptBase):
+
+ def _install(self, reqs, ws, scripts):
+ options = self.options
+ return zc.buildout.easy_install.scripts(
+ reqs, ws, options['executable'],
+ options['bin-directory'],
+ scripts=scripts,
+ extra_paths=self.extra_paths,
+ interpreter=options.get('interpreter'),
+ initialization=options.get('initialization', ''),
+ arguments=options.get('arguments', ''),
+ relative_paths=self._relative_paths
+ )
+
+
def get_bool(options, name, default=False):
value = options.get(name)
if not value:
Modified: zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/selecting-python.txt
===================================================================
--- zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/selecting-python.txt 2010-02-10 22:09:17 UTC (rev 108915)
+++ zc.buildout/branches/gary-4/zc.recipe.egg_/src/zc/recipe/egg/selecting-python.txt 2010-02-11 01:04:30 UTC (rev 108916)
@@ -35,7 +35,7 @@
... index = http://www.python.org/pypi/
...
... [python2.4]
- ... executable = %(python23)s
+ ... executable = %(python24)s
...
... [demo]
... recipe = zc.recipe.egg
@@ -43,7 +43,7 @@
... find-links = %(server)s
... python = python2.4
... interpreter = py-demo
- ... """ % dict(server=link_server, python23=other_executable))
+ ... """ % dict(server=link_server, python24=other_executable))
Now, if we run the buildout:
More information about the checkins
mailing list