[Checkins] SVN: zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/ implementation for allowed-eggs-from-site-packages
Gary Poster
gary.poster at canonical.com
Fri Sep 25 20:52:31 EDT 2009
Log message for revision 104562:
implementation for allowed-eggs-from-site-packages
Changed:
U zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/CHANGES.txt
U zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/src/zc/buildout/buildout.py
U zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/src/zc/buildout/buildout.txt
U zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/src/zc/buildout/easy_install.py
U zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/src/zc/buildout/tests.py
U zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/zc.recipe.egg_/src/zc/recipe/egg/egg.py
-=-
Modified: zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/CHANGES.txt
===================================================================
--- zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/CHANGES.txt 2009-09-26 00:51:19 UTC (rev 104561)
+++ zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/CHANGES.txt 2009-09-26 00:52:31 UTC (rev 104562)
@@ -18,6 +18,27 @@
zc.buildout's own buildout.cfg dogfoods this option. This defaults
to 'true', which is very similar to buildout's previous behavior.
+ * Another new option, 'allowed-eggs-from-site-packages', lets you specify
+ a whitelist of project names of eggs that are allowed to come from
+ your Python's site-packages. This lets you more tightly control your use
+ of site-packages.
+
+ This option interacts with the ``include-site-packages`` option in the
+ following ways.
+
+ If ``include-site-packages`` is true, then
+ ``allowed-eggs-from-site-packages`` filters what eggs from site-packages
+ may be chosen. Therefore, if ``allowed-eggs-from-site-packages`` is an
+ empty list, then no eggs from site-packages are chosen, but site-packages
+ will still be included at the end of path lists.
+
+ If ``include-site-packages`` is false, the value of
+ ``allowed-eggs-from-site-packages`` is irrelevant.
+
+ The same pattern holds true for the ``include-site-packages-for-buildout``
+ option, except only the bin/buildout script is affected by that
+ interaction.
+
* A new boolean option, 'include-site-packages-for-buildout', does the
same thing but only for the bin/buildout script. This can be important
for getting recipes and their dependencies without conflicts. This
Modified: zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/src/zc/buildout/buildout.py
===================================================================
--- zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/src/zc/buildout/buildout.py 2009-09-26 00:51:19 UTC (rev 104561)
+++ zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/src/zc/buildout/buildout.py 2009-09-26 00:52:31 UTC (rev 104562)
@@ -114,6 +114,7 @@
_buildout_default_options = _annotate_section({
'allow-hosts': '*',
'allow-picked-versions': 'true',
+ 'allowed-eggs-from-site-packages': '*',
'bin-directory': 'bin',
'develop-eggs-directory': 'develop-eggs',
'eggs-directory': 'eggs',
@@ -293,6 +294,12 @@
zc.buildout.easy_install.include_site_packages(
include_site_packages=='true')
+ allowed_eggs_from_site_packages = tuple(
+ name.strip() for name in
+ options['allowed-eggs-from-site-packages'].split('\n'))
+ zc.buildout.easy_install.allowed_eggs_from_site_packages(
+ allowed_eggs_from_site_packages)
+
include_site_packages_for_buildout = options[
'include-site-packages-for-buildout']
if include_site_packages_for_buildout not in ('true', 'false'):
Modified: zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/src/zc/buildout/buildout.txt
===================================================================
--- zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/src/zc/buildout/buildout.txt 2009-09-26 00:51:19 UTC (rev 104561)
+++ zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/src/zc/buildout/buildout.txt 2009-09-26 00:52:31 UTC (rev 104562)
@@ -728,6 +728,8 @@
DEFAULT_VALUE
allow-picked-versions= true
DEFAULT_VALUE
+ allowed-eggs-from-site-packages= *
+ DEFAULT_VALUE
bin-directory= bin
DEFAULT_VALUE
develop= recipes
@@ -2224,6 +2226,7 @@
[buildout]
allow-hosts = *
allow-picked-versions = true
+ allowed-eggs-from-site-packages = *
bin-directory = /sample-buildout/bin
develop-eggs-directory = /sample-buildout/develop-eggs
directory = /sample-buildout
@@ -2282,6 +2285,39 @@
"allow-picked-versions" is "false," instead of picking the best match,
buildout will raise an error. This helps enforce repeatability.
+allowed-eggs-from-site-packages
+ Sometimes you need or want to control what eggs from site-packages are
+ used. The allowed-eggs-from-site-packages option allows you to specify a
+ whitelist of project names that may be included from site-packages. You
+ can use globs to specify the value. It defaults to a single value of '*',
+ indicating that any package may come from site-packages.
+
+ Here's a usage example::
+
+ [buildout]
+ ...
+
+ allowed-eggs-from-site-packages =
+ demo
+ bigdemo
+ zope.*
+
+ This option interacts with the ``include-site-packages`` option in the
+ following ways.
+
+ If ``include-site-packages`` is true, then
+ ``allowed-eggs-from-site-packages`` filters what eggs from site-packages
+ may be chosen. Therefore, if ``allowed-eggs-from-site-packages`` is an
+ empty list, then no eggs from site-packages are chosen, but site-packages
+ will still be included at the end of path lists.
+
+ If ``include-site-packages`` is false, the value of
+ ``allowed-eggs-from-site-packages`` is irrelevant.
+
+ The same pattern holds true for the ``include-site-packages-for-buildout``
+ option, except only the bin/buildout script is affected by that
+ interaction.
+
bin-directory
The directory path where scripts are written. This can be a
relative path, which is interpreted relative to the directory
@@ -2566,6 +2602,7 @@
You can also specify more locations to search for distributions using
the `find-links` option. See its description above.
+
Controlling the installation database
-------------------------------------
Modified: zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/src/zc/buildout/easy_install.py
===================================================================
--- zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/src/zc/buildout/easy_install.py 2009-09-26 00:51:19 UTC (rev 104561)
+++ zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/src/zc/buildout/easy_install.py 2009-09-26 00:52:31 UTC (rev 104562)
@@ -19,6 +19,7 @@
"""
import distutils.errors
+import fnmatch
import glob
import logging
import os
@@ -210,6 +211,7 @@
_allow_picked_versions = True
_always_unzip = False
_include_site_packages = True
+ _allowed_eggs_from_site_packages = ('*',)
def __init__(self,
dest=None,
@@ -222,6 +224,7 @@
versions=None,
use_dependency_links=None,
include_site_packages=None,
+ allowed_eggs_from_site_packages=None,
allow_hosts=('*',)
):
self._dest = dest
@@ -247,6 +250,9 @@
path = (path and path[:] or [])
if include_site_packages is not None:
self._include_site_packages = include_site_packages
+ if allowed_eggs_from_site_packages is not None:
+ self._allowed_eggs_from_site_packages = tuple(
+ allowed_eggs_from_site_packages)
stdlib, self._site_packages = _get_system_packages(executable)
if self._include_site_packages:
path.extend(buildout_and_setuptools_path)
@@ -268,6 +274,21 @@
if versions is not None:
self._versions = versions
+ _allowed_eggs_from_site_packages_regex = None
+ def allow_site_package_egg(self, name):
+ if (not self._include_site_packages or
+ not self._allowed_eggs_from_site_packages):
+ # If the answer is a blanket "no," perform a shortcut.
+ return False
+ if self._allowed_eggs_from_site_packages_regex is None:
+ pattern = '(%s)' % (
+ '|'.join(
+ fnmatch.translate(name)
+ for name in self._allowed_eggs_from_site_packages),
+ )
+ self._allowed_eggs_from_site_packages_regex = re.compile(pattern)
+ return bool(self._allowed_eggs_from_site_packages_regex.match(name))
+
def _satisfied(self, req, source=None):
# We get all distributions that match the given requirement. If we are
# not supposed to include site-packages for the given egg, we also
@@ -279,8 +300,8 @@
# path in our _site_packages.
dists = [dist for dist in self._env[req.project_name] if (
dist in req and (
- self._include_site_packages or
- dist.location not in self._site_packages)
+ dist.location not in self._site_packages or
+ self.allow_site_package_egg(dist.project_name))
)
]
if not dists:
@@ -494,16 +515,14 @@
# .egg-link, such as one for setuptools or zc.buildout installed by
# zc.buildout.buildout.Buildout.bootstrap, can indirectly include a
# path in our _site_packages.
- dists = [dist for dist in index[requirement.project_name]
- if ((dist in requirement)
- and
- (self._include_site_packages or
- dist.location not in self._site_packages)
- and
- ((not source) or
- (dist.precedence == pkg_resources.SOURCE_DIST)
- )
- )
+ dists = [dist for dist in index[requirement.project_name] if (
+ dist in requirement and (
+ dist.location not in self._site_packages or
+ self.allow_site_package_egg(dist.project_name))
+ and (
+ (not source) or
+ (dist.precedence == pkg_resources.SOURCE_DIST))
+ )
]
# If we prefer final dists, filter for final and use the
@@ -890,6 +909,12 @@
Installer._include_site_packages = bool(setting)
return old
+def allowed_eggs_from_site_packages(setting=None):
+ old = Installer._allowed_eggs_from_site_packages
+ if setting is not None:
+ Installer._allowed_eggs_from_site_packages = tuple(setting)
+ return old
+
def use_dependency_links(setting=None):
old = Installer._use_dependency_links
if setting is not None:
@@ -913,10 +938,12 @@
executable=sys.executable, always_unzip=None,
path=None, working_set=None, newest=True, versions=None,
use_dependency_links=None, include_site_packages=None,
- allow_hosts=('*',)):
+ allowed_eggs_from_site_packages=None, allow_hosts=('*',)):
installer = Installer(dest, links, index, executable, always_unzip, path,
newest, versions, use_dependency_links,
include_site_packages=include_site_packages,
+ allowed_eggs_from_site_packages=
+ allowed_eggs_from_site_packages,
allow_hosts=allow_hosts)
return installer.install(specs, working_set)
@@ -925,15 +952,15 @@
links=(), index=None,
executable=sys.executable,
path=None, newest=True, versions=None, include_site_packages=None,
- allow_hosts=('*',)):
+ allowed_eggs_from_site_packages=None, allow_hosts=('*',)):
installer = Installer(dest, links, index, executable, True, path, newest,
versions,
include_site_packages=include_site_packages,
+ allowed_eggs_from_site_packages=
+ allowed_eggs_from_site_packages,
allow_hosts=allow_hosts)
return installer.build(spec, build_ext)
-
-
def _rm(*paths):
for path in paths:
if os.path.isdir(path):
@@ -1024,11 +1051,12 @@
undo.reverse()
[f() for f in undo]
-
-def working_set(specs, executable, path, include_site_packages=None):
+def working_set(specs, executable, path, include_site_packages=None,
+ allowed_eggs_from_site_packages=None):
return install(
specs, None, executable=executable, path=path,
- include_site_packages=include_site_packages)
+ include_site_packages=include_site_packages,
+ allowed_eggs_from_site_packages=allowed_eggs_from_site_packages)
def get_path(working_set, executable, extra_paths=(),
include_site_packages=True):
Modified: zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/src/zc/buildout/tests.py
===================================================================
--- zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/src/zc/buildout/tests.py 2009-09-26 00:51:19 UTC (rev 104561)
+++ zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/src/zc/buildout/tests.py 2009-09-26 00:52:31 UTC (rev 104562)
@@ -3030,6 +3030,249 @@
"""
+def allowed_eggs_from_site_packages():
+ """
+Sometimes you need or want to control what eggs from site-packages are used.
+The allowed-eggs-from-site-packages option allows you to specify a whitelist of
+project names that may be included from site-packages. You can use globs to
+specify the value. It defaults to a single value of '*', indicating that any
+package may come from site-packages.
+
+This option interacts with include-site-packages in the following ways.
+
+If include-site-packages is true, then allowed-eggs-from-site-packages filters
+what eggs from site-packages may be chosen. If allowed-eggs-from-site-packages
+is an empty list, then no eggs from site-packages are chosen, but site-packages
+will still be included at the end of path lists.
+
+If include-site-packages is false, allowed-eggs-from-site-packages is
+irrelevant.
+
+This test shows the interaction with the zc.buildout.easy_install API. Another
+test below (allow_site_package_eggs_option) shows using it with a buildout.cfg.
+
+Our "primed_executable" has the "demoneeded," "other," and "setuptools"
+packages available. We'll simply be asking for "other" here.
+
+ >>> primed_executable = get_executable_with_site_packages()
+
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['other'], example_dest, links=[], executable=primed_executable,
+ ... index=None,
+ ... allowed_eggs_from_site_packages=['demoneeded', 'other'])
+ >>> [dist.project_name for dist in workingset]
+ ['other']
+
+That worked fine. It would work fine for a glob too.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['other'], example_dest, links=[], executable=primed_executable,
+ ... index=None,
+ ... allowed_eggs_from_site_packages=['demoneeded', '?th*'])
+ >>> [dist.project_name for dist in workingset]
+ ['other']
+
+But now let's try again with 'other' not allowed.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['other'], example_dest, links=[], executable=primed_executable,
+ ... index=None,
+ ... allowed_eggs_from_site_packages=['demoneeded'])
+ Traceback (most recent call last):
+ ...
+ MissingDistribution: Couldn't find a distribution for 'other'.
+
+Here's the same, but with an empty list.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['other'], example_dest, links=[], executable=primed_executable,
+ ... index=None,
+ ... allowed_eggs_from_site_packages=[])
+ Traceback (most recent call last):
+ ...
+ MissingDistribution: Couldn't find a distribution for 'other'.
+
+Of course, this doesn't stop us from getting a package from elsewhere. Here,
+we add a link server.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['other'], example_dest, executable=primed_executable,
+ ... links=[link_server], index=link_server+'index/',
+ ... allowed_eggs_from_site_packages=['demoneeded'])
+ >>> [dist.project_name for dist in workingset]
+ ['other']
+ >>> [dist.location for dist in workingset]
+ ['/site-packages-example-install/other-1.0-py2.6.egg']
+
+Finally, here's an example of an interaction we described above: we say that it
+is OK to allow the "other" egg to come from site-packages, but we don't
+include-site-packages.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> rmdir(example_dest)
+ >>> example_dest = tmpdir('site-packages-example-install')
+ >>> workingset = zc.buildout.easy_install.install(
+ ... ['other'], example_dest, links=[], executable=primed_executable,
+ ... index=None, include_site_packages=False,
+ ... allowed_eggs_from_site_packages=['other'])
+ Traceback (most recent call last):
+ ...
+ MissingDistribution: Couldn't find a distribution for 'other'.
+
+ """
+
+def allowed_eggs_from_site_packages_option():
+ """
+As introduced in the previous test, the allowed-eggs-from-site-packages option
+allows you to specify a whitelist of project names that may be included from
+site-packages.
+
+This test shows the option being used in a buildout. We try to limit these
+tests to those that test additional parts of the code beyond those tested in
+the test above.
+
+The buildout defaults to a whitelist of ('*',), or any project name. The
+buildout configuration option defaults are managed separately from the
+zc.buildout.easy_install API defaults, so we show this here.
+
+ >>> from zc.buildout.buildout import Buildout
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... ''')
+ >>> buildout_instance = Buildout('buildout.cfg', ())
+ >>> buildout_instance['buildout']['allowed-eggs-from-site-packages']
+ '*'
+ >>> zc.buildout.easy_install.allowed_eggs_from_site_packages()
+ ('*',)
+
+In the test below, our "primed_executable" has the "demoneeded," "other," and "se
+packages available. We'll simply be asking for "other" here. The default
+value of '*' will allow it. This confirms behaviorally what we saw above.
+
+ >>> primed_executable = get_executable_with_site_packages()
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... parts = eggs
+ ... find-links =
+ ...
+ ... [primed_python]
+ ... executable = %(primed_executable)s
+ ...
+ ... [eggs]
+ ... recipe = zc.recipe.egg:eggs
+ ... python = primed_python
+ ... eggs = other
+ ... ''' % globals())
+
+ >>> print system(primed_executable+" "+buildout)
+ Installing eggs.
+ <BLANKLINE>
+
+Here we explicitly use a "*" for the same result. This also shows that we
+correctly parse a single-line value.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... parts = eggs
+ ... find-links =
+ ... allowed-eggs-from-site-packages = *
+ ...
+ ... [primed_python]
+ ... executable = %(primed_executable)s
+ ...
+ ... [eggs]
+ ... recipe = zc.recipe.egg:eggs
+ ... python = primed_python
+ ... eggs = other
+ ... ''' % globals())
+
+ >>> print system(primed_executable+" "+buildout)
+ Updating eggs.
+ <BLANKLINE>
+
+Specifying the egg exactly will work as well. This shows we correctly
+parse a multi-line value.
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... parts = eggs
+ ... find-links =
+ ... allowed-eggs-from-site-packages = demoneeded
+ ... other
+ ...
+ ... [primed_python]
+ ... executable = %(primed_executable)s
+ ...
+ ... [eggs]
+ ... recipe = zc.recipe.egg:eggs
+ ... python = primed_python
+ ... eggs = other
+ ... ''' % globals())
+
+ >>> print system(primed_executable+" "+buildout)
+ Updating eggs.
+ <BLANKLINE>
+
+It will also work if we use a glob ("*" or "?"). (We won't show that here
+because we already tested it in the previous doctest.)
+
+However, if we do not include "other" in the "allowed-eggs-from-site-packages"
+key, we get an error, because the packages are not available in any links, and
+they are not allowed to come from the executable's site packages. (We won't
+show that here because we already tested it in the previous doctest.)
+
+Finally, here's a test with an empty value. It shows that we parse an empty
+value correctly, and verifies that we really are controlling what eggs are
+allowed, because we see that we were unable to get "other".
+
+ >>> zc.buildout.easy_install.clear_index_cache()
+ >>> write('buildout.cfg',
+ ... '''
+ ... [buildout]
+ ... parts = eggs
+ ... find-links =
+ ... allowed-eggs-from-site-packages =
+ ...
+ ... [primed_python]
+ ... executable = %(primed_executable)s
+ ...
+ ... [eggs]
+ ... recipe = zc.recipe.egg:eggs
+ ... eggs = other
+ ... ''' % globals())
+ >>> print system(primed_executable+" "+buildout)
+ Uninstalling eggs.
+ Installing eggs.
+ Couldn't find index page for 'other' (maybe misspelled?)
+ Getting distribution for 'other'.
+ While:
+ Installing eggs.
+ Getting distribution for 'other'.
+ Error: Couldn't find a distribution for 'other'.
+ <BLANKLINE>
+
+ """
+
def develop_with_modules():
"""
Distribution setup scripts can import modules in the distribution directory:
Modified: zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/zc.recipe.egg_/src/zc/recipe/egg/egg.py
===================================================================
--- zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/zc.recipe.egg_/src/zc/recipe/egg/egg.py 2009-09-26 00:51:19 UTC (rev 104561)
+++ zc.buildout/branches/gary-6-allowed-eggs-from-site-packages/zc.recipe.egg_/src/zc/recipe/egg/egg.py 2009-09-26 00:52:31 UTC (rev 104562)
@@ -75,15 +75,21 @@
orig_distributions = distributions[:]
distributions.extend(extra)
+ kw = {
+ 'allowed_eggs_from_site_packages': tuple(
+ name.strip() for name in
+ options.get(
+ 'allowed-eggs-from-site-packages',
+ b_options['allowed-eggs-from-site-packages']).split('\n'))}
if self.buildout['buildout'].get('offline') == 'true':
ws = zc.buildout.easy_install.working_set(
distributions, options['executable'],
[options['develop-eggs-directory'],
options['eggs-directory']],
include_site_packages = self.include_site_packages,
+ **kw
)
else:
- kw = {}
if 'unzip' in options:
kw['always_unzip'] = get_bool(options, 'unzip')
ws = zc.buildout.easy_install.install(
More information about the checkins
mailing list