[Checkins] SVN: z3c.dependencychecker/trunk/ Import from thehealthagency's internal svn
Reinout van Rees
reinout at vanrees.org
Tue Dec 8 09:14:10 EST 2009
Log message for revision 106285:
Import from thehealthagency's internal svn
Changed:
A z3c.dependencychecker/trunk/
A z3c.dependencychecker/trunk/CHANGES.txt
A z3c.dependencychecker/trunk/README.txt
A z3c.dependencychecker/trunk/TODO.txt
A z3c.dependencychecker/trunk/bootstrap.py
A z3c.dependencychecker/trunk/buildout.cfg
A z3c.dependencychecker/trunk/local-checkouts/
A z3c.dependencychecker/trunk/setup.py
A z3c.dependencychecker/trunk/src/
A z3c.dependencychecker/trunk/src/z3c/
A z3c.dependencychecker/trunk/src/z3c/__init__.py
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/USAGE.txt
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/__init__.py
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/dependencychecker.py
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/importchecker.py
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/__init__.py
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/dependencychecker.txt
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/importchecker.txt
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/setup.py_in
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/__init__.py_in
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/stdlib_imports.py_in
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/tests/
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/tests/__init__.py_in
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/tests/test_setup.py_in
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/unusedimports.py_in
A z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/test_setup.py
-=-
Added: z3c.dependencychecker/trunk/CHANGES.txt
===================================================================
--- z3c.dependencychecker/trunk/CHANGES.txt (rev 0)
+++ z3c.dependencychecker/trunk/CHANGES.txt 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1,27 @@
+Changelog of z3c.dependencychecker
+==================================
+
+0.2 (unreleased)
+----------------
+
+- Added tests. Initial quick test puts coverage at 86%.
+
+- Fixed bug in test requirement detection.
+
+- Added documentation.
+
+
+0.1 (2009-12-02)
+----------------
+
+- Also reporting on unneeded imports.
+
+- Added note on re-running buildout after a setup.py change.
+
+- Added zcml lookup to detect even more missing imports.
+
+- Added reporting on missing regular and test imports.
+
+- Grabbing existing requirements from egginfo directory.
+
+- Copied over Martijn Faassen's zope importchecker script.
Property changes on: z3c.dependencychecker/trunk/CHANGES.txt
___________________________________________________________________
Added: svn:eol-style
+ native
Added: z3c.dependencychecker/trunk/README.txt
===================================================================
--- z3c.dependencychecker/trunk/README.txt (rev 0)
+++ z3c.dependencychecker/trunk/README.txt 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1,48 @@
+z3c.dependencychecker
+=====================
+
+Checks which imports are done and compares them to what's in ``setup.py`` and
+warn when discovering missing or unneeded dependencies.
+
+.. contents::
+
+
+What it does
+------------
+
+z3c.dependencychecker reports on:
+
+- **Unused imports**: pyflakes is another tool that does this (and that also
+ reports on missing variables inside the files).
+
+- **Missing (test) requirements**: imports without a corresponding requirement
+ in the ``setup.py``. There might be false alarms, but at least you've got a
+ (hopefully short) list of items to check.
+
+ Watch out for packages that have a different name than how they're imported.
+ For instance a requirement on ``pydns`` which is used as ``import DNS`` in
+ your code: pydns and DNS lead to separate "missing requirements: DNS" and
+ "unneeded requirements: pydns" warnings.
+
+- **Unneeded (test) requirements**: requirements in your setup.py that aren't
+ imported anywhere in your code. You *might* need them because not
+ everything needs to be imported. It at least gives you a much smaller list
+ to check by hand.
+
+ If something is only imported in a test file, it shouldn't be in the generic
+ defaults. Currently you don't get a separate list of requirements that
+ should be moved from the regular to the test requirements.
+
+Credits
+-------
+
+z3c.dependencychecker is a different application/packaging of zope's
+importchecker utility. It has been used in quite some projects, I grabbed a
+copy from `lovely.recipe's checkout
+<http://bazaar.launchpad.net/~vcs-imports/lovely.recipe/trunk/annotate/head%3A/src/lovely/recipe/importchecker/importchecker.py>`_.
+
+- Martijn Faassen wrote the original importchecker script.
+
+- `Reinout van Rees <http://reinout.vanrees.org>`_ (`The Health Agency
+ <http://www.thehealthagency.com>`_) added the dependency checker
+ functionality and packaged it.
Property changes on: z3c.dependencychecker/trunk/README.txt
___________________________________________________________________
Added: svn:eol-style
+ native
Added: z3c.dependencychecker/trunk/TODO.txt
===================================================================
--- z3c.dependencychecker/trunk/TODO.txt (rev 0)
+++ z3c.dependencychecker/trunk/TODO.txt 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1,16 @@
+TODO
+====
+
+- Add tests for zcml ``package="another.import"`` detection.
+
+- Improve test coverage.
+
+- Try it on more projects and gather feedback.
+
+- Add some extra fallbacks for often-used packages like PIL (which is really
+ Imaging when you import it).
+
+- Optionally check imports inside doctests.
+
+- Optionally report separately on dependencies that should be moved from the
+ regular to the test dependencies.
Property changes on: z3c.dependencychecker/trunk/TODO.txt
___________________________________________________________________
Added: svn:eol-style
+ native
Added: z3c.dependencychecker/trunk/bootstrap.py
===================================================================
--- z3c.dependencychecker/trunk/bootstrap.py (rev 0)
+++ z3c.dependencychecker/trunk/bootstrap.py 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1,84 @@
+##############################################################################
+#
+# Copyright (c) 2006 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""Bootstrap a buildout-based project
+
+Simply run this script in a directory containing a buildout.cfg.
+The script accepts buildout command-line options, so you can
+use the -c option to specify an alternate configuration file.
+
+$Id: bootstrap.py 102545 2009-08-06 14:49:47Z chrisw $
+"""
+
+import os, shutil, sys, tempfile, urllib2
+
+tmpeggs = tempfile.mkdtemp()
+
+is_jython = sys.platform.startswith('java')
+
+try:
+ import pkg_resources
+except ImportError:
+ ez = {}
+ exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py'
+ ).read() in ez
+ ez['use_setuptools'](to_dir=tmpeggs, download_delay=0)
+
+ import pkg_resources
+
+if sys.platform == 'win32':
+ def quote(c):
+ if ' ' in c:
+ return '"%s"' % c # work around spawn lamosity on windows
+ else:
+ return c
+else:
+ def quote (c):
+ return c
+
+cmd = 'from setuptools.command.easy_install import main; main()'
+ws = pkg_resources.working_set
+
+if len(sys.argv) > 2 and sys.argv[1] == '--version':
+ VERSION = '==%s' % sys.argv[2]
+ args = sys.argv[3:] + ['bootstrap']
+else:
+ VERSION = ''
+ args = sys.argv[1:] + ['bootstrap']
+
+if is_jython:
+ import subprocess
+
+ assert subprocess.Popen([sys.executable] + ['-c', quote(cmd), '-mqNxd',
+ quote(tmpeggs), 'zc.buildout' + VERSION],
+ env=dict(os.environ,
+ PYTHONPATH=
+ ws.find(pkg_resources.Requirement.parse('setuptools')).location
+ ),
+ ).wait() == 0
+
+else:
+ assert os.spawnle(
+ os.P_WAIT, sys.executable, quote (sys.executable),
+ '-c', quote (cmd), '-mqNxd', quote (tmpeggs), 'zc.buildout' + VERSION,
+ dict(os.environ,
+ PYTHONPATH=
+ ws.find(pkg_resources.Requirement.parse('setuptools')).location
+ ),
+ ) == 0
+
+ws.add_entry(tmpeggs)
+ws.require('zc.buildout' + VERSION)
+import zc.buildout.buildout
+zc.buildout.buildout.main(args)
+shutil.rmtree(tmpeggs)
Property changes on: z3c.dependencychecker/trunk/bootstrap.py
___________________________________________________________________
Added: svn:eol-style
+ native
Added: z3c.dependencychecker/trunk/buildout.cfg
===================================================================
--- z3c.dependencychecker/trunk/buildout.cfg (rev 0)
+++ z3c.dependencychecker/trunk/buildout.cfg 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1,56 @@
+[buildout]
+unzip = true
+prefer-final = true
+versions = versions
+extensions = buildout.dumppickedversions
+parts =
+ test
+ omelette
+ console_scripts
+develop =
+ .
+eggs =
+ z3c.dependencychecker
+
+
+[versions]
+# Specific pins
+z3c.dependencychecker =
+# buildout.dumppickedversions
+collective.recipe.omelette = 0.9
+eazysvn = 1.11.0
+ipython = 0.10
+martian = 0.12
+pep8 = 0.4.2
+plone.recipe.alltests = 1.2
+z3c.testsetup = 0.6.1
+zc.recipe.egg = 1.2.2
+zc.recipe.testrunner = 1.2.0
+zest.releaser = 3.1
+zope.exceptions = 3.5.2
+zope.interface = 3.5.3
+zope.testing = 3.8.3
+
+
+[test]
+recipe = zc.recipe.testrunner
+defaults = ['--tests-pattern', '^f?tests$', '-v', '-c']
+eggs =
+ z3c.dependencychecker
+ z3c.dependencychecker[test]
+
+
+[console_scripts]
+recipe = zc.recipe.egg
+interpreter = python
+eggs =
+ ${buildout:eggs}
+ eazysvn
+ ipython
+ pep8
+ zest.releaser
+
+
+[omelette]
+recipe = collective.recipe.omelette
+eggs = ${buildout:eggs}
Added: z3c.dependencychecker/trunk/setup.py
===================================================================
--- z3c.dependencychecker/trunk/setup.py (rev 0)
+++ z3c.dependencychecker/trunk/setup.py 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1,42 @@
+from setuptools import setup, find_packages
+import os.path
+
+version = '0.2dev'
+
+long_description = '\n\n'.join([
+ open('README.txt').read(),
+ open(os.path.join('src', 'z3c', 'dependencychecker', 'USAGE.txt')).read(),
+ open('TODO.txt').read(),
+ open('CHANGES.txt').read(),
+ ])
+
+setup(name='z3c.dependencychecker',
+ version=version,
+ description="",
+ long_description=long_description,
+ # Get strings from http://www.python.org/pypi?%3Aaction=list_classifiers
+ classifiers=[],
+ keywords=[],
+ author='The Health Agency',
+ author_email='techniek at thehealthagency.com',
+ url='http://www.thehealthagency.com',
+ license='ZPL',
+ package_dir={'': 'src'},
+ packages=find_packages('src'),
+ namespace_packages=['z3c'],
+ include_package_data=True,
+ zip_safe=False,
+ install_requires=[
+ 'setuptools',
+ ],
+ extras_require = {
+ 'test': [
+ 'z3c.testsetup>=0.3',
+ 'zope.testing',
+ ],
+ },
+ entry_points={
+ 'console_scripts':
+ ['dependencychecker = z3c.dependencychecker.dependencychecker:main'
+ ]},
+ )
Property changes on: z3c.dependencychecker/trunk/setup.py
___________________________________________________________________
Added: svn:eol-style
+ native
Added: z3c.dependencychecker/trunk/src/z3c/__init__.py
===================================================================
--- z3c.dependencychecker/trunk/src/z3c/__init__.py (rev 0)
+++ z3c.dependencychecker/trunk/src/z3c/__init__.py 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1 @@
+__import__('pkg_resources').declare_namespace(__name__)
Property changes on: z3c.dependencychecker/trunk/src/z3c/__init__.py
___________________________________________________________________
Added: svn:eol-style
+ native
Added: z3c.dependencychecker/trunk/src/z3c/dependencychecker/USAGE.txt
===================================================================
--- z3c.dependencychecker/trunk/src/z3c/dependencychecker/USAGE.txt (rev 0)
+++ z3c.dependencychecker/trunk/src/z3c/dependencychecker/USAGE.txt 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1,84 @@
+Usage of z3c.dependencychecker
+==============================
+
+.. :doctest:
+
+
+Installation
+------------
+
+Either install z3c.dependencychecker globally (``easy_install
+z3c.dependencychecker``) or install it in your buildout.
+
+
+Usage
+-----
+
+Run the ``dependencychecker`` or ``bin/dependencychecker`` script from your
+project's root folder and it will report on your dependencies.
+
+By default, it looks in the ``src/`` directory for your sources.
+Alternatively, you can specify a start directory yourself, for instance
+``'.'`` if there's no ``src/`` directory.
+
+We have a sample project in a temp directory:
+
+ >>> sample1_dir
+ '/TESTTEMP/sample1'
+ >>> ls(sample1_dir)
+ setup.py
+ src
+
+For our test, we call the main() method, just like the ``dependencychecker``
+script would.
+
+ >>> import os
+ >>> os.chdir(sample1_dir)
+ >>> from z3c.dependencychecker import dependencychecker
+ >>> dependencychecker.main()
+ Unused imports
+ ==============
+ /TESTTEMP/sample1/src/sample1/unusedimports.py:7: tempfile
+ /TESTTEMP/sample1/src/sample1/unusedimports.py:4: zest.releaser
+ /TESTTEMP/sample1/src/sample1/unusedimports.py:6: os
+ <BLANKLINE>
+ Missing test requirements
+ =========================
+ reinout.hurray
+ <BLANKLINE>
+ Unneeded requirements
+ =====================
+ unneeded.req
+ <BLANKLINE>
+ Unneeded test requirements
+ ==========================
+ zope.testing
+ <BLANKLINE>
+ Note: requirements are taken from the egginfo dir, so you need
+ to re-run buildout (or setup.py or whatever) for changes in
+ setup.py to have effect.
+ <BLANKLINE>
+
+So z3c.dependencychecker reports on:
+
+- **Unused imports**: pyflakes is another tool that does this (and that also
+ reports on missing variables inside the files).
+
+- **Missing (test) requirements**: imports without a corresponding requirement
+ in the ``setup.py``. There might be false alarms, but at least you've got a
+ (hopefully short) list of items to check.
+
+ Watch out for packages that have a different name than how they're imported.
+ For instance a requirement on ``pydns`` which is used as ``import DNS`` in
+ your code: pydns and DNS lead to separate "missing requirements: DNS" and
+ "unneeded requirements: pydns" warnings.
+
+- **Unneeded (test) requirements**: requirements in your setup.py that aren't
+ imported anywhere in your code. You *might* need them because not
+ everything needs to be imported. It at least gives you a much smaller list
+ to check by hand.
+
+ If something is only imported in a test file, it shouldn't be in the generic
+ defaults. Currently you don't get a separate list of requirements that
+ should be moved from the regular to the test requirements.
+
Property changes on: z3c.dependencychecker/trunk/src/z3c/dependencychecker/USAGE.txt
___________________________________________________________________
Added: svn:eol-style
+ native
Added: z3c.dependencychecker/trunk/src/z3c/dependencychecker/__init__.py
===================================================================
--- z3c.dependencychecker/trunk/src/z3c/dependencychecker/__init__.py (rev 0)
+++ z3c.dependencychecker/trunk/src/z3c/dependencychecker/__init__.py 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1 @@
+# package
Property changes on: z3c.dependencychecker/trunk/src/z3c/dependencychecker/__init__.py
___________________________________________________________________
Added: svn:eol-style
+ native
Added: z3c.dependencychecker/trunk/src/z3c/dependencychecker/dependencychecker.py
===================================================================
--- z3c.dependencychecker/trunk/src/z3c/dependencychecker/dependencychecker.py (rev 0)
+++ z3c.dependencychecker/trunk/src/z3c/dependencychecker/dependencychecker.py 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1,249 @@
+##############################################################################
+#
+# Copyright (c) 2003 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+import commands
+import fnmatch
+import os
+import re
+import sys
+
+import pkg_resources
+
+from z3c.dependencychecker import importchecker
+
+
+ZCML_PACKAGE_PATTERN = re.compile(r"""
+\s # Whitespace.
+package= #
+['\"] # Single or double quote.
+(?P<import> # Start of 'import' variable.
+\S+ # Non-whitespace string.
+) # End of 'import' variable.
+['\"] # Single or double quote.
+""", re.VERBOSE)
+
+
+def print_unused_imports(unused_imports):
+ found = []
+ for path in sorted(unused_imports.keys()):
+ for (module, line_number) in unused_imports[path]:
+ found.append((path, line_number, module))
+ if found:
+ print "Unused imports"
+ print "=============="
+ for (path, line_number, module) in found:
+ print "%s:%s: %s" % (path, line_number, module)
+ print
+
+
+def name_from_setup():
+ cmd = "%s setup.py --name" % sys.executable
+ name = commands.getoutput(cmd).strip()
+ if 'traceback' in name.lower():
+ print "You probably don't have setuptools installed globally"
+ print "Or there's an error in your setup.py."
+ print "Try running this by hand:"
+ print cmd
+ # Use buildout's setuptools_loc environ hack.
+ sys.exit(1)
+ return name
+
+
+def existing_requirements():
+ """Extract install and test requirements"""
+ name = name_from_setup()
+ # Who on earth made it so earth-shattering impossible to get your hands on
+ # the egg info stuff via an api? We'll do it mostly by hand...
+ egginfo_dir_name = name + '.egg-info'
+ if egginfo_dir_name in os.listdir(os.getcwd()):
+ egginfo_dir = egginfo_dir_name
+ else:
+ egginfo_dir = os.path.join(os.getcwd(), 'src', egginfo_dir_name)
+ requires_filename = os.path.join(egginfo_dir, 'requires.txt')
+ if not os.path.exists(requires_filename):
+ print "No %s found, exiting" % requires_filename
+ sys.exit(1)
+ lines = [line.strip() for line in open(requires_filename).readlines()]
+ lines = [line for line in lines if line]
+ install_required = []
+ test_required = []
+ for line in lines:
+ if line.startswith('['):
+ break
+ req = pkg_resources.Requirement.parse(line)
+ install_required.append(req.project_name)
+ start = False
+ for line in lines:
+ if line == '[test]':
+ start = True
+ continue
+ if not start:
+ continue
+ if line.startswith('['):
+ break
+ req = pkg_resources.Requirement.parse(line)
+ test_required.append(req.project_name)
+
+ # The project itself is of course both available and needed.
+ install_required.append(name)
+
+ # Distribute says it is setuptools. Setuptools also includes
+ # pkg_resources.
+ if 'distribute' in install_required:
+ install_required.append('setuptools')
+ if 'setuptools' in install_required:
+ install_required.append('pkg_resources')
+
+ return (install_required, test_required)
+
+
+def filter_missing(imports, required):
+ missing = []
+ for needed in imports:
+ found = False
+ for req in required:
+ if req == needed:
+ found = True
+ if needed.startswith(req + '.'):
+ # 're' should not match 'reinout.something', that's why we
+ # check with an extra dot.
+ found = True
+ if not found:
+ missing.append(needed)
+ return missing
+
+
+def filter_unneeded(imports, required):
+ name = name_from_setup()
+ imports.append(name) # We always use ourselves, obviously.
+ imports = set(imports)
+ required = set(required)
+ setuptools_and_friends = set(
+ ['distribute', 'setuptools', 'pkg_resources'])
+ required = required - setuptools_and_friends
+
+ unneeded = []
+ for req in required:
+ found = False
+ for module in imports:
+ if module.startswith(req):
+ found = True
+ if not found:
+ unneeded.append(req)
+ return unneeded
+
+
+def _detect_modules(sample_module):
+ sample_file = os.path.realpath(sample_module.__file__)
+ stdlib_dir = os.path.dirname(sample_file)
+ stdlib_extension = os.path.splitext(sample_file)[1]
+ stdlib_files = os.listdir(stdlib_dir)
+ modules = []
+ for stdlib_file in stdlib_files:
+ module, extension = os.path.splitext(stdlib_file)
+ if extension == stdlib_extension:
+ modules.append(module)
+ if 'py' in stdlib_extension:
+ # Also check directories with __init__.py* in them.
+ init_file = '__init__' + stdlib_extension
+ extra_modules = [name for name in os.listdir(stdlib_dir)
+ if os.path.exists(os.path.join(
+ stdlib_dir, name, init_file))]
+ modules += extra_modules
+ return modules
+
+
+def stdlib_modules():
+ py_module = os
+ import datetime
+ dynload_module = datetime
+ modules = _detect_modules(py_module) + _detect_modules(dynload_module)
+ modules.append('sys')
+ return list(set(modules))
+
+
+def includes_from_zcml(path):
+ modules = []
+ test_modules = []
+ for path, dirs, files in os.walk(path):
+ for zcml in [os.path.abspath(os.path.join(path, filename))
+ for filename in files
+ if fnmatch.fnmatch(filename, '*.zcml')]:
+ contents = open(zcml).read()
+ found = [module for module in
+ re.findall(ZCML_PACKAGE_PATTERN, contents)
+ if not module.startswith('.')]
+ if 'test' in zcml:
+ # ftesting.zcml, mostly.
+ test_modules += found
+ else:
+ modules += found
+ return modules, test_modules
+
+
+def print_modules(modules, heading):
+ if modules:
+ print heading
+ print '=' * len(heading)
+ for module in modules:
+ print " ", module
+ print
+
+
+def main():
+ if len(sys.argv) > 1:
+ path = sys.argv[1]
+ else:
+ # Default
+ path = os.path.join(os.getcwd(), 'src')
+ path = os.path.abspath(path)
+ if not os.path.isdir(path):
+ print "Unknown path:", path
+ sys.exit(1)
+
+ db = importchecker.ImportDatabase(path)
+ # TODO: find zcml files
+ db.findModules()
+ unused_imports = db.getUnusedImports()
+ test_imports = db.getImportedModuleNames(tests=True)
+ install_imports = db.getImportedModuleNames(tests=False)
+ (install_required, test_required) = existing_requirements()
+ stdlib = stdlib_modules()
+ (zcml_imports, zcml_test_imports) = includes_from_zcml(path)
+
+ print_unused_imports(unused_imports)
+
+ install_missing = filter_missing(install_imports + zcml_imports,
+ install_required + stdlib)
+ print_modules(install_missing, "Missing requirements")
+
+ test_missing = filter_missing(test_imports + zcml_test_imports,
+ install_required + test_required + stdlib)
+ print_modules(test_missing, "Missing test requirements")
+
+ install_unneeded = filter_unneeded(install_imports + zcml_imports,
+ install_required)
+ print_modules(install_unneeded, "Unneeded requirements")
+
+ test_unneeded = filter_unneeded(test_imports + zcml_test_imports,
+ test_required)
+ print_modules(test_unneeded, "Unneeded test requirements")
+
+
+ if install_missing or test_missing or install_unneeded or test_unneeded:
+ print "Note: requirements are taken from the egginfo dir, so you need"
+ print "to re-run buildout (or setup.py or whatever) for changes in "
+ print "setup.py to have effect."
+ print
+
+ # TODO: report on unneeded requirements
Property changes on: z3c.dependencychecker/trunk/src/z3c/dependencychecker/dependencychecker.py
___________________________________________________________________
Added: svn:eol-style
+ native
Added: z3c.dependencychecker/trunk/src/z3c/dependencychecker/importchecker.py
===================================================================
--- z3c.dependencychecker/trunk/src/z3c/dependencychecker/importchecker.py (rev 0)
+++ z3c.dependencychecker/trunk/src/z3c/dependencychecker/importchecker.py 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1,298 @@
+##############################################################################
+#
+# Copyright (c) 2003 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""Import checker
+
+This utility finds is based on the zope importchecker script, prints
+out unused imports, imports that are only for tests packages and
+runtime imports.
+
+$Id: importchecker.py 96082 2009-02-04 16:37:38Z adamg $
+"""
+import compiler
+import os
+import os.path
+
+
+def _findDottedNamesHelper(node, result):
+ more_node = node
+ name = node.__class__.__name__
+ if name == 'Getattr':
+ dotted = []
+ while name == 'Getattr':
+ dotted.append(node.attrname)
+ node = node.expr
+ name = node.__class__.__name__
+ if name == 'Name':
+ dotted.append(node.name)
+ dotted.reverse()
+ for i in range(1, len(dotted)):
+ result.append('.'.join(dotted[:i]))
+ result.append('.'.join(dotted))
+ return
+ elif name == 'Name':
+ result.append(node.name)
+ return
+ elif name == 'AssAttr':
+ return
+ for child in more_node.getChildNodes():
+ _findDottedNamesHelper(child, result)
+
+
+def findDottedNames(node):
+ """Find dotted names in an AST tree node
+ """
+ result = []
+ _findDottedNamesHelper(node, result)
+ return result
+
+
+class ImportFinder:
+ """An instance of this class will be used to walk over a compiler AST
+ tree (a module). During that operation, the appropriate methods of
+ this visitor will be called
+ """
+
+ def __init__(self):
+ self._map = {}
+
+ def visitFrom(self, stmt):
+ """Will be called for 'from foo import bar' statements
+ """
+ x = stmt.asList()
+ module_name = x[0]
+ names = x[1]
+ if module_name == '__future__':
+ # we don't care what's imported from the future
+ return
+ names_dict = {}
+ for orig_name, as_name in names:
+ # we don't care about from import *
+ if orig_name == '*':
+ continue
+ if as_name is None:
+ name = orig_name
+ else:
+ name = as_name
+ names_dict[name] = orig_name
+ self._map.setdefault(module_name, {'names': names_dict,
+ 'lineno': stmt.lineno})
+
+ def visitImport(self, stmt):
+ """Will be called for 'import foo.bar' statements
+ """
+ for orig_name, as_name in stmt.names:
+ if as_name is None:
+ name = orig_name
+ else:
+ name = as_name
+ self._map.setdefault(orig_name, {'names': {name: orig_name},
+ 'lineno': stmt.lineno})
+
+ def getMap(self):
+ return self._map
+
+
+def findImports(mod):
+ """Find import statements in module and put the result in a mapping.
+ """
+ visitor = ImportFinder()
+ compiler.walk(mod, visitor)
+ return visitor.getMap()
+
+
+class Module:
+ """This represents a python module.
+ """
+
+ def __init__(self, path):
+ mod = compiler.parseFile(path)
+ self._path = path
+ self._map = findImports(mod)
+ self._dottednames = findDottedNames(mod)
+
+ def getPath(self):
+ """Return the path to this module's file.
+ """
+ return self._path
+
+ def getImportedModuleNames(self):
+ """Return the names of imported modules.
+ """
+ return self._map.keys()
+
+ def getImportNames(self):
+ """Return the names of imports; add dottednames as well.
+ """
+ result = []
+ map = self._map
+ for module_name in map.keys():
+ for usedname, originalname in map[module_name]['names'].items():
+ result.append((originalname, module_name))
+ # add any other name that we could be using
+ for dottedname in self._dottednames:
+ usednamedot = usedname + '.'
+ if dottedname.startswith(usednamedot):
+ attrname = dottedname[len(usednamedot):].split('.')[0]
+ result.append((attrname, module_name))
+
+ return result
+
+ def getUnusedImports(self):
+ """Get unused imports of this module (the whole import info).
+ """
+ result = []
+ for value in self._map.values():
+ for usedname, originalname in value['names'].items():
+ if usedname not in self._dottednames:
+ result.append((originalname, value['lineno']))
+ return result
+
+
+class ModuleFinder:
+
+ def __init__(self):
+ self._files = []
+
+ def visit(self, arg, dirname, names):
+ """This method will be called when we walk the filesystem
+ tree. It looks for python modules and stored their filenames.
+ """
+ for name in names:
+ # get all .py files that aren't weirdo emacs droppings
+ if name.endswith('.py') and not name.startswith('.#'):
+ self._files.append(os.path.join(dirname, name))
+
+ def getModuleFilenames(self):
+ return self._files
+
+
+def findModules(path):
+ """Find python modules in the given path and return their absolute
+ filenames in a sequence.
+ """
+ finder = ModuleFinder()
+ os.path.walk(path, finder.visit, ())
+ return finder.getModuleFilenames()
+
+
+class ImportDatabase:
+ """This database keeps tracks of imports.
+
+ It allows to NOT report cases where a module imports something
+ just so that another module can import it (import dependencies).
+ """
+
+ def __init__(self, root_path):
+ self._root_path = root_path
+ self._modules = {}
+ self._names = {}
+
+ def resolveDottedModuleName(self, dotted_name, module):
+ """Return path to file representing module, or None if no such
+ thing. Can do this relative from module.
+ """
+ dotted_path = dotted_name.replace('.', '/')
+ # try relative import first
+ path = os.path.join(os.path.dirname(module.getPath()), dotted_path)
+ path = self._resolveHelper(path)
+ if path is not None:
+ return path
+ # absolute import (assumed to be from this tree)
+ if os.path.isfile(os.path.join(self._root_path, '__init__.py')):
+ startpath, dummy = os.path.split(self._root_path)
+ else:
+ startpath = self._root_path
+ return self._resolveHelper(os.path.join(startpath, dotted_path))
+
+ def _resolveHelper(self, path):
+ if os.path.isfile(path + '.py'):
+ return path + '.py'
+ if os.path.isdir(path):
+ path = os.path.join(path, '__init__.py')
+ if os.path.isfile(path):
+ return path
+ return None
+
+ def findModules(self):
+ """Find modules in the given path.
+ """
+ for modulepath in findModules(self._root_path):
+ module = Module(modulepath)
+ self.addModule(module)
+
+ def addModule(self, module):
+ """Add information about a module to the database. A module in
+ this case is not a python module object, but an instance of
+ the above defined Module class.w
+ """
+ self_path = module.getPath()
+ # do nothing if we already know about it
+ if self_path in self._modules:
+ return
+
+ self._modules[self_path] = module
+
+ # add imported names to internal names mapping; this will
+ # allow us identify dependent imports later
+ names = self._names
+ for name, from_module_name in module.getImportNames():
+ path = self.resolveDottedModuleName(from_module_name, module)
+ t = (path, name)
+ modulepaths = names.get(t, {})
+ if not self_path in modulepaths:
+ modulepaths[self_path] = 1
+ names[t] = modulepaths
+
+ def getUnusedImports(self):
+ """Get unused imports of all known modules.
+ """
+ result = {}
+ for path, module in self._modules.items():
+ result[path] = self.getUnusedImportsInModule(module)
+ return result
+
+ def getImportedModuleNames(self, tests=False):
+ """returns all names imported by modules"""
+ result = set()
+ import os
+ for path, module in self._modules.items():
+ # remove .py
+ parts = path[:-3].split(os.path.sep)
+ isTest = 'tests' in parts or 'testing' in parts \
+ or 'ftests' in parts
+ if (tests and isTest) or (not tests and not isTest):
+ result.update(module.getImportedModuleNames())
+ return sorted(result)
+
+ def getUnusedImportsInModule(self, module):
+ """Get all unused imports in a module.
+ """
+ result = []
+ for name, lineno in module.getUnusedImports():
+ if not self.isNameImportedFrom(name, module):
+ result.append((name, lineno))
+ return result
+
+ def isNameImportedFrom(self, name, module):
+ """Return true if name is imported from module by another module.
+ """
+ return (module.getPath(), name) in self._names
+
+ def getModulesImportingNameFrom(self, name, module):
+ """Return list of known modules that import name from module.
+ """
+ result = []
+ for path in self._names.get((module.getPath(), name), {}).keys():
+ result.append(self._modules[path])
+ return result
Property changes on: z3c.dependencychecker/trunk/src/z3c/dependencychecker/importchecker.py
___________________________________________________________________
Added: svn:eol-style
+ native
Added: z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/__init__.py
===================================================================
--- z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/__init__.py (rev 0)
+++ z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/__init__.py 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1 @@
+# package
Property changes on: z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/__init__.py
___________________________________________________________________
Added: svn:eol-style
+ native
Added: z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/dependencychecker.txt
===================================================================
--- z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/dependencychecker.txt (rev 0)
+++ z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/dependencychecker.txt 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1,8 @@
+"""
+.. :doctest:
+
+Sample test:
+
+ >>> from z3c.dependencychecker import dependencychecker
+
+"""
Property changes on: z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/dependencychecker.txt
___________________________________________________________________
Added: svn:eol-style
+ native
Added: z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/importchecker.txt
===================================================================
--- z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/importchecker.txt (rev 0)
+++ z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/importchecker.txt 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1,9 @@
+"""
+.. :doctest:
+
+Sample test:
+
+ >>> from z3c.dependencychecker import importchecker
+
+
+"""
Property changes on: z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/importchecker.txt
___________________________________________________________________
Added: svn:eol-style
+ native
Added: z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/setup.py_in
===================================================================
--- z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/setup.py_in (rev 0)
+++ z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/setup.py_in 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1,33 @@
+from setuptools import setup, find_packages
+
+
+setup(name='sample1',
+ version='0.2dev',
+ description='',
+ long_description='',
+ classifiers=[],
+ keywords=[],
+ author='The Health Agency',
+ author_email='techniek at thehealthagency.com',
+ url='http://www.thehealthagency.com',
+ license='ZPL',
+ package_dir={'': 'src'},
+ packages=find_packages('src'),
+ include_package_data=True,
+ zip_safe=False,
+ install_requires=[
+ # Don't forget to add this by hand to the
+ # src/sample1.egg-info/requires.txt file!
+ 'setuptools',
+ 'zest.releaser',
+ 'unneeded.req',
+ ],
+ extras_require = {
+ 'test': [
+ # Don't forget to add this by hand to the
+ # src/sample1.egg-info/requires.txt file!
+ 'z3c.testsetup>=0.3',
+ 'zope.testing',
+ ],
+ },
+ )
Added: z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/__init__.py_in
===================================================================
--- z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/__init__.py_in (rev 0)
+++ z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/__init__.py_in 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1 @@
+#package
Added: z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/stdlib_imports.py_in
===================================================================
--- z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/stdlib_imports.py_in (rev 0)
+++ z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/stdlib_imports.py_in 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1,16 @@
+"""Imports from the standard library"""
+
+# os is a stdlib, so os.path imports should be OK
+import os.path
+
+import tempfile
+import random
+from datetime import datetime
+
+
+# "use" them.
+
+tempfile.mkdtemp(prefix="not really")
+random.thingy
+datetime.now()
+os.path.join(me, you)
\ No newline at end of file
Added: z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/tests/__init__.py_in
===================================================================
--- z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/tests/__init__.py_in (rev 0)
+++ z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/tests/__init__.py_in 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1 @@
+#package
Added: z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/tests/test_setup.py_in
===================================================================
--- z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/tests/test_setup.py_in (rev 0)
+++ z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/tests/test_setup.py_in 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1,10 @@
+"""Using our test dependency and also using one that's missing"""
+
+# This one's OK:
+import z3c.testsetup
+# This one's missing:
+import reinout.hurray
+
+
+z3c.testsetup.something()
+reinout.hurray.hip_hip()
Added: z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/unusedimports.py_in
===================================================================
--- z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/unusedimports.py_in (rev 0)
+++ z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/sample1/src/sample1/unusedimports.py_in 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1,7 @@
+"""Imports that are available, but that we do not use"""
+
+# The project we explicitly depend on:
+import zest.releaser
+# Some stdlib import
+import os
+import tempfile
Added: z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/test_setup.py
===================================================================
--- z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/test_setup.py (rev 0)
+++ z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/test_setup.py 2009-12-08 14:14:10 UTC (rev 106285)
@@ -0,0 +1,85 @@
+##############################################################################
+#
+# Copyright (c) 2003 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+import os
+import re
+import shutil
+import sys
+import tempfile
+
+import pkg_resources
+
+from zope.testing import renormalizing
+import z3c.testsetup
+
+
+checker = renormalizing.RENormalizing([
+ # Temp directory from setup() (including /private OSX madness).
+ (re.compile(
+ '/private%s/dependencychecker[^/]+' % re.escape(
+ tempfile.gettempdir())),
+ '/TESTTEMP'),
+ (re.compile(
+ '%s/dependencychecker[^/]+' % re.escape(tempfile.gettempdir())),
+ '/TESTTEMP'),
+ # Just the default /tmp directory.
+ (re.compile(re.escape(tempfile.gettempdir())),
+ '/TMPDIR'),
+ ])
+
+
+def ls(directory):
+ for item in sorted(os.listdir(directory)):
+ if item.startswith('.'):
+ continue
+ print item
+
+
+def setup(test):
+ """Set up tempdir with sample project"""
+ test.orig_sysargv = sys.argv[:]
+ test.orig_dir = os.getcwd()
+ sys.argv[1:] = []
+ test.tempdir = tempfile.mkdtemp(prefix='dependencychecker')
+ sample1_source = pkg_resources.resource_filename(
+ 'z3c.dependencychecker.tests', 'sample1')
+ sample1_dir = os.path.join(test.tempdir, 'sample1')
+ test.globs['sample1_dir'] = sample1_dir
+ test.globs['ls'] = ls
+ shutil.copytree(sample1_source, sample1_dir)
+ # To prevent the sample .py files to be picked up by ourselves or other
+ # tools, I'm postfixing them with ``_in``, now we get to rename them.
+ # Same for zcml files.
+ for (dirpath, dirnames, filenames) in os.walk(sample1_dir):
+ for filename in filenames:
+ if not filename.endswith('_in'):
+ continue
+ new_filename = filename.replace('_in', '')
+ source = os.path.join(dirpath, filename)
+ target = os.path.join(dirpath, new_filename)
+ os.rename(source, target)
+
+
+def teardown(test):
+ """Clean up"""
+ #print "Not zapping", test.tempdir
+ shutil.rmtree(test.tempdir)
+ sys.argv[:] = test.orig_sysargv
+ os.chdir(test.orig_dir)
+
+
+test_suite = z3c.testsetup.register_all_tests(
+ 'z3c.dependencychecker',
+ checker=checker,
+ setup=setup,
+ teardown=teardown)
Property changes on: z3c.dependencychecker/trunk/src/z3c/dependencychecker/tests/test_setup.py
___________________________________________________________________
Added: svn:eol-style
+ native
More information about the checkins
mailing list