[Checkins] SVN: z3c.recipe.filetemplate/branches/gary-powertools/ tests are significant, are incomplete, and pass
Gary Poster
gary at modernsongs.com
Tue Apr 28 22:08:25 EDT 2009
Log message for revision 99569:
tests are significant, are incomplete, and pass
Changed:
_U z3c.recipe.filetemplate/branches/gary-powertools/
U z3c.recipe.filetemplate/branches/gary-powertools/setup.py
U z3c.recipe.filetemplate/branches/gary-powertools/z3c/recipe/filetemplate/README.txt
U z3c.recipe.filetemplate/branches/gary-powertools/z3c/recipe/filetemplate/__init__.py
U z3c.recipe.filetemplate/branches/gary-powertools/z3c/recipe/filetemplate/tests.py
U z3c.recipe.filetemplate/branches/gary-powertools/z3c/recipe/filetemplate/tests.txt
-=-
Property changes on: z3c.recipe.filetemplate/branches/gary-powertools
___________________________________________________________________
Added: svn:externals
+ bootstrap svn://svn.zope.org/repos/main/zc.buildout/trunk/bootstrap
Modified: z3c.recipe.filetemplate/branches/gary-powertools/setup.py
===================================================================
--- z3c.recipe.filetemplate/branches/gary-powertools/setup.py 2009-04-29 02:07:37 UTC (rev 99568)
+++ z3c.recipe.filetemplate/branches/gary-powertools/setup.py 2009-04-29 02:08:24 UTC (rev 99569)
@@ -27,6 +27,7 @@
namespace_packages=['z3c', 'z3c.recipe'],
install_requires=['setuptools',
'zc.buildout',
+ 'zc.recipe.egg',
],
zip_safe=True,
entry_points="""
Modified: z3c.recipe.filetemplate/branches/gary-powertools/z3c/recipe/filetemplate/README.txt
===================================================================
--- z3c.recipe.filetemplate/branches/gary-powertools/z3c/recipe/filetemplate/README.txt 2009-04-29 02:07:37 UTC (rev 99568)
+++ z3c.recipe.filetemplate/branches/gary-powertools/z3c/recipe/filetemplate/README.txt 2009-04-29 02:08:24 UTC (rev 99569)
@@ -1,14 +1,21 @@
+``z3c.recipe.filetemplate``
+***************************
+
+===========
+Basic Usage
+===========
+
With the ``z3c.recipe.filetemplate`` buildout recipe you can automate
the generation of text files from templates. Upon execution, the
-recipe will read a number of template files, perform a simple variable
+recipe will read a number of template files, perform variable
substitution and write the result to the corresponding output files.
For example, consider this simple template for a text file:
- >>> write(sample_buildout, 'helloworld.txt.in',
- ... """
- ... Hello ${world}!
- ... """)
+ >>> write(sample_buildout, 'helloworld.txt.in',
+ ... """
+ ... Hello ${world}!
+ ... """)
Now let's create a buildout configuration so that we can substitute
the values in this file. All we have to do is define a part that uses
@@ -17,22 +24,291 @@
whitespace). Then we can add arbitrary parameters to the section.
Those will be used to fill the variables in the template:
- >>> write(sample_buildout, 'buildout.cfg',
- ... """
- ... [buildout]
- ... parts = file
- ...
- ... [file]
- ... recipe = z3c.recipe.filetemplate
- ... files = helloworld.txt
- ... world = Philipp
- ... """)
+ >>> write(sample_buildout, 'buildout.cfg',
+ ... """
+ ... [buildout]
+ ... parts = message
+ ...
+ ... [message]
+ ... recipe = z3c.recipe.filetemplate
+ ... files = helloworld.txt
+ ... world = Philipp
+ ... """)
After executing buildout, we can see that ``$world`` has indeed been
replaced by ``Philipp``:
- >>> print system(buildout)
- Installing file.
+ >>> print system(buildout)
+ Installing message.
- >>> cat(sample_buildout, 'helloworld.txt')
- Hello Philipp!
+ >>> cat(sample_buildout, 'helloworld.txt')
+ Hello Philipp!
+
+Note that the output file uses the same permission bits as found on the input
+file.
+
+ >>> import stat
+ >>> import os
+ >>> input = os.path.join(sample_buildout, 'helloworld.txt.in')
+ >>> output = input[:-3]
+ >>> os.chmod(input, 0755)
+ >>> stat.S_IMODE(os.stat(input).st_mode) == 0755
+ True
+ >>> stat.S_IMODE(os.stat(output).st_mode) == 0755
+ False
+ >>> print system(buildout)
+ Uninstalling message.
+ Installing message.
+ >>> stat.S_IMODE(os.stat(output).st_mode) == 0755
+ True
+
+Source Folders and Globs
+========================
+
+By default, the recipe looks for a ``.in`` file relative to the buildout root,
+and places it in the same folder relative to the buildout root. However, if
+you don't want to clutter up the destination folder, you can add a prefix to
+the source folder. Here is an example.
+
+Note that, for the destination, intermediate folders are created if they do not
+exist.
+
+ >>> write(sample_buildout, 'buildout.cfg',
+ ... """
+ ... [buildout]
+ ... parts = message
+ ...
+ ... [message]
+ ... recipe = z3c.recipe.filetemplate
+ ... source-directory = template
+ ... world = Philipp
+ ... """)
+ >>> mkdir(sample_buildout, 'template')
+ >>> mkdir(sample_buildout, 'template', 'etc')
+ >>> mkdir(sample_buildout, 'template', 'bin')
+ >>> write(sample_buildout, 'template', 'etc', 'helloworld.conf.in',
+ ... """
+ ... Hello ${world} from the etc dir!
+ ... """)
+ >>> write(sample_buildout, 'template', 'bin', 'helloworld.sh.in',
+ ... """
+ ... Hello ${world} from the bin dir!
+ ... """)
+ >>> os.chmod(
+ ... os.path.join(
+ ... sample_buildout, 'template', 'bin', 'helloworld.sh.in'),
+ ... 0711)
+ >>> ls(sample_buildout)
+ - .installed.cfg
+ d bin
+ - buildout.cfg
+ d develop-eggs
+ d eggs
+ - helloworld.txt
+ - helloworld.txt.in
+ d parts
+ d template
+ >>> ls(sample_buildout, 'bin')
+ - buildout
+ >>> print system(buildout)
+ Uninstalling message.
+ Installing message.
+ >>> ls(sample_buildout)
+ - .installed.cfg
+ d bin
+ - buildout.cfg
+ d develop-eggs
+ d eggs
+ d etc
+ - helloworld.txt.in
+ d parts
+ d template
+ >>> ls(sample_buildout, 'bin')
+ - buildout
+ - helloworld.sh
+ >>> cat(sample_buildout, 'bin', 'helloworld.sh')
+ Hello Philipp from the bin dir!
+ >>> stat.S_IMODE(os.stat(os.path.join(
+ ... sample_buildout, 'bin', 'helloworld.sh')).st_mode) == 0711
+ True
+ >>> ls(sample_buildout, 'etc')
+ - helloworld.conf
+ >>> cat(sample_buildout, 'etc', 'helloworld.conf')
+ Hello Philipp from the etc dir!
+
+Substituting from Other Sections
+================================
+
+Substitutions can also come from other sections in the buildout, using the
+standard buildout syntax.
+
+ >>> write(sample_buildout, 'helloworld.txt.in',
+ ... """
+ ... Hello ${world}. I used these parts: ${buildout:parts}.
+ ... """)
+ >>> write(sample_buildout, 'buildout.cfg',
+ ... """
+ ... [buildout]
+ ... parts = message
+ ...
+ ... [message]
+ ... recipe = z3c.recipe.filetemplate
+ ... files = helloworld.txt
+ ... world = Philipp
+ ... """)
+
+ >>> print system(buildout)
+ Uninstalling message.
+ Installing message.
+
+ >>> cat(sample_buildout, 'helloworld.txt')
+ Hello Philipp. I used these parts: message.
+
+Sharing variables
+=================
+
+The recipe allows extending one or more sections, to decrease repetition. For
+instance, consider the following buildout.
+
+ >>> write(sample_buildout, 'buildout.cfg',
+ ... """
+ ... [buildout]
+ ... parts = message
+ ...
+ ... [template_defaults]
+ ... mygreeting = Hi
+ ... myaudience = World
+ ...
+ ... [message]
+ ... recipe = z3c.recipe.filetemplate
+ ... files = helloworld.txt
+ ... extends = template_defaults
+ ...
+ ... myaudience = everybody
+ ... """)
+
+The "message" section now has values extended from the "template_defaults"
+section, and overwritten locally. A template of
+``${mygreeting}, ${myaudience}!``...
+
+ >>> write(sample_buildout, 'helloworld.txt.in',
+ ... """
+ ... ${mygreeting}, ${myaudience}!
+ ... """)
+
+...would thus result in ``Hi, everybody!``.
+
+ >>> print system(buildout)
+ Uninstalling message.
+ Installing message.
+
+ >>> cat(sample_buildout, 'helloworld.txt')
+ Hi, everybody!
+
+Specifying paths
+================
+
+You can specify eggs and extra-paths in the recipe. If you do, three
+predefined options will be available in the recipe's options for the template.
+If "paths" are the non-zip paths, and "all_paths" are all paths, then the
+options would be defined roughly as given here:
+
+ os-paths: (os.pathsep).join(paths)
+ string-paths: ', '.join(repr(p) for p in all_paths)
+ space-paths: ' '.join(paths)
+
+For instance, consider this example.
+
+ >>> write(sample_buildout, 'buildout.cfg',
+ ... """
+ ... [buildout]
+ ... parts = message
+ ...
+ ... [message]
+ ... recipe = z3c.recipe.filetemplate
+ ... files = helloworld.txt
+ ... eggs = demo<0.3
+ ...
+ ... find-links = %(server)s
+ ... index = %(server)s/index
+ ... """ % dict(server=link_server))
+
+
+ >>> write(sample_buildout, 'helloworld.txt.in',
+ ... """
+ ... Hello! Here are the paths for the ${eggs} eggs.
+ ... OS paths:
+ ... ${os-paths}
+ ... ---
+ ... String paths:
+ ... ${string-paths}
+ ... ---
+ ... Space paths:
+ ... ${space-paths}
+ ... """)
+
+ >>> print system(buildout)
+ Getting distribution for 'demo<0.3'.
+ Got demo 0.2.
+ Getting distribution for 'demoneeded'.
+ Got demoneeded 1.2c1.
+ Uninstalling message.
+ Installing message.
+
+ >>> cat(sample_buildout, 'helloworld.txt') # doctest:+ELLIPSIS
+ Hello! Here are the paths for the demo<0.3 eggs.
+ OS paths:
+ .../eggs/demo-0.2...egg:.../eggs/demoneeded-1.2c1...egg
+ ---
+ String paths:
+ '.../eggs/demo-0.2...egg', '.../eggs/demoneeded-1.2c1...egg'
+ ---
+ Space paths:
+ .../eggs/demo-0.2...egg .../eggs/demoneeded-1.2c1...egg
+
+You can specify extra-paths as well, which will go at the end of the egg paths.
+
+ >>> write(sample_buildout, 'buildout.cfg',
+ ... """
+ ... [buildout]
+ ... parts = message
+ ...
+ ... [message]
+ ... recipe = z3c.recipe.filetemplate
+ ... files = helloworld.txt
+ ... eggs = demo<0.3
+ ... extra-paths = ${buildout:directory}/foo
+ ...
+ ... find-links = %(server)s
+ ... index = %(server)s/index
+ ... """ % dict(server=link_server))
+
+ >>> print system(buildout)
+ Uninstalling message.
+ Installing message.
+
+ >>> cat(sample_buildout, 'helloworld.txt') # doctest:+ELLIPSIS
+ Hello! Here are the paths for the demo<0.3 eggs.
+ OS paths:
+ ...demo...:...demoneeded...:.../sample-buildout/foo
+ ---
+ String paths:
+ '...demo...', '...demoneeded...', '.../sample-buildout/foo'
+ ---
+ Space paths:
+ ...demo... ...demoneeded... .../sample-buildout/foo
+
+Defining options in Python
+============================
+
+You can specify that certain variables should be interpreted as Python.
+
+XXX
+
+ [buildout]
+ parts = message
+
+ [message]
+ recipe = z3c.recipe.filetemplate
+ files = helloworld.txt
+ interpreted-options = path-separator=os.pathsep
Modified: z3c.recipe.filetemplate/branches/gary-powertools/z3c/recipe/filetemplate/__init__.py
===================================================================
--- z3c.recipe.filetemplate/branches/gary-powertools/z3c/recipe/filetemplate/__init__.py 2009-04-29 02:07:37 UTC (rev 99568)
+++ z3c.recipe.filetemplate/branches/gary-powertools/z3c/recipe/filetemplate/__init__.py 2009-04-29 02:08:24 UTC (rev 99569)
@@ -1,46 +1,181 @@
+import fnmatch
+import logging
import os
+import re
+import stat
import string
-import logging
+import sys
+import zc.recipe.egg
import zc.buildout
+import zc.buildout.easy_install
+ABS_PATH_ERROR = ('%s is an absolute path. Paths must be '
+ 'relative to the buildout directory.')
+
class FileTemplate(object):
def __init__(self, buildout, name, options):
self.buildout = buildout
self.name = name
self.options = options
-
- def install(self, update=False):
+ self.logger=logging.getLogger(self.name)
+ # get defaults from extended sections
+ defaults = {}
+ extends = self.options.get('extends', '').split()
+ extends.reverse()
+ for section_name in extends:
+ defaults.update(self.buildout[section_name])
+ for key, value in defaults.items():
+ self.options.setdefault(key, value)
+ # set up paths for eggs, if given
+ if 'eggs' in self.options:
+ self.eggs = zc.recipe.egg.Scripts(buildout, name, options)
+ orig_distributions, ws = self.eggs.working_set()
+ # we want ws, eggs.extra_paths, eggs._relative_paths
+ all_paths = [
+ zc.buildout.easy_install.realpath(dist.location)
+ for dist in ws]
+ all_paths.sort()
+ all_paths.extend(
+ zc.buildout.easy_install.realpath(path)
+ for path in self.eggs.extra_paths)
+ else:
+ all_paths = []
+ paths = [path for path in all_paths if not path.endswith('.zip')]
+ self.options['os-paths'] = (os.pathsep).join(paths)
+ self.options['string-paths'] = ', '.join(repr(p) for p in all_paths)
+ self.options['space-paths'] = ' '.join(paths)
+ # get and check the files to be created
+ self.filenames = self.options.get('files', '*').split()
+ self.source_dir = self.options.get('source-directory', '').strip()
here = self.buildout['buildout']['directory']
- filenames = self.options['files'].split()
- logger = logging.getLogger(self.name)
-
- for filename in filenames:
+ self.destination_dir = here
+ if self.source_dir:
+ self.recursive = True
+ if os.path.isabs(self.source_dir):
+ self._user_error(ABS_PATH_ERROR, self.source_dir)
+ self.source_dir = os.path.normpath(os.path.join(
+ here, self.source_dir))
+ if not self.source_dir.startswith(here):
+ self._user_error(
+ 'source-directory must be within the buildout directory')
+ else:
+ self.recursive = False
+ self.options['source-directory'] = ''
+ self.source_dir = self.buildout['buildout']['directory']
+ source_patterns = []
+ for filename in self.filenames:
if os.path.isabs(filename):
- msg = ('%s is an absolute path. File paths must be '
- 'relative to the buildout directory.' % filename)
- logger.error(msg)
- raise zc.buildout.UserError(msg)
+ self._user_error(ABS_PATH_ERROR, filename)
+ if self.source_dir:
+ if '/' in filename:
+ self._user_error(
+ 'Slashes cannot be in file names when a source '
+ 'directory is used: %s.',
+ filename)
+ else:
+ if not os.path.normpath(
+ os.path.join(self.source_dir, filename)
+ ).startswith(self.source_dir):
+ # path used ../ to get out of buildout dir
+ self._user_error(
+ 'source files must be within the buildout directory')
+ source_patterns.append('%s.in' % filename)
+ unmatched = set(source_patterns)
+ unexpected_dirs = []
+ self.actions = [] # each entry is tuple of
+ # (relative path, source last-modified-time, mode)
+ if self.recursive:
+ def visit(ignored, dirname, names):
+ relative_prefix = dirname[len(self.source_dir)+1:]
+ file_info = {}
+ for name in names:
+ val = os.path.join(relative_prefix, name)
+ source = os.path.join(self.source_dir, val)
+ statinfo = os.stat(source)
+ last_modified = statinfo.st_mtime
+ if stat.S_ISREG(statinfo.st_mode):
+ file_info[name] = (
+ val, last_modified, statinfo.st_mode)
+ found = set()
+ for pattern in source_patterns:
+ # val is relative to
+ matching = fnmatch.filter(file_info, pattern)
+ if matching:
+ unmatched.discard(pattern)
+ found.update(matching)
+ for name in found:
+ self.actions.append(file_info[name])
+ os.path.walk(
+ self.source_dir, visit, None)
+ else:
+ for val in source_patterns:
+ source = os.path.join(self.source_dir, val)
+ if os.path.exists(source):
+ unmatched.discard(val)
+ statinfo = os.stat(source)
+ last_modified = statinfo.st_mtime
+ if not stat.S_ISREG(statinfo.st_mode):
+ unexpected_dirs.append(source)
+ else:
+ self.actions.append(
+ (val, last_modified, statinfo.st_mode))
+ # This is supposed to be a flag so that when source files change, the
+ # recipe knows to reinstall.
+ self.options['_actions'] = repr(self.actions)
+ if unexpected_dirs:
+ self._user_error(
+ 'Expected file but found directory: %s',
+ ', '.join(unexpected_dirs))
+ if unmatched:
+ self._user_error(
+ 'No template found for these file names: %s',
+ ', '.join(unmatched))
+ interpreted = self.options.get('interpreted-options')
+ if interpreted:
+ globs = {'__builtins__': __builtins__, 'os': os, 'sys': sys}
+ locs = {'name': name, 'options': options, 'buildout': buildout,
+ 'paths': paths, 'all_paths': all_paths}
+ for value in interpreted.split():
+ if value:
+ key, expression = value.split('=', 1)
+ options[key] = str(eval(expression, globs, locs))
+ def _user_error(self, msg, *args):
+ msg = msg % args
+ self.logger.error(msg)
+ raise zc.buildout.UserError(msg)
- absname = os.path.join(here, filename)
+ def install(self):
+ already_exists = [
+ rel_path for rel_path, last_mod, st_mode in self.actions
+ if os.path.exists(
+ os.path.join(self.destination_dir, rel_path[:-3]))
+ ]
+ if already_exists:
+ self._user_error(
+ 'Destinations already exist: %s. Please make sure that '
+ 'you really want to generate these automatically. Then '
+ 'move them away.', ', '.join(already_exists))
+ for rel_path, last_mod, st_mode in self.actions:
+ source = os.path.join(self.source_dir, rel_path)
+ dest = os.path.join(self.destination_dir, rel_path[:-3])
+ mode=stat.S_IMODE(st_mode)
+ template=open(source).read()
+ template=re.sub(r"\$\{([^:]+?)\}", r"${%s:\1}" % self.name,
+ template)
+ self._create_paths(os.path.dirname(dest))
+ result=open(dest, "wt")
+ result.write(self.options._sub(template, []))
+ result.close()
+ os.chmod(dest, mode)
+ self.options.created(rel_path[:-3])
+ return self.options.created()
- if not os.path.exists(absname + '.in'):
- msg = 'No template found at %s.in.' % filename
- logger.error(msg)
- raise zc.buildout.UserError(msg)
+ def _create_paths(self, path):
+ if not os.path.exists(path):
+ self._create_paths(os.path.dirname(path))
+ os.mkdir(path)
+ self.options.created(path)
- if not update and os.path.exists(absname):
- msg = ('File %s already exists. Please make sure that you '
- 'really want to have it generated automatically. Then '
- 'move it away.' % filename)
- logger.error(msg)
- raise zc.buildout.UserError(msg)
-
- templ = string.Template(open(absname + '.in').read())
- outfile = open(absname, 'w')
- outfile.write(templ.substitute(self.options))
- outfile.close()
- return filenames
-
def update(self):
- return self.install(update=True)
+ pass
Modified: z3c.recipe.filetemplate/branches/gary-powertools/z3c/recipe/filetemplate/tests.py
===================================================================
--- z3c.recipe.filetemplate/branches/gary-powertools/z3c/recipe/filetemplate/tests.py 2009-04-29 02:07:37 UTC (rev 99568)
+++ z3c.recipe.filetemplate/branches/gary-powertools/z3c/recipe/filetemplate/tests.py 2009-04-29 02:08:24 UTC (rev 99569)
@@ -1,8 +1,9 @@
import zc.buildout.testing
+import zc.buildout.tests
from zope.testing import doctest
def setUp(test):
- zc.buildout.testing.buildoutSetUp(test)
+ zc.buildout.tests.easy_install_SetUp(test)
zc.buildout.testing.install_develop('z3c.recipe.filetemplate', test)
def test_suite():
Modified: z3c.recipe.filetemplate/branches/gary-powertools/z3c/recipe/filetemplate/tests.txt
===================================================================
--- z3c.recipe.filetemplate/branches/gary-powertools/z3c/recipe/filetemplate/tests.txt 2009-04-29 02:07:37 UTC (rev 99568)
+++ z3c.recipe.filetemplate/branches/gary-powertools/z3c/recipe/filetemplate/tests.txt 2009-04-29 02:08:24 UTC (rev 99569)
@@ -58,14 +58,12 @@
... """)
>>> print system(buildout)
- Uninstalling multiple.
- Installing evil.
- evil: /etc/passwd.in is an absolute path. File paths must be
- relative to the buildout directory.
+ evil: /etc/passwd.in is an absolute path. Paths must be relative to the buildout directory.
While:
- Installing evil.
- Error: /etc/passwd.in is an absolute path. File paths must be
- relative to the buildout directory.
+ Installing.
+ Getting section evil.
+ Initializing part evil.
+ Error: /etc/passwd.in is an absolute path. Paths must be relative to the buildout directory.
Missing template
@@ -85,13 +83,15 @@
... """)
>>> print system(buildout)
- Installing notthere.
- notthere: No template found at doesntexist.in.
+ notthere: No template found for these file names: doesntexist.in
While:
- Installing notthere.
- Error: No template found at doesntexist.in.
+ Installing.
+ Getting section notthere.
+ Initializing part notthere.
+ Error: No template found for these file names: doesntexist.in
+
Already existing file
---------------------
@@ -119,15 +119,16 @@
... """)
>>> print system(buildout)
+ Uninstalling multiple.
Installing alreadythere.
- alreadythere: File alreadyhere.txt already exists. Please make sure
- that you really want to have it generated automatically.
- Then move it away.
+ alreadythere: Destinations already exist: alreadyhere.txt.in. Please make
+ sure that you really want to generate these automatically.
+ Then move them away.
While:
Installing alreadythere.
- Error: File alreadyhere.txt already exists. Please make sure
- that you really want to have it generated automatically.
- Then move it away.
+ Error: Destinations already exist: alreadyhere.txt.in. Please make sure
+ that you really want to generate these automatically. Then move
+ them away.
Missing variables
@@ -155,4 +156,5 @@
Installing missing.
While:
Installing missing.
- Error: Missing option: missing:world
+ Error: Referenced option does not exist: missing world
+
More information about the Checkins
mailing list