[Checkins] SVN: manuel/trunk/ merge in branches/benji-add-test-cases/ (plus a change note)
Benji York
benji at zope.com
Fri Jul 10 09:13:50 EDT 2009
Log message for revision 101789:
merge in branches/benji-add-test-cases/ (plus a change note)
Changed:
U manuel/trunk/CHANGES.txt
UU manuel/trunk/src/index.txt
U manuel/trunk/src/manuel/README.txt
U manuel/trunk/src/manuel/__init__.py
A manuel/trunk/src/manuel/testcase.py
U manuel/trunk/src/manuel/testing.py
U manuel/trunk/src/manuel/tests.py
-=-
Modified: manuel/trunk/CHANGES.txt
===================================================================
--- manuel/trunk/CHANGES.txt 2009-07-10 06:19:04 UTC (rev 101788)
+++ manuel/trunk/CHANGES.txt 2009-07-10 13:13:49 UTC (rev 101789)
@@ -1,6 +1,13 @@
CHANGES
=======
+1.0.0b2 (2009-07-10)
+--------------------
+
+- add the ability to identify and run subsets of documents (using the -t switch
+ of zope.testing's testrunner for example)
+
+
1.0.0b1 (2009-06-24)
--------------------
Modified: manuel/trunk/src/index.txt
===================================================================
--- manuel/trunk/src/index.txt 2009-07-10 06:19:04 UTC (rev 101788)
+++ manuel/trunk/src/index.txt 2009-07-10 13:13:49 UTC (rev 101789)
@@ -20,7 +20,7 @@
def my_func(bar):
return foo.baz(bar)
-Incidentally, the implementation of :mod:`manuel.codeblock` is only 22 lines of
+Incidentally, the implementation of :mod:`manuel.codeblock` is only 23 lines of
code.
The plug-ins included in Manuel make good examples while being quite useful in
@@ -63,7 +63,11 @@
:ref:`manuel.isolation <isolation>`
makes it easier to have test isolation in doctests
+:ref:`manuel.testcase <testcase>`
+ identify parts of tests as individual test cases so they can be run
+ independently
+
.. reset-globs
.. _getting-started:
@@ -79,7 +83,8 @@
import manuel.doctest
import manuel.footnote
- m = manuel.doctest.Manuel() + manuel.footnote.Manuel()
+ m = manuel.doctest.Manuel()
+ m += manuel.footnote.Manuel()
You would then pass the Manuel instance to a :class:`manuel.testing.TestSuite`,
including the names of documents you want to process:
@@ -213,19 +218,26 @@
================
When writing documentation the need often arrises to describe the contents of
-files or other non-python information. You may also want to put that
+files or other non-Python information. You may also want to put that
information under test. :mod:`manuel.capture` helps with that.
-If you were writing the problems for a programming contest, you might want to
-describe the input and output files for each challenge.
+For example, if you were writing the problems for a programming contest, you
+might want to describe the input and output files for each challenge, but you
+want to be sure that your examples are correct.
-You would then show the contestant the expected output of their program. But
-you want to be sure that your examples are correct.
-
To do that you might write your document like this:
::
+ Challenge 1
+ ===========
+
+ Write a program that sorts the numbers in a file.
+
+
+ Example
+ -------
+
Given this example input file::
6
@@ -248,11 +260,11 @@
20
65
- .. -> expected
+ .. -> output
- >>> correct = '\n'.join(
- ... map(str, sorted(map(int, input.splitlines())))) + '\n'
- >>> expected == correct
+ >>> input_lines = input.splitlines()
+ >>> correct = '\n'.join(map(str, sorted(map(int, input_lines)))) + '\n'
+ >>> output == correct
True
.. -> source
@@ -276,9 +288,38 @@
Of course, lines that start with ".. " are reST comments, so when the document
is rendered with docutils or Sphinx, the tests will dissapear and only the
-intended document contents will remain.
+intended document contents will remain. Like so::
+ Challenge 1
+ ===========
+ Write a program that sorts the numbers in a file.
+
+
+ Example
+ -------
+
+ Given this example input file::
+
+ 6
+ 1
+ 8
+ 20
+ 11
+ 65
+ 2
+
+ Your program should generare this output file::
+
+ 1
+ 2
+ 6
+ 8
+ 11
+ 20
+ 65
+
+
.. reset-globs
.. _code-blocks:
@@ -492,7 +533,7 @@
>>> document.process_with(m, globs={})
>>> print document.formatted()
-It is possible to reference more than one footnote on a single line.
+It is also possible to reference more than one footnote on a single line.
::
@@ -740,6 +781,188 @@
>>> print document.formatted(),
+.. reset-globs
+.. _testcase:
+
+Identifying Test Cases
+======================
+
+If you want parts of a document to be individually accessible as test cases (to
+be able to run just a particular subset of them, for example), a parser can
+create a region that marks the beginning of a new test case.
+
+Two ways of identifying test cases are included in :mod:`manuel.testcase`:
+
+1. by section headings
+
+2. by explicit ".. test-case: NAME" markers.
+
+Grouping Tests by Heading
+-------------------------
+
+::
+
+ First Section
+ =============
+
+ Some prose.
+
+ >>> print 'first test case'
+
+ Some more prose.
+
+ >>> print 'still in the first test case'
+
+ Second Section
+ ==============
+
+ Even more prose.
+
+ >>> print 'second test case'
+
+.. -> source
+
+ >>> import manuel
+ >>> import manuel.testcase
+ >>> document = manuel.Document(source)
+ >>> m = manuel.testcase.SectionManuel()
+ >>> m += manuel.doctest.Manuel()
+ >>> document.process_with(m, globs={})
+ >>> print document.formatted(),
+ File "<memory>"...
+ Failed example:
+ print 'first test case'
+ Expected nothing
+ Got:
+ first test case
+ File "<memory>"...
+ Failed example:
+ print 'still in the first test case'
+ Expected nothing
+ Got:
+ still in the first test case
+ File "<memory>"...
+ Failed example:
+ print 'second test case'
+ Expected nothing
+ Got:
+ second test case
+
+.. now lets see if the regions are grouped as we expect
+
+ >>> import manuel.testing
+ >>> for regions in manuel.testing.group_regions_by_test_case(document):
+ ... print (regions.location, regions.id)
+ ('<memory>', None)
+ ('<memory>', 'First Section')
+ ('<memory>', 'Second Section')
+
+Given the above document, if you're using zope.testing's testrunner (located in bin/test), you could run just the tests in the second section with this command::
+
+ bin/test -t "file-name.txt:Second Section"
+
+Or, exploiting the fact that -t does a regex search (as opposed to a match)::
+
+ bin/test -t file-name.txt:Second
+
+
+Grouping Tests Explicitly
+-------------------------
+
+If you would like to identify test cases separately from sections, you can
+identify them with a marker::
+
+ First Section
+ =============
+
+ The following test will be in a test case that is not individually
+ identifiable.
+
+ >>> print 'first test case (unidentified)'
+
+ Some more prose.
+
+ .. test-case: first-named-test-case
+
+ >>> print 'first identified test case'
+
+
+ Second Section
+ ==============
+
+ The test case markers don't have to immediately proceed a test.
+
+ .. test-case: second-named-test-case
+
+ Even more prose.
+
+ >>> print 'second identified test case'
+
+.. -> source
+
+ >>> document = manuel.Document(source)
+ >>> m = manuel.testcase.MarkerManuel()
+ >>> m += manuel.doctest.Manuel()
+ >>> document.parse_with(m)
+ >>> for regions in manuel.testing.group_regions_by_test_case(document):
+ ... print regions.location, regions.id
+ <memory> None
+ <memory> first-named-test-case
+ <memory> second-named-test-case
+
+Again, given the above document and zope.testing, you could run just the second
+set of tests with this command::
+
+ bin/test -t file-name.txt:second-named-test-case
+
+Or, exploiting the fact that -t does a regex search again::
+
+ bin/test -t file-name.txt:second
+
+Even though the tests are individually accessable doesn't mean that they can't
+all be run at the same time::
+
+ bin/test -t file-name.txt
+
+Also, if you create a hierarchy of names, you can run groups of tests at a
+time. For example, lets say that you append "-important" to all your really
+important tests, you could then run the important tests for a single document
+like so::
+
+ bin/test -t 'file-name.txt:.*-important$'
+
+or all the "important" tests no matter what file they are in::
+
+ bin/test -t '-important$'
+
+Both Methods
+------------
+
+You can also combine more than one test case identification method if you want.
+Here's an example of building a Manuel stack that has doctests and both flavors
+of test case identification:
+
+.. code-block:: python
+
+ import manuel.doctest
+ import manuel.testcase
+
+ m = manuel.doctest.Manuel()
+ m += manuel.testcase.SectionManuel()
+ m += manuel.testcase.MarkerManuel()
+
+.. make sure above finds all the test cases appropriately
+
+ >>> document.parse_with(m)
+ >>> for regions in manuel.testing.group_regions_by_test_case(document):
+ ... print regions.location, regions.id
+ <memory> None
+ <memory> First Section
+ <memory> first-named-test-case
+ <memory> Second Section
+ <memory> second-named-test-case
+
+
Further Reading
===============
Property changes on: manuel/trunk/src/index.txt
___________________________________________________________________
Deleted: svn:mergeinfo
-
Modified: manuel/trunk/src/manuel/README.txt
===================================================================
--- manuel/trunk/src/manuel/README.txt 2009-07-10 06:19:04 UTC (rev 101788)
+++ manuel/trunk/src/manuel/README.txt 2009-07-10 13:13:49 UTC (rev 101789)
@@ -555,3 +555,23 @@
Additional Information:
a = 1
b = 2
+
+
+Defining Test Cases
+===================
+
+If you want parts of a document to be accessable individually as test cases (to
+be able to run just a particular part of a document, for example), a parser can
+create a region that marks the beginning of a new test case.
+
+.. code-block:: python
+
+ new_test_case_regex = re.compile(r'^.. new-test-case: \w+', re.MULTILINE)
+
+ def parse(document):
+ for region in document.find_regions(new_test_case_regex):
+ document.claim_region(region)
+ id = region.start_match.group(1)
+ region.parsed = manuel.testing.TestCaseMarker(id)
+
+XXX finish this section
Modified: manuel/trunk/src/manuel/__init__.py
===================================================================
--- manuel/trunk/src/manuel/__init__.py 2009-07-10 06:19:04 UTC (rev 101788)
+++ manuel/trunk/src/manuel/__init__.py 2009-07-10 13:13:49 UTC (rev 101789)
@@ -129,13 +129,64 @@
end += 1
return end
-class Document(object):
- def __init__(self, source, location='<memory>'):
+class RegionContainer(object):
+
+ location = '<memory>'
+ id = None
+
+ def __init__(self):
+ self.regions = []
+
+ def parse_with(self, m):
+ for parser in sort_handlers(m.parsers):
+ parser(self)
+
+ def evaluate_with(self, m, globs):
+ globs = GlobWrapper(globs)
+ for region in list(self):
+ for evaluater in sort_handlers(m.evaluaters):
+ evaluater(region, self, globs)
+
+ def format_with(self, m):
+ for formatter in sort_handlers(m.formatters):
+ formatter(self)
+
+ def process_with(self, m, globs):
+ """Run all phases of document processing using a Manuel instance.
+ """
+ self.parse_with(m)
+ self.evaluate_with(m, globs)
+ self.format_with(m)
+
+ def formatted(self):
+ """Return a string of all non-boolean-false formatted regions.
+ """
+ return ''.join(region.formatted for region in self if region.formatted)
+
+ def append(self, region):
+ self.regions.append(region)
+
+ def __iter__(self):
+ """Iterate over all regions of the document.
+ """
+ return iter(self.regions)
+
+ def __nonzero__(self):
+ return bool(self.regions)
+
+
+class Document(RegionContainer):
+
+ def __init__(self, source, location=None):
+ RegionContainer.__init__(self)
+
+ if location is not None:
+ self.location = location
+
self.source = newlineify(source)
- self.location = location
- self.regions = [Region(lineno=1, source=source)]
+ self.append(Region(lineno=1, source=source))
self.shadow_regions = []
def find_regions(self, start, end=None):
@@ -255,38 +306,7 @@
def insert_region_after(self, marker_region, new_region):
self.insert_region('after', marker_region, new_region)
- def parse_with(self, m):
- for parser in sort_handlers(m.parsers):
- parser(self)
- def evaluate_with(self, m, globs):
- globs = GlobWrapper(globs)
- for region in list(self):
- for evaluater in sort_handlers(m.evaluaters):
- evaluater(region, self, globs)
-
- def format_with(self, m):
- for formatter in sort_handlers(m.formatters):
- formatter(self)
-
- def process_with(self, m, globs):
- """Run all phases of document processing using a Manuel instance.
- """
- self.parse_with(m)
- self.evaluate_with(m, globs)
- self.format_with(m)
-
- def formatted(self):
- """Return a string of all non-boolean-false formatted regions.
- """
- return ''.join(region.formatted for region in self if region.formatted)
-
- def __iter__(self):
- """Iterate over all regions of the document.
- """
- return iter(self.regions)
-
-
class Manuel(object):
def __init__(self, parsers=None, evaluaters=None, formatters=None):
Copied: manuel/trunk/src/manuel/testcase.py (from rev 101788, manuel/branches/benji-add-test-cases/src/manuel/testcase.py)
===================================================================
--- manuel/trunk/src/manuel/testcase.py (rev 0)
+++ manuel/trunk/src/manuel/testcase.py 2009-07-10 13:13:49 UTC (rev 101789)
@@ -0,0 +1,45 @@
+import manuel
+import manuel.testing
+import re
+import string
+import textwrap
+
+punctuation = re.escape(string.punctuation)
+SECTION_TITLE = re.compile(r'^.+$', re.MULTILINE)
+SECTION_UNDERLINE = re.compile('^[' + punctuation + ']+\s*$', re.MULTILINE)
+MARKER = re.compile(r'^.. test-case: (\S+)', re.MULTILINE)
+
+def find_section_headers(document):
+ for region in document.find_regions(SECTION_TITLE, SECTION_UNDERLINE):
+ # regions that represent titles will have two lines
+ if region.source.count('\n') != 2:
+ continue
+
+ title, underline = region.source.splitlines()
+
+ # the underline has to be the same length as or longer than the title
+ if len(underline) < len(title):
+ continue
+
+ # ok, this is a region we want
+ document.claim_region(region)
+
+ test_case_name = title.strip()
+ region.parsed = manuel.testing.TestCaseMarker(test_case_name)
+
+
+def find_markers(document):
+ for region in document.find_regions(MARKER):
+ document.claim_region(region)
+ test_case_name = region.start_match.group(1)
+ region.parsed = manuel.testing.TestCaseMarker(test_case_name)
+
+
+class SectionManuel(manuel.Manuel):
+ def __init__(self):
+ manuel.Manuel.__init__(self, [find_section_headers])
+
+
+class MarkerManuel(manuel.Manuel):
+ def __init__(self):
+ manuel.Manuel.__init__(self, [find_markers])
Modified: manuel/trunk/src/manuel/testing.py
===================================================================
--- manuel/trunk/src/manuel/testing.py 2009-07-10 06:19:04 UTC (rev 101788)
+++ manuel/trunk/src/manuel/testing.py 2009-07-10 13:13:49 UTC (rev 101789)
@@ -1,3 +1,4 @@
+import itertools
import manuel
import os.path
import unittest
@@ -7,23 +8,22 @@
__all__ = ['TestSuite']
+class TestCaseMarker(object):
+
+ def __init__(self, id=''):
+ self.id = id
+
+
class TestCase(unittest.TestCase):
- def __init__(self, m, document, setUp=None, tearDown=None, globs=None):
+ def __init__(self, m, regions, globs, setUp=None, tearDown=None):
unittest.TestCase.__init__(self)
self.manuel = m
- self.document = document
+ self.regions = regions
+ self.globs = globs
self.setUp_func = setUp
self.tearDown_func = tearDown
- if globs is None:
- self.globs = {}
- else:
- self.globs = globs
- # we want to go ahead and do the parse phase so the countTestCases
- # method can get a good idea of how many tests there are
- document.parse_with(m)
-
def setUp(self):
if self.setUp_func is not None:
self.setUp_func(self)
@@ -33,9 +33,9 @@
self.tearDown_func(self)
def runTest(self):
- self.document.evaluate_with(self.manuel, self.globs)
- self.document.format_with(self.manuel)
- results = [r.formatted for r in self.document if r.formatted]
+ self.regions.evaluate_with(self.manuel, self.globs)
+ self.regions.format_with(self.manuel)
+ results = [r.formatted for r in self.regions if r.formatted]
if results:
DIVIDER = '-'*70 + '\n'
raise doctest.DocTestFailureException(
@@ -44,21 +44,65 @@
def debug(self):
self.setUp()
self.manuel.debug = True
- self.document.evaluate_with(self.manuel, self.globs)
+ self.regions.evaluate_with(self.manuel, self.globs)
self.tearDown()
def countTestCases(self):
- return len([r for r in self.document if r.parsed])
+ return len([r for r in self.regions if r.parsed])
def id(self):
- return self.document.location
+ return self.regions.id
def shortDescription(self):
- return "Manuel Test: " + self.document.location
+ if self.regions.id:
+ return self.regions.location + ':' + self.regions.id
+ else:
+ return self.regions.location
__str__ = __repr__ = shortDescription
+def group_regions_by_test_case(document):
+ """Generate groups of regions according to which testcase they belong"""
+ document_iter = iter(document)
+ marker = None
+ while True:
+ accumulated_regions = manuel.RegionContainer()
+ while True:
+ region = None # being defensive
+ try:
+ region = next(document_iter)
+ except StopIteration:
+ if not accumulated_regions:
+ break
+ else:
+ accumulated_regions.append(region)
+
+ if not isinstance(region.parsed, TestCaseMarker):
+ continue
+
+ # we just found a test case marker or hit the end of the
+ # document
+
+ # figure out what this test case's ID is
+ accumulated_regions.location = document.location
+ if marker is not None and marker.parsed.id:
+ accumulated_regions.id = marker.parsed.id
+
+ yield accumulated_regions
+ marker = region
+ break
+
+ # if there are no more regions, stop
+ try:
+ region = next(document_iter)
+ except StopIteration:
+ break
+
+ # put the region we peeked at back so the inner loop can consume it
+ document_iter = itertools.chain([region], document_iter)
+
+
def TestSuite(m, *paths, **kws):
"""A unittest suite that processes files with Manuel
@@ -83,6 +127,7 @@
"""
suite = unittest.TestSuite()
+ globs = kws.pop('globs', {})
# walk up the stack frame to find the module that called this function
for depth in range(2, 5):
@@ -95,10 +140,15 @@
for path in paths:
if os.path.isabs(path):
- abs_path = path
+ abs_path = os.path.normpath(path)
else:
- abs_path = doctest._module_relative_path(calling_module, path)
+ abs_path = os.path.abspath(
+ doctest._module_relative_path(calling_module, path))
+
document = manuel.Document(open(abs_path).read(), location=abs_path)
- suite.addTest(TestCase(m, document, **kws))
+ document.parse_with(m)
+ for regions in group_regions_by_test_case(document):
+ suite.addTest(TestCase(m, regions, globs, **kws))
+
return suite
Modified: manuel/trunk/src/manuel/tests.py
===================================================================
--- manuel/trunk/src/manuel/tests.py 2009-07-10 06:19:04 UTC (rev 101788)
+++ manuel/trunk/src/manuel/tests.py 2009-07-10 13:13:49 UTC (rev 101789)
@@ -4,6 +4,7 @@
import manuel.codeblock
import manuel.doctest
import manuel.ignore
+import manuel.testcase
import manuel.testing
import os.path
import re
@@ -32,8 +33,6 @@
tests = ['../index.txt', 'table-example.txt', 'README.txt', 'bugs.txt']
- tests = map(get_abs_path, tests)
-
m = manuel.ignore.Manuel()
m += manuel.doctest.Manuel(optionflags=optionflags, checker=checker)
m += manuel.codeblock.Manuel()
More information about the Checkins
mailing list