[Grok-dev] need advice on testing

Uli Fouquet uli at gnufix.de
Thu May 22 22:15:13 EDT 2008

Hi there,

Brandon Craig Rhodes wrote:
> I have a concrete testing problem, which I'd really love input on from
> some of you more-experienced Grok folks.  If any of you with opinions on
> testing could spare a few moments to reply, I think that this would be a
> useful situation for examining two debates that simmer occasionally:
> first, that of unit tests against docfiles, and second, the issue of
> which testing module to use - zope.testing, versus z3c.testsetup, versus
> Nose or whatever else someone wants to suggest.

I am not a testing guru myself, so the following are only (sometimes not
very inspired) thoughts from my side.

Don't know about Nose. I think zope.testing and z3c.testsetup are not in
a 'versus' but in a 'built-upon' relation. While z3c.testsetup finds
files that contain tests, zope.testing processes them - after being
found - and turns them into something usable by a testrunner, a

Testfile --found-by------> Testfinder (z3c.testsetup/your own code)
 |                           |
 |                        driven by
 |                           |
 +-------processed-by----> Testprocessor (zope.testing)--->TestSuite


> >>> pprint(create_person({
> >>>     lastname='Smith',
> >>>     firstname='Edward',
> >>>     ssn=19876223,
> >>>     role='Employee',
> >>>     title='Research Assistant'}))
> {'accounts': [{'gid': 482,
>                'uid': 1072,
>                'username': 'esmith7'}],
>  'firstname': 'Edward',
>  'guests': [],
>  'lastname': 'Smith',
>  'lastupdate': datetime.datetime(2008, ...)
>  'role': 'Employee',
>  'ssn': 19876223,
>  'title': 'Research Assistant'}
> My voyage so far has gone something like this:
[snip: pprint is fine]
>  - But the test result is all-or-nothing; if there's any difference in
>    return values - and one return value I was just testing is 34 lines
>    long (it includes a person with several accounts and campus guests) -
>    then you just get the message "Failed example ... Expected: <this>
>    Got: <that>", where <this> and <that> are the big 34-line blocks of
>    text.  It can take quite a bit of time to find the typo that makes
>    the result different from the expected value!
>  - So, I would like the test results to be compared with a unified diff
>    that would point out the one or two lines that were wrong.  I noticed
>    that zope.testing package, which Grok test scripts seem to be based
>    on, offer a --udiff option!  But when I tried running:
>      $ bin/test --udiff
>    it made no difference at all.

This is strange. Did you run tests on a project with the doctest setup
you show below? In Grok source itself this option has no effect, because
the test setups set their own report style which overrides commandline

>  - Why does --udiff make no difference?  Maybe because I'm activating my
>    docfiles manually by creating my own instance of DocTestSuite,
>    instead of letting zope.testing.testrunner use its own specialized
>    version of a doctest runner?  That's just a wild guess; the code I
>    use to call my docfiles, because I couldn't find any other way, is:
>    def test_suite():
>        return unittest.TestSuite((
>            doctest.DocTestSuite('iamapi.api'),
>            doctest.DocTestSuite('iamapi.app'),
>            doctest.DocFileSuite('../doc/security.txt'),
>            doctest.DocFileSuite('./guests.txt',
>            optionflags=doctest.ELLIPSIS),
>            ))
>    See?  I'm creating my own instances of DocFileSuite, that probably
>    lack whatever magic allows --udiff to operate.

This looks fine to me. You should be able to use --udiff. What happens,
if you append::


This way you disable commandline report options for guests.txt, but
should at least get UDIFF-ed output.

>   But how can I make
>    zope.testing "see" my docfile, instead of turning it into a test
>    myself.

You can't. Well, you could use the DocTestFinder from
zope.testing.doctest, but in the long run, z3c.testsetup might be more
comfortable, because it also finds non-doctests etc. once in a row.

>  - Here we hit the barrier of the immensely frustrating zope.testing
>    documentation.  It's somehow thousands of lines long, and yet never
>    actually explains (that I can find) the very most basic issue it has
>    to communicate: the exact rules about how it finds tests within your
>    module!  The *only mention* it makes of how it actually finds tests
>    is one sentence describing the "tests-pattern" argument: "Tell the
>    test runner how to recognize modules or packages containing tests."
>    That's it.  No mention of what the argument's format is, or what it
>    does with the files or directories matched.  This is unbelievable.
>    Maybe I'm supposed to read the source code to know how to use it?
>    Then why have docs in the first place? :-)

This is a misunderstanding, which is not neccessarily easy to see. I
needed some time myself to figure it out. The testrunner collects tests,
but not doctests. It expects Python modules that define a `test_suite`
callable, just as you did in your testsetup above. These and only these
are tests in the sense of testrunner.

With this clarification it should be clear, why the testrunner does not
mention .txt or other formats at all. It only looks for Python sources.

>    [Edit: Nope!  I've just checked the zope.testing.testrunner.run()
>    function, and it lacks entirely any docstring, much less a useful
>    description of its arguments.  Unbelievable.]

Hmm, you tried ./bin/test --help ?

>  - I have looked briefly at z3c.testsetup, since zope.testing won't
>    share with me the secrets of how it finds tests (my question, you'll
>    recall, is how to get it to find my docfiles - maybe a .docfile or
>    .doctest extension? or are they not supported?).  I can't see that
>    z3c.testsetup even mentions docfiles in its documentation, so I'm not
>    pursuing it as an option at this point.

What documentation did you look at? The grok.zope.org docs, the
cheeseshop title page? The txt files in the sources? The word 'doctest'
appears at least 76 times in the txt docs of the package and the word
'docfile' at least six times. 

Peter Bengtson and I tried really hard to give every sort of docs
eventually needed. There is a beginners HowTo on grok.zope.org and
there's plenty of in-depth documentation in the package itself. The
grok.zope.org docs provide also links to other sources. What is still
missing is a Complete Grokkers Guide to the Testing Universe. I admit

z3c.testsetup is, BTW, what should make your own test setup above
shorter (it won't vanish) and NDIFF-ed by default. When I try the
following test with z3c.testsetup::

  A Test
  :Test-Layer: unit

     >>> from pprint import pprint
     >>> a = dict(c=1, a=1, b='9345378399599355', 
     ...          a1='1987275732987529', b1='0989879879879878787989')
     >>> pprint(a)
     {'a': 1,
      'a1': '1987275732987529',
      'b': '9345378398599355',
      'b1': '0989879879879878787989',
      'c': 1}

There is one digit wrong in the expected output. Which one?

Failed example:
Differences (ndiff with -expected +actual):
      {'a': 1,
       'a1': '1987275732987529',
    -  'b': '9345378398599355',
    ?                 ^
    +  'b': '9345378399599355',
    ?                 ^
       'b1': '0989879879879878787989',
       'c': 1}

Ah, now I can spot it. With coloured output (testrunner option '-c')
this becomes even more visible as the false output will appear in bloody
red and other lines in friendly green.

The testsetup here was (tests.py)::

  import z3c.testsetup
  test_suite = z3c.testsetup.register_all_tests('sample')

where 'sample' is the dotted name of the package I want to be searched
for tests (including all subdirectories). Every .txt and .rst file
therein will be examined, scanned for a marker (`:Test-Layer: unit`,
`:Test-Layer: functional` or `:Test-Layer: python`) and fed to the
testrunner afterwards.

Given your own example above, which collects different modules and .txt
files, which seem to be placed partially outside a Python package, a
plain z3c.testsetup will become difficult at current state. But your use
case could be worth an improvement to make it capable of your use case
as well.

>  - Okay, now I need you unittest guys to weigh in.  Every month or two
>    we get to hear several of you guys talk about how doctests, while
>    maybe good for sanity-checking actual documentation, are unreadable
>    and unmaintainable for doing actual unit testing of function
>    behaviors.  This is your chance, unittest fans!  While I sit here
>    stymied and frustrated with my inability to get a reasonable doctest
>    comparison between two compound return values, you can step in and
>    save the day: how, in a readable unit test, would I test for the
>    return value in the above example in such a way that I could quickly
>    be shown the essential difference between the struct that I was
>    expecting and the one that I received?
>  - Meanwhile, do you zope.testing.testrunner fans - who talk about how
>    advanced it is, and how we ought to keep using it rather than moving
>    Grok to using Nose or something else more widely used testing system
>    - have any secrets to share about how I could use it to run my
>    docfiles with --udiff actually working?

>  - I read through the zope.testing.testrunner documentation one last
>    time while typing the above, and I notice that, way up at the top, it
>    advertises a "doctest.py" file.  Using the "DocTestSuite" and
>    "DocFileSuite" objects it defines in place of the unittest/doctest
>    ones, I am able to suddenly see unified diffs!  Yay!

Congrats :-) I wonder, however, why DocFileSuite didn't work. Testing it
here, it worked.

> But having typed all of this, I'll go ahead and hit "send". :-) Two of
> my questions, I think, are still valid: first, how can I do this more
> easily using unittests; and, second, how can I get my docfiles (if I
> were to keep using them) to be auto-detected rather than having to keep
> a dratted list of them in a separate testing file?

To be honest, I do not understand question one. What exactly do you want
to be done more easily?

For question two: use z3c.testsetup.

> And, I hope that the above is useful for showing how a somewhat-Python-
> savvy, but still rather new to Grok, user processes the frustrating
> steps involved in getting testing configured well.  I hope that by
> preserving this in email, we can go in and try to smooth out each rough
> spot so that the next users to come along - who might not be as
> dedicated to continuing to use Grok as I am, but just trying it out -
> have an easier time of things.
> I understand, of course, that we're most all volunteers on this effort,
> and that even the bits of zope.testing.testrunner documentation that do
> exist are there because someone volunteered to write them when they
> could have been doing something else.  But I'm going to leave in place,
> un-edited, the frustration expressed above, because I think that it
> captures a moment that could have made someone who was just trying out
> Grok throw up their hands and try something else.

I agree with you in one important point: there is still too little
consistent testing documentation. Therefore I am glad, you sent this.
What, in the end, _is_ a testrunner? What role plays 'zope.testing'?
What kind of tests are available? What are good practices in testing? We
need an extensive tutorial for that.

Kind regards,


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: Dies ist ein digital signierter Nachrichtenteil
Url : http://mail.zope.org/pipermail/grok-dev/attachments/20080523/e13e099e/attachment.bin

More information about the Grok-dev mailing list