[Grok-dev] need advice on testing

[Christian Theune]

Hi,

(sorry that I snipped around in your text)

On Thu, May 22, 2008 at 04:04:44PM -0400, Brandon Craig Rhodes wrote:
>  - Doing this with a doctest with pprint() wrapped around the call, as
>    shown above, works pretty well, since that stabilizes the order of
>    the values returned, and puts them on separate lines instead of
>    leaving them all scrunched up on one line.  The test is readable.

That's the way I do it.

>  - So, I would like the test results to be compared with a unified diff
>    that would point out the one or two lines that were wrong.  I noticed
>    that zope.testing package, which Grok test scripts seem to be based
>    on, offer a --udiff option!  But when I tried running:
> 
>      $ bin/test --udiff
> 
>    it made no difference at all.

I prefer NDIFF, but that's a matter of taste.

>  - Why does --udiff make no difference?  Maybe because I'm activating my
>    docfiles manually by creating my own instance of DocTestSuite,
>    instead of letting zope.testing.testrunner use its own specialized
>    version of a doctest runner?  That's just a wild guess; the code I
>    use to call my docfiles, because I couldn't find any other way, is:
> 
>    def test_suite():
>        return unittest.TestSuite((
>            doctest.DocTestSuite('iamapi.api'),
>            doctest.DocTestSuite('iamapi.app'),
>            doctest.DocFileSuite('../doc/security.txt'),
>            doctest.DocFileSuite('./guests.txt',
>            optionflags=doctest.ELLIPSIS),
>            ))

For a short-term solution, you can either in your specific test example use:

>>> pprint(...)     # doctest: +REPORT_NDIFF

or  in the DocFileSuite) use the following option flags:

    optionflags=doctest.ELLIPSIS|doctest.REPORT_NDIFF

>    See?  I'm creating my own instances of DocFileSuite, that probably
>    lack whatever magic allows --udiff to operate.  But how can I make
>    zope.testing "see" my docfile, instead of turning it into a test
>    myself.

The specific issue is that the optionflags argument overrides whatever you
have set on the command line because the test runner sets those as global
options for the doctest module. However, the optionflags, when set, do not
consider any pre-set global options.

>  - Here we hit the barrier of the immensely frustrating zope.testing
>    documentation.  It's somehow thousands of lines long, and yet never
>    actually explains (that I can find) the very most basic issue it has
>    to communicate: the exact rules about how it finds tests within your
>    module!  The *only mention* it makes of how it actually finds tests
>    is one sentence describing the "tests-pattern" argument: "Tell the
>    test runner how to recognize modules or packages containing tests."
>    That's it.  No mention of what the argument's format is, or what it
>    does with the files or directories matched.  This is unbelievable.
>    Maybe I'm supposed to read the source code to know how to use it?
>    Then why have docs in the first place? :-)
> 
>    [Edit: Nope!  I've just checked the zope.testing.testrunner.run()
>    function, and it lacks entirely any docstring, much less a useful
>    description of its arguments.  Unbelievable.]

Please see my refactoring of zope.testing.testrunner on zope.testing's trunk.
Things should become more clear. I'm still cleaning it up though.

>  - I have looked briefly at z3c.testsetup, since zope.testing won't
>    share with me the secrets of how it finds tests (my question, you'll
>    recall, is how to get it to find my docfiles - maybe a .docfile or
>    .doctest extension? or are they not supported?).  I can't see that
>    z3c.testsetup even mentions docfiles in its documentation, so I'm not
>    pursuing it as an option at this point.

In the future, z3c.testsetup can become a very simple plugin for zope.testing.

>  - Okay, now I need you unittest guys to weigh in.  Every month or two
>    we get to hear several of you guys talk about how doctests, while
>    maybe good for sanity-checking actual documentation, are unreadable
>    and unmaintainable for doing actual unit testing of function
>    behaviors.  This is your chance, unittest fans!  While I sit here
>    stymied and frustrated with my inability to get a reasonable doctest
>    comparison between two compound return values, you can step in and
>    save the day: how, in a readable unit test, would I test for the
>    return value in the above example in such a way that I could quickly
>    be shown the essential difference between the struct that I was
>    expecting and the one that I received?

See above modulo the current zope.testing annoyances. ;)
 
>  - Meanwhile, do you zope.testing.testrunner fans - who talk about how
>    advanced it is, and how we ought to keep using it rather than moving
>    Grok to using Nose or something else more widely used testing system
>    - have any secrets to share about how I could use it to run my
>    docfiles with --udiff actually working?

Currently no, but this specific problem (how doctest global options are set)
is on my list to clean up.

>  - I read through the zope.testing.testrunner documentation one last
>    time while typing the above, and I notice that, way up at the top, it
>    advertises a "doctest.py" file.  Using the "DocTestSuite" and
>    "DocFileSuite" objects it defines in place of the unittest/doctest
>    ones, I am able to suddenly see unified diffs!  Yay!

I'm a bit surprised about that. I know that zope.testing.doctest is a bit more
current than Python's (after all it came out of the Zope community) with a few
more features in there (like footnotes).

> But having typed all of this, I'll go ahead and hit "send". :-) Two of
> my questions, I think, are still valid: first, how can I do this more
> easily using unittests; and, second, how can I get my docfiles (if I
> were to keep using them) to be auto-detected rather than having to keep
> a dratted list of them in a separate testing file?

Better test discovery and getting rid of the need of test_suite() is another
refactoring/feature I want to perform on zope.testing.

> And, I hope that the above is useful for showing how a somewhat-Python-
> savvy, but still rather new to Grok, user processes the frustrating
> steps involved in getting testing configured well.  I hope that by
> preserving this in email, we can go in and try to smooth out each rough
> spot so that the next users to come along - who might not be as
> dedicated to continuing to use Grok as I am, but just trying it out -
> have an easier time of things.
> 
> I understand, of course, that we're most all volunteers on this effort,
> and that even the bits of zope.testing.testrunner documentation that do
> exist are there because someone volunteered to write them when they
> could have been doing something else.  But I'm going to leave in place,
> un-edited, the frustration expressed above, because I think that it
> captures a moment that could have made someone who was just trying out
> Grok throw up their hands and try something else.

Test runner is currently so frustrating because the code is extremely
underengineered.

Christian

PS: I like your style of writing, but I was really scared by the amount of
text. ;)

-- 
Christian Theune · ct at gocept.com
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1
Zope and Plone consulting and development