[Grok-dev] need advice on testing

Brandon Craig Rhodes brandon at rhodesmill.org
Thu May 22 16:04:44 EDT 2008

I have a concrete testing problem, which I'd really love input on from
some of you more-experienced Grok folks.  If any of you with opinions on
testing could spare a few moments to reply, I think that this would be a
useful situation for examining two debates that simmer occasionally:
first, that of unit tests against docfiles, and second, the issue of
which testing module to use - zope.testing, versus z3c.testsetup, versus
Nose or whatever else someone wants to suggest.

I have XML-RPC functions which are called by other applications.
Because the whole idea is that XML is supposed to be self-documenting, I
avoided designing my calls like:

 create_person('Smith', 'Edward', 19876223, 'Employee', 'Research Assistant')

because who, after all, will remember which order the arguments go in?
Instead I use an XML-RPC "struct" (represented in Python as a dict with
only strings as keys) to pass the arguments.  I then return some structs
that describe the newly created object's properties.  So a doctest for
this routine, for example, might look something like:

>>> pprint(create_person({
>>>     lastname='Smith',
>>>     firstname='Edward',
>>>     ssn=19876223,
>>>     role='Employee',
>>>     title='Research Assistant'}))
{'accounts': [{'gid': 482,
               'uid': 1072,
               'username': 'esmith7'}],
 'firstname': 'Edward',
 'guests': [],
 'lastname': 'Smith',
 'lastupdate': datetime.datetime(2008, ...)
 'role': 'Employee',
 'ssn': 19876223,
 'title': 'Research Assistant'}

My voyage so far has gone something like this:

 - Doing this with a normal doctest is a nightmare, since the dictionary
   can print out in any order.

 - Doing this with a doctest with pprint() wrapped around the call, as
   shown above, works pretty well, since that stabilizes the order of
   the values returned, and puts them on separate lines instead of
   leaving them all scrunched up on one line.  The test is readable.

 - But the test result is all-or-nothing; if there's any difference in
   return values - and one return value I was just testing is 34 lines
   long (it includes a person with several accounts and campus guests) -
   then you just get the message "Failed example ... Expected: <this>
   Got: <that>", where <this> and <that> are the big 34-line blocks of
   text.  It can take quite a bit of time to find the typo that makes
   the result different from the expected value!

 - So, I would like the test results to be compared with a unified diff
   that would point out the one or two lines that were wrong.  I noticed
   that zope.testing package, which Grok test scripts seem to be based
   on, offer a --udiff option!  But when I tried running:

     $ bin/test --udiff

   it made no difference at all.

 - Why does --udiff make no difference?  Maybe because I'm activating my
   docfiles manually by creating my own instance of DocTestSuite,
   instead of letting zope.testing.testrunner use its own specialized
   version of a doctest runner?  That's just a wild guess; the code I
   use to call my docfiles, because I couldn't find any other way, is:

   def test_suite():
       return unittest.TestSuite((

   See?  I'm creating my own instances of DocFileSuite, that probably
   lack whatever magic allows --udiff to operate.  But how can I make
   zope.testing "see" my docfile, instead of turning it into a test

 - Here we hit the barrier of the immensely frustrating zope.testing
   documentation.  It's somehow thousands of lines long, and yet never
   actually explains (that I can find) the very most basic issue it has
   to communicate: the exact rules about how it finds tests within your
   module!  The *only mention* it makes of how it actually finds tests
   is one sentence describing the "tests-pattern" argument: "Tell the
   test runner how to recognize modules or packages containing tests."
   That's it.  No mention of what the argument's format is, or what it
   does with the files or directories matched.  This is unbelievable.
   Maybe I'm supposed to read the source code to know how to use it?
   Then why have docs in the first place? :-)

   [Edit: Nope!  I've just checked the zope.testing.testrunner.run()
   function, and it lacks entirely any docstring, much less a useful
   description of its arguments.  Unbelievable.]

 - I have looked briefly at z3c.testsetup, since zope.testing won't
   share with me the secrets of how it finds tests (my question, you'll
   recall, is how to get it to find my docfiles - maybe a .docfile or
   .doctest extension? or are they not supported?).  I can't see that
   z3c.testsetup even mentions docfiles in its documentation, so I'm not
   pursuing it as an option at this point.

 - Okay, now I need you unittest guys to weigh in.  Every month or two
   we get to hear several of you guys talk about how doctests, while
   maybe good for sanity-checking actual documentation, are unreadable
   and unmaintainable for doing actual unit testing of function
   behaviors.  This is your chance, unittest fans!  While I sit here
   stymied and frustrated with my inability to get a reasonable doctest
   comparison between two compound return values, you can step in and
   save the day: how, in a readable unit test, would I test for the
   return value in the above example in such a way that I could quickly
   be shown the essential difference between the struct that I was
   expecting and the one that I received?

 - Meanwhile, do you zope.testing.testrunner fans - who talk about how
   advanced it is, and how we ought to keep using it rather than moving
   Grok to using Nose or something else more widely used testing system
   - have any secrets to share about how I could use it to run my
   docfiles with --udiff actually working?

 - I read through the zope.testing.testrunner documentation one last
   time while typing the above, and I notice that, way up at the top, it
   advertises a "doctest.py" file.  Using the "DocTestSuite" and
   "DocFileSuite" objects it defines in place of the unittest/doctest
   ones, I am able to suddenly see unified diffs!  Yay!

But having typed all of this, I'll go ahead and hit "send". :-) Two of
my questions, I think, are still valid: first, how can I do this more
easily using unittests; and, second, how can I get my docfiles (if I
were to keep using them) to be auto-detected rather than having to keep
a dratted list of them in a separate testing file?

And, I hope that the above is useful for showing how a somewhat-Python-
savvy, but still rather new to Grok, user processes the frustrating
steps involved in getting testing configured well.  I hope that by
preserving this in email, we can go in and try to smooth out each rough
spot so that the next users to come along - who might not be as
dedicated to continuing to use Grok as I am, but just trying it out -
have an easier time of things.

I understand, of course, that we're most all volunteers on this effort,
and that even the bits of zope.testing.testrunner documentation that do
exist are there because someone volunteered to write them when they
could have been doing something else.  But I'm going to leave in place,
un-edited, the frustration expressed above, because I think that it
captures a moment that could have made someone who was just trying out
Grok throw up their hands and try something else.

Brandon Craig Rhodes   brandon at rhodesmill.org   http://rhodesmill.org/brandon

More information about the Grok-dev mailing list