[Grok-dev] Re: RFC: Grokking tests
faassen at startifact.com
Thu Mar 29 17:50:01 EDT 2007
I've dug through the code a bit more, and I have some comments:
* I'm trying to understand how tests are being grokked. Basically you've
automated the old way we collected tests by using grok's module grokker.
Our testing strategy is Grok is very special as we have modules that
have test code as well as code to grok. This therefore looks more like
internal code to help grok self-test than code that is also applicable
to the outside world. Can you explain how external users use this? Or
was the goal to automate grok's internal testing strategy?
* Instead I'd expect people saying stuff like this:
.. support for setUp, tearDown and globals here ... :)
and this being picked up by the grokker. That's an extra place to
register the tests for grokking, true, but it would replace the need for
test_suite. unittest.TestCase subclasses could be grokked straight away,
even. Instead, you still require people to write a test_suite() function
and call grok_tests() manually. It'd be nice if we could do away with
this entirely. I realize that makes it harder to use the test runner's
existing scanning procedure.
* I feel uncomfortable that the grokker is just trying to make a module
into a doctest and registers them whenever possible.
* we need to refactor things so we can have several, separate grokking
phases. I have the impression testing.py right now has to do too much.
(grok_tree in particular)
* is the grok.doctest directive being used anywhere? I am grepping but
can't seem to find any case where it's being used. What am I missing?
Anyway, the fundamental issue I have is that this isn't what I had in
mind when the word 'test grokker' was mentioned. I was imagining a
system as sketched out above, where you have a grok.Doctest base class
to register file-based doctests. We can pick up unittest.TestCase
automatically. This way people can just focus on writing tests and grok
will pick them up with minimum hassle. Maybe we still require people put
them in modules or packages 'tests' and 'ftests' as Grok's convention
(and Zope 3's, in fact).
Instead grok_tests is now a scanning procedure that can be used to scan
a package for grok-style tests. This cuts away some code, automates a
messy piece of logic we had for test registration, and reuses some more
of grok internally.
I think it's still worthwhile to do this, but I'd like to be very
careful we don't publish this anywhere as the way grok tells you to test
your code. I'm not sure therefore whether this should be called
'grok.testing', as this implies reusable code.
Of course whether we can do something I sketched out depends on whether
we can convince Zope's testing infrastructure to actually find these
things as tests without us having to write anything called test_suite()
anywhere. In fact it's kind of funny, as Zope 3's test collector comes
closest in activity to what grok does already. :)
More information about the Grok-dev