[Zope-dev] Test fixture concepts

Tres Seaver tseaver at palladion.com
Mon Mar 28 10:45:27 EDT 2011


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/28/2011 10:04 AM, Jim Fulton wrote:
> On Sun, Mar 27, 2011 at 11:13 AM, Martin Aspeli
> <optilude+lists at gmail.com> wrote:
>> Hi,
>>
>> On 27 March 2011 15:54, Uli Fouquet <uli at gnufix.de> wrote:
>>
>>> The (limited) experiences with py.test, however, were awesome. Some
>>> points that are quite cool IMHO:
>>>
>>> - Easy finding of tests: just write some ``test_function`` in a
>>>  ``test_module`` and it will be found and executed. That also makes
>>>  py.test tests more readable and maybe more intuitive.
>>
>> I'm not sure this is always a good idea. It feels a bit implicit, and
>> having a base class isn't really a big problem, IMHO. It seems a bit
>> like the kind of thing that sounds cool (look, it's even easier!), but
>> in practice makes little difference.
> 
> +1
> +1
> +1
> 
> This is especially important for doctests (and manuel) or any
> situation where setup is important and where you can't really guess.
> 
>>> - py.test is more widespread in the Python community (that's my
>>>  impression; I can't proof it)
>>
>> What about nose?
> 
> It looks to me like a layerish mechanism might be possible in nose, or
> at least like
> zope.testing layers could be integrated with nose.
> 
>>
>>> - Support of unittest/unittest2: you can write standard lib setups
>>>  (defining TestCases; no need to also write testsuite-setup stuff) and
>>>  they will be found/executed. zope.testrunner for instance does not
>>>  support the new `setUpClass`/`tearDownClass` concept of unittest2
>>>  (yes, you would use layers in that case; but it might be nice if
>>>  zope.testrunner  would support also class-wide fixtures in
>>>  unittest2-style; people from other worlds might expect that to work).
>>
>> zope.testing should definitely gain support for the new unittest2
>> hooks. That wouldn't be very hard, though. ;-)
> 
> I assume you mean zope.testrunner.
> 
>>
>>> Main drawbacks I see on py.test side are:
>>>
>>> - Lack of layer support (yet). Maybe we can do something about that in
>>>  `zope.pytest` based on `plone.testing.layer`.
>>>
>>> - Limited doctest support. It is quite difficult (AFAIK) to define
>>>  fixtures for doctests or to even set the usual doctest options
>>>  (``ELLIPSIS``, ``NORMALIZE_WHITESPACE``, ...) at setup time. Doctests
>>>  are simply collected and executed and not much finetuning is possible.
>>
>> With zope.testrunner, you *do* need a test_suite method to run
>> doctests. I think that's a good thing. Look at plone.testing's README
>> for examples.
> 
> Again, +1
> 
> If I were to use nose or py.tests, I would want to adopt an explicit
> style, which I believe is possible w nose.

FWIW, I thought the same, but haven't missed 'test_suite' appreciably
after a week or so of acclimatization to the nose discovery mechanism.
The only wrinkle I know of is that one doesn't use base classes for test
cases where the base class itself derives from unittest.TestCase:
instead, the base is just used as a "pure mixin".


>> FWIW, I think we should stop using .txt doctests for unit tests.
> 
> I disagree, of course.
> 
>> Doctests should be used to test *documentation* ("the examples are
>> valid").
> 
> Manuel is *much* better for that.  (Of course, manuel is arguably a form of
> doctest.)

The Sphinx doctest integration[1] ('make doctest') supports both classic
"interpreter prompt" doctests and a more useful (IMHO) "code-output"
form, such as:

.. testcode::

   1+1        # this will give no output!
   print 2+2  # this will give output

.. testoutput::

   4


[1] http://sphinx.pocoo.org/ext/doctest.html

>> For actual unit tests, writing tests in a unittest class is
>> almost always better in the long run. doctests don't scale well and
>> discourage the kind of ad-hoc "this seems broken, I'll just write a
>> quick test" or "I just fixed a bug, better add a regression test"
>> testing.
> 
> You're just not using them correctly. :)

The vast majority of the doctest testcases in zope.* packages fall into
this category:  poor isolation, lots of edge cases which would obscure
any real narrative docs, of which there are almost none.  I believe the
conflict is intrinsic, here, and not just an accident of careless /
naive implementation:  exercising all the preconditions of the contract
of the function-under-test makes for really poor documentation, but is
essential to good testing.

> One thing I hate about unittest is the javiotic ceremony it involves.
> Doctests can cut down on the clutter a lot. I believe that that is
> py.test's strength as well..

"I'll take explicit for $1000, Alex."

> More generally, I'd love to see us adopt another test runner so that
> we can stop maintianing zope.testrunner.  When it was written at
> the turn of the century, there weren't good alternatives.  Personally,
> I think maintaining it is boring.

Nose and coverage have been pretty good to repoze.* and pyramid_*.



Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk2QnwcACgkQ+gerLs4ltQ74FQCdGX9twswg6eAy9cnpABAZRrbo
pvoAniGfpBliwnFjqFrV5uqpFcePI2tk
=cqN7
-----END PGP SIGNATURE-----



More information about the Zope-Dev mailing list