[Grok-dev] Re: How to hook up custom layers with z3c.testsetup?

Philipp von Weitershausen philipp at weitershausen.de
Tue Jun 24 09:49:31 EDT 2008

El 24 Jun 2008, a las 15:17 , Uli Fouquet escribió:
> Philipp von Weitershausen wrote:
>> Uli Fouquet wrote:
>>> The marker 'unit|functional|python' therefore not only decides about
>>> applying or not applying a layer, but also about different test  
>>> loaders,
>>> different setUp/tearDown-methods, etc.
>> So tests of type 'unit' will get an automatic setUp/tearDown? What do
>> these do? Do they call zope.testing.cleanup.cleanUp()?
> That's exactly what they do currently.

Good to know. I think the identifiers 'python' and 'unit' are a bit  
confusing. At least they're not self-explanatory in what they imply  
(see below for my suggestions regarding names).

[... lots of agreement]

>> I think it should be used and simply mapped to the 'functional'
>> test type. Or better, the layer should simply be referred to from the
>> test, like I suggested::
>>   :Test-Layer: myapp.tests.FunctionalLayer
> The first approach, mapping myapp.tests.FunctionalLayer, would IMHO  
> only
> shift the problem of automatically finding ftesting.zcml to
> automatically finding the FunctionalLayer definition. Furthermore it
> would continue the problem of different layers for different test  
> files.

Right, I was only suggesting the mapping from names to layers for  
legacy reasons. The canonical way should simply be referring to the  
layer object.

> The second approach, referring to the layer definition from the test
> file, sounds good to me. There is, however, one problem with it (see
> below).
>>>> I think it could be improved in a very simple way: In doctests, you
>>>> simply refer to the dotted name of the layer, e.g.:
>>>>   :Test-Layer: myapp.tests.IntegrationLayer
>>>> z3c.testsetup just finds the doctests, aggregates them, applies the
>>>> layer they specify and returns them to the test runner. By default,
>>>> there'd be a FunctionalLayer in grokproject (like there's already)
>>>> which is being referred to from the minimal test in app.py.
>>>> WIth this, people can easily create new layers (just create another
>>>> ZCML file, another ZCMLLayer definition in tests.py) and easily  
>>>> hook
>>>> them up to their doctests.
>>>> What do you think?
>>> Nice idea!
>>> I agree with the idea in general and personally do not like the  
>>> tag name
>>> 'Test-Layer' to declare a test _type_ at all.
>>> The problem with it is, that the name is now in use and also its
>>> semantics. We would therefore run some people into terrible  
>>> trouble when
>>> we change it. What we need here might be a 'migration' policy. What
>>> about the following?
>>> In the beginning let the test types be declared also by another  
>>> keyword
>>>  - :Test-Type: unit|functional|python
>> Why do we still need this? If we support arbitrary layers, these  
>> should
>> become superfluous. In fact, following the rule that there should be
>> only way to do things, this should go away.
>> The 'fucntional' type would be absorbed by using the FunctionalLayer
>> definition, 'unit' would be a special layer that has the setup and
>> teardown that you mention above and 'python' would be, um, nothing, I
>> suppose?
> Right, 'python' means no setup for this file. But it means also: load
> tests from this file.

Ah, gotcha.

> Just to be sure: if I understand you correctly, you would propose  
> setups
> like this::
>  # tests.py
>  import myapp
>  from zope.app.testing.functional import ZCMLLayer
>  from zope.testing import cleanup
>  from grok.testing import register_all_tests
>  class UnitDocTestLayer(object):
>     def setUp(self, *args, **kw):
>       pass
>     def tearDown(self, *args, **kw):
>       cleanup.cleanUp()

This layer isn't entirely correct. We don't want cleanup to happen at  
the end of all tests in the layer, we want it to happen after every  
test (so that the unit tests are isolated). The way to ensure that is  
to implement testSetUp() and testTearDown() methods on the layer. The  
test runner will call these before and after every individual test  
within the layer:

   class AutoCleanUpLayer(object):

       def testSetUp(cls):
           cleanup.cleanUp()  # can't hurt :)

       def testTearDown(cls):

Note my usage of @classmethod which is allows not having to  
instantiate the class. This is necessary because layer objects are  
expected to have a __bases__ attribute.

Also note that this layer is completely generic, so it could be in  
z3c.testsetup. I'd be fine with mapping this layer to an alias, so  
that instead of

   :Test-Layer: z3c.testsetup.layers.AutoCleanUpLayer

you could simply say:

   :Test-Layer: auto-cleanup

Or something like that. Note that I've deliberately not called this  
"UnitLayer" or something like that, because other tests (that are not  
on this layer) may also be unit tests. In fact, the tests you gave the  
identifier 'python' are unit tests as well! THis is why I was so  
confused about 'python' vs. 'unit'.

>  ftesting_zcml = os.path.join(
>    os.path.dirname(myapp.__file__), 'ftesting.zcml')
>  FunctionalLayer = ZCMLLayer(ftesting_zcml, __name__,  
> 'FunctionalLayer')

Only this would stay with my suggestion from above.

> To be honest, such a tests.py as above scares me a bit, although it
> might be autogenerated by grokproject.

I agree, it would be scary. That's why I'm suggesting to move the  
generic bits into the generic z3c.testsetup package. I know we can  
autogenerate a lot with grokproject, but I prefer keeping the amount  
of autogenerated code to a minimum.

> Another problem we run into here is: the marker not only says what  
> layer
> to use, but it also says that the current file is a test file at  
> all. It
> separates test files from non-test files.

This is a good point.

> For 'python' tests there would be no layer and so there would be no
> marker. We could possibly provide a dummy-layer for that purpose, but
> that might confuse people and sounds wrong to me.

We should probably separate the marker from the layer directive, e.g.:

   :Test-Layer: auto-cleanup




   :Test-Layer: myapp.tests.IntegrationLayer

would all be valid ways of defining doctests.

> The questions then remains, how to mark 'python' tests. They can be
> found by the testrunner, of course, based on the filename. But this
> would mean to tell people: if you have this kind of test, mark it by  
> one
> of those strings. If you have another kind of test, then use one of  
> the
> following filenames. One time you mark it by a certain string in the
> test file, another time by configuring the testrunner. Having all  
> kinds
> of tests marked in a similar manner is nice IMO and (beside the layer
> problem) easy to understand.

Yup. I agree. Can't find a better argument for a marker directive  
like :DocTest: :)

>> A ZCML file can be turned into a layer with a one-liner in  
>> tests.py. And
>> thanks to grokproject's default FunctionalLayer, people can easily  
>> see
>> how to do this. Ivo figured *that* out very easily, and if not, he
>> could've always taken my book where exactly this is documented, or  
>> taken
>> any other Zope package. I don't think we should reinvent well- 
>> documented
>> conventions from Zope.
> That's true for grokprojects. But what about other packages written  
> from
> scratch? I really don't like to break well-documented and introduced
> practices but if people are not forced to setup their own layers  
> (while
> they still have the possibility to do so, if they wish), this might be
> nice. Beside this it is clear to me, that Ivos problem came from a
> crappy layer handling in the current package.
>> The only really difficult problem for Ivo was hooking up his layer to
>> the test. Hence my suggestion to simply specify the dotted layer  
>> name in
>> the test file.
> I agree completely. Layers should be declarable in the test files.
> Because of the above mentioned problem I only would propose to mark  
> that
> layer declaration with a different tag and possibly change the name of
> the marker strings.
>>> So we could move the layer-related parameters step by step from  
>>> the test
>>> setup file to the real test files, where they might belong. After  
>>> some
>>> releases then 'Test-Layer' could become a synonym for 'Test- 
>>> Layerdef'
>>> and the latter might vanish completely some day.
>> Well, following my urge for simplification, I suggest we allow
>> Test-Layer to specify dotted names of layers, and for BBB, we still
>> allow 'unit', 'python' and 'functional' while phasing them out.
> How should we distinguish those types of tests then? I might have  
> missed
> an important point here. If I understood you right, there is no need  
> for
> such differentiation, but it is not enirely clear to me, why this is  
> the
> case. Note, that currently tests of the different types are setup  
> using
> different methods:
> 'unit' leads to a call to zope.testing.doctest.DocFileSuite
> 'functional' leads to a call to
>       zope.app.testing.functional.FunctionalDocFileSuite

If you look at the code of FunctionalDocFileSuite, all it does is  
create a regular DocFileSuite and apply a layer to it, but then we  
apply our own layer to it. To cut a short story short, using  
FunctionalDocFileSuite has been unnecessary for quite some time (ever  
since we introduced custom fucntional layers for each package). Let's  
simply use DocFileSuite and apply our custom layer. (Grok itself does  
this, too!)

> 'python' leads to a call to unittest.defaultTestLoader

Interesting. How does it pick up doctests then? How is this different  
from using the regular DocFileSuite?

(Note that zope.testing.doctest and Python's doctest are basically the  
same, the zope.testing.doctest one just has a couple more features  
that aren't available in every Python version's doctest module. I  
think always using zope.testing.doctest is safe, in fact, I'd  
recommend it, and it shouldn't be a problem since z3c.testsetup  
depends on zope.testing anyway.)

> The marker string therefore cares also for different test loaders  
> (which
> might be done better). Furthermore it separates testfiles from
> non-testfiles. I do not see, how all this could be handled with a  
> layer
> definition.
> Beside this the :Test-Layerdef: marker seems to be a good idea anyway,
> so I will start implementing this :-)

Great. Note that I think having both :Test-Layer: and :Test-Layerdef:  
seems a bit confusing. I think we should simply use :Test-Layer:.

More information about the Grok-dev mailing list