[Grok-dev] Re: How to hook up custom layers with z3c.testsetup?
Philipp von Weitershausen
philipp at weitershausen.de
Tue Jun 24 05:44:39 EDT 2008
Uli Fouquet wrote:
> I think you might be confused by the marker name 'Test-Layer'. I can
> understand that. It should better be named 'Test-Type' or similar.
>> Also, I've not managed to make out the difference between the 'python'
>> and the 'unit' setting for this parameter as described by the README.
> There are currently three test types supported: (1) unit doctests
> ('unit'), (2) functional doctests ('functional') and (3) non-doctests
> ('python'). The latters are registered by `unittest.defaultTestLoader`
> and must take care themselves for setup, any functional layers etc. The
> z3c.testsetup package currently only cares for the layer setup of the
> second type and must confess, that I've never setup a layer for a normal
> unit test yet.
> The marker 'unit|functional|python' therefore not only decides about
> applying or not applying a layer, but also about different test loaders,
> different setUp/tearDown-methods, etc.
So tests of type 'unit' will get an automatic setUp/tearDown? What do
these do? Do they call zope.testing.cleanup.cleanUp()?
>> I read the docs but I was much confused about all the assumptions the
>> package makes. I think even a medium-sized application will quickly
>> outgrow these assumptions.
> That might be right. For me, however, it became handy in several
> projects and I think for others too.
I never doubted that :).
> Anyway, when your testing setup
> becomes too complex, you're certainly better off setting up the tests
> yourself and in the good old-fashioned way.
I must strongly disagree here, because we "trick" people into thinking
that test setup is very easy. Then they follow down the road that we
paved with z3c.testsetup, only to tell them when it outgrows their
needs, they need to do things *completely differently*. I would rather
not break the linearity of the learning curve there. In fact, Grok is
all about making the learning curve linear...
> The package is not a
> replacement for any other testing environment but only a 'frontend' that
> sets up some defaults. Lots of 'assumptions' (I'd call it 'defaults')
> therefore are needed. Without it the package would not make much sense.
I agree that defaults are a good idea and I'm not proposing to change
much with the defaults. I'm proposing to go further than what it can
already do, so we won't break people's assumptions (in this case Ivo's
rather valid assumption that integration tests can't be too different
from functional tests).
> For the layer stuff, however, I'd agree that this is not solved
> satisfactorily yet. It is quite difficult to setup different layers with
> the package. Your proposal below sounds good in that respect.
>> For instance, guessing that
>> :Test-Layer: functional
>> *implicitly* loads ftesting.zcml didn't even cross my mind for a
>> second, I thought it was somehow hooked up to FunctionalLayer.
> Don't know what you mean here. I might be wrong but when setting up a
> functional doctest, there is always some ZCML file involved, isn't it?
> The default behaviour of the zope.app.testing package, if I remember
> correctly, is to look for such a file somewhere in the depths of
> `parts/test/` or so.
The default behaviour of the zope.app.testing package was to look in
INSTANCE/etc/ftesting.zcml. This was so utterly problematic that nobody
uses it anymore. If you look at the code in zope.app.testing.functional,
you'll have concentrate very hard, not because it's complicated but to
prevent vomiting ;).
We now explicitly define layers, take a look at virtually every Zope
package out there that has ftests, including Grok. I don't think we
should go back to making the same mistake again and implicitly load
files based on naming conventions and magic places. After all, there
*is* a FunctionalLayer definition in a standard grokproject. Isn't it
used? I think it should be used and simply mapped to the 'functional'
test type. Or better, the layer should simply be referred to from the
test, like I suggested::
> Compared to this the new default is more reasonable, I think.
>> I think it could be improved in a very simple way: In doctests, you
>> simply refer to the dotted name of the layer, e.g.:
>> :Test-Layer: myapp.tests.IntegrationLayer
>> z3c.testsetup just finds the doctests, aggregates them, applies the
>> layer they specify and returns them to the test runner. By default,
>> there'd be a FunctionalLayer in grokproject (like there's already)
>> which is being referred to from the minimal test in app.py.
>> WIth this, people can easily create new layers (just create another
>> ZCML file, another ZCMLLayer definition in tests.py) and easily hook
>> them up to their doctests.
>> What do you think?
> Nice idea!
> I agree with the idea in general and personally do not like the tag name
> 'Test-Layer' to declare a test _type_ at all.
> The problem with it is, that the name is now in use and also its
> semantics. We would therefore run some people into terrible trouble when
> we change it. What we need here might be a 'migration' policy. What
> about the following?
> In the beginning let the test types be declared also by another keyword
> - :Test-Type: unit|functional|python
Why do we still need this? If we support arbitrary layers, these should
become superfluous. In fact, following the rule that there should be
only way to do things, this should go away.
The 'fucntional' type would be absorbed by using the FunctionalLayer
definition, 'unit' would be a special layer that has the setup and
teardown that you mention above and 'python' would be, um, nothing, I
> The keyword :Test-Layer: might be supported for a while, but should
> issue deprecation warnings when used.
> For the real layer setup I could imagine some (optional) additional
> markers (declarations), like::
> - :Test-Layername: NameOfLayer
> - :Test-LayerZCML: filename.zcml (looked up in package root)
> - :Test-Layerdef: dotted.name.of.definition
Too much choice for my taste. I think we really only need the last one.
A ZCML file can be turned into a layer with a one-liner in tests.py. And
thanks to grokproject's default FunctionalLayer, people can easily see
how to do this. Ivo figured *that* out very easily, and if not, he
could've always taken my book where exactly this is documented, or taken
any other Zope package. I don't think we should reinvent well-documented
conventions from Zope.
The only really difficult problem for Ivo was hooking up his layer to
the test. Hence my suggestion to simply specify the dotted layer name in
the test file.
> So we could move the layer-related parameters step by step from the test
> setup file to the real test files, where they might belong. After some
> releases then 'Test-Layer' could become a synonym for 'Test-Layerdef'
> and the latter might vanish completely some day.
Well, following my urge for simplification, I suggest we allow
Test-Layer to specify dotted names of layers, and for BBB, we still
allow 'unit', 'python' and 'functional' while phasing them out.
More information about the Grok-dev