[Zope-dev] Running the 3.4 KGS tests: 9 fail, 6 errors

[Marius Gedminas]

I finally discovered zope.release (and by "discovered" I mean Stephan
Richter told me about it on IRC, again, but this time I had the time to
take a look).  It has a script that generates a buildout of all the KGS
packages so you could run the tests on all of them:

  svn co svn+ssh://svn.zope.org/repos/main/zope.release/branches/3.4 zope.release
  cd zope.release
  buildout
  bin/generate-buildout
  cd test
  buildout

For some reason, the test script tries to run all the tests found in my
/usr/lib/python2.5/site-packages, some of which then fail.

  vi bin/test
    remove site-packages

  bin/test -pvc

I see 9 failures and 4 errors on my system (Python 2.5 on 32-bit Linux):

  * z3c.coverage 1.1.1 has a nondeterministic test (assumes the order of
    the names returned by os.listdir()).  This has been fixed in SVN,
    but nobody has made a 1.1.2 release and updated the KGS.  I don't
    have PyPI rights to release z3c.coverage.

  * z3c.form 1.8.0 is not compatible with Python 2.5 (assumes
    OverflowWarning)

  * z3c.rml 0.7.3 tests fail to find random TTF font files and fail.

  * zc.zope3recipes 0.6.1 is trying to get a distribution for setuptools,
    recursively, until it gets a stack overflow error

  * zope.server 3.4.2 is very unhappy about some bad file descriptors

  * the test runner keels over at the end of the unit test run, with
    AttributeError: 'module' object has no attribute 'DemoLayer'

If I then continue by skipping the unit tests

  bin/test -pvc -f

all the tests in layers pass.

I have interest in fixing these errors, especially the test runner one
(because it's important), and the z3c.coverage one (because it's easy).
I don't know if Python 2.5 compatibility is important for the 3.4 KGS,
but the z3c.form problem seems to be easy to fix.

I don't believe I have the expertise to handle the zc.zope3recipes
recursive setuptools failure.  Although it looks like it could provide
me with a few hours of entertaining debugging, I've no confidence I can
fix it.  The zope.server multi-threaded bad file descriptor errors also
seem tricky.

A lot of tests like to spew random output to stdout/stderr, which makes
interpreting the test output harder (and probably breaking the test
runner if it has to run any of those tests in a subprocess, e.g. if you
want to run those tests in parallel).  A crusade to clean that up is out
of scope for 3.4, IMHO.

I think I'll run the test suite again on a 64-bit Linux machine, for
extra fun.  And maybe do that for Python 2.4 as well.

Marius Gedminas
-- 
Always proofread carefully to seon e if you any words out.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: Digital signature
Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080727/b2787d11/attachment.bin