[Checkins] SVN: zope.testrunner/trunk/ Fix --shuffle nondeterminism when multiple test layers are present.

[Marius Gedminas]

Log message for revision 129212:
  Fix --shuffle nondeterminism when multiple test layers are present.
  
  All tests now pass on all supported Python versions (except PyPy).
  
  

Changed:
  U   zope.testrunner/trunk/CHANGES.txt
  U   zope.testrunner/trunk/src/zope/testrunner/shuffle.py
  U   zope.testrunner/trunk/src/zope/testrunner/testrunner-shuffle.txt

-=-
Modified: zope.testrunner/trunk/CHANGES.txt
===================================================================
--- zope.testrunner/trunk/CHANGES.txt	2013-02-08 08:54:36 UTC (rev 129211)
+++ zope.testrunner/trunk/CHANGES.txt	2013-02-08 09:20:53 UTC (rev 129212)
@@ -6,12 +6,16 @@
 
 - Dropped use of zope.fixers (LP: #1118877).
 
-- Fixed tox test error reporting; fixed tests on Pythons 2.6, 3.1, and 3.2.
+- Fixed tox test error reporting; fixed tests on Pythons 2.6, 3.1, 3.2, and
+  3.3.
 
 - Fix --shuffle ordering on Python 3.2 to be the same as it was on older Python
   versions.
 
+- Fix --shuffle nondeterminism when multiple test layers are present.
+  Note: this will likely change the order of tests for the same --shuffle-seed.
 
+
 4.1.0 (2013-02-07)
 ==================
 

Modified: zope.testrunner/trunk/src/zope/testrunner/shuffle.py
===================================================================
--- zope.testrunner/trunk/src/zope/testrunner/shuffle.py	2013-02-08 08:54:36 UTC (rev 129211)
+++ zope.testrunner/trunk/src/zope/testrunner/shuffle.py	2013-02-08 09:20:53 UTC (rev 129212)
@@ -38,7 +38,8 @@
         if sys.version_info >= (3, 2):
             # in case somebody tries to use a string as the seed
             rng.seed(self.seed, version=1)
-        for layer, suite in list(self.runner.tests_by_layer_name.items()):
+        # Be careful to shuffle the layers in a deterministic order!
+        for layer, suite in sorted(self.runner.tests_by_layer_name.items()):
             # Test suites cannot be modified through a public API.  We thus
             # take a mutable copy of the list of tests of that suite, shuffle
             # that and replace the test suite instance with a new one of the

Modified: zope.testrunner/trunk/src/zope/testrunner/testrunner-shuffle.txt
===================================================================
--- zope.testrunner/trunk/src/zope/testrunner/testrunner-shuffle.txt	2013-02-08 08:54:36 UTC (rev 129211)
+++ zope.testrunner/trunk/src/zope/testrunner/testrunner-shuffle.txt	2013-02-08 09:20:53 UTC (rev 129212)
@@ -46,6 +46,12 @@
 seed number that was printed out and run it again using the ``--shuffle-seed``
 option. The order is guaranteed to be the same.
 
+    There was an issue with the guaranteed order in zope.testrunner before
+    4.0.2: the order could change depending on dictionary iteration order, if
+    you had multiple test layers.  This bug was fixed in 4.0.2, but a side
+    effect of the fix means that the new guaranteed order is likely to be
+    different from what you used to get with an older zope.testrunner.
+
 For example, using the seed number 0 will give us the following, stable, list of
 tests:
 
@@ -53,30 +59,30 @@
     >>> testrunner.run_internal(defaults, argv)
     Tests were shuffled using seed number 0.
     Listing zope.testrunner.layer.UnitTests tests:
-      test_y0 (sample1.sample13.sampletests.TestA)
       test_y0 (sample1.sampletestsf.TestA)
+      test_y0 (sample1.sampletests.test_one)
+      test_y0 (sample1.sampletests.test1.TestA)
       test_y0 (sample1.sampletestsf)
       test_y0 (sample1.sampletests.test_one.TestA)
-      test_y0 (sample1.sampletests.test_one)
+      test_y0 (sample1.sample13.sampletests)
+      test_y0 (sample1.sample13.sampletests.TestA)
+      test_y0 (sample1.sample11.sampletests)
+      test_y0 (sample1.sample11.sampletests.TestA)
       test_y0 (sample1.sampletests.test1)
-      test_y0 (sample1.sample11.sampletests.TestA)
-      test_y0 (sample1.sampletests.test1.TestA)
-      test_y0 (sample1.sample11.sampletests)
-      test_y0 (sample1.sample13.sampletests)
     False
     >>> testrunner.run_internal(defaults, argv)
     Tests were shuffled using seed number 0.
     Listing zope.testrunner.layer.UnitTests tests:
-      test_y0 (sample1.sample13.sampletests.TestA)
       test_y0 (sample1.sampletestsf.TestA)
+      test_y0 (sample1.sampletests.test_one)
+      test_y0 (sample1.sampletests.test1.TestA)
       test_y0 (sample1.sampletestsf)
       test_y0 (sample1.sampletests.test_one.TestA)
-      test_y0 (sample1.sampletests.test_one)
+      test_y0 (sample1.sample13.sampletests)
+      test_y0 (sample1.sample13.sampletests.TestA)
+      test_y0 (sample1.sample11.sampletests)
+      test_y0 (sample1.sample11.sampletests.TestA)
       test_y0 (sample1.sampletests.test1)
-      test_y0 (sample1.sample11.sampletests.TestA)
-      test_y0 (sample1.sampletests.test1.TestA)
-      test_y0 (sample1.sample11.sampletests)
-      test_y0 (sample1.sample13.sampletests)
     False
 
 Whereas using the seed number 42 will give us the following, different but
@@ -86,30 +92,30 @@
     >>> testrunner.run_internal(defaults, argv)
     Tests were shuffled using seed number 42.
     Listing zope.testrunner.layer.UnitTests tests:
+      test_y0 (sample1.sample13.sampletests.TestA)
       test_y0 (sample1.sample13.sampletests)
+      test_y0 (sample1.sampletests.test1)
+      test_y0 (sample1.sampletests.test1.TestA)
+      test_y0 (sample1.sample11.sampletests.TestA)
+      test_y0 (sample1.sampletestsf)
+      test_y0 (sample1.sampletests.test_one)
       test_y0 (sample1.sample11.sampletests)
-      test_y0 (sample1.sampletestsf)
-      test_y0 (sample1.sample13.sampletests.TestA)
+      test_y0 (sample1.sampletestsf.TestA)
       test_y0 (sample1.sampletests.test_one.TestA)
-      test_y0 (sample1.sample11.sampletests.TestA)
-      test_y0 (sample1.sampletestsf.TestA)
-      test_y0 (sample1.sampletests.test_one)
-      test_y0 (sample1.sampletests.test1.TestA)
-      test_y0 (sample1.sampletests.test1)
     False
     >>> testrunner.run_internal(defaults, argv)
     Tests were shuffled using seed number 42.
     Listing zope.testrunner.layer.UnitTests tests:
+      test_y0 (sample1.sample13.sampletests.TestA)
       test_y0 (sample1.sample13.sampletests)
+      test_y0 (sample1.sampletests.test1)
+      test_y0 (sample1.sampletests.test1.TestA)
+      test_y0 (sample1.sample11.sampletests.TestA)
+      test_y0 (sample1.sampletestsf)
+      test_y0 (sample1.sampletests.test_one)
       test_y0 (sample1.sample11.sampletests)
-      test_y0 (sample1.sampletestsf)
-      test_y0 (sample1.sample13.sampletests.TestA)
+      test_y0 (sample1.sampletestsf.TestA)
       test_y0 (sample1.sampletests.test_one.TestA)
-      test_y0 (sample1.sample11.sampletests.TestA)
-      test_y0 (sample1.sampletestsf.TestA)
-      test_y0 (sample1.sampletests.test_one)
-      test_y0 (sample1.sampletests.test1.TestA)
-      test_y0 (sample1.sampletests.test1)
     False
 
 Selecting a seed number without ``--shuffle``