[Zope-Coders] core dump in Zope 2.7 test suite

[Jeremy Hylton]

I tried to run the full Zope test suite after merging all the changes
from the ZODB 3.2 branch.  I used "python test.py -vv --all" where
python is a recent build of CVS python.

It failed with a core dump partway through the PageTemplates tests.
checkBatchIteration
(Products.PageTemplates.tests.testHTMLTests.HTMLTests) ... ok
checkFancyLoop (Products.PageTemplates.tests.testHTMLTests.HTMLTests)
... ok
checkGlobalsShadowLocals
(Products.PageTemplates.tests.testHTMLTests.HTMLTests) ... ok
checkI18nTranslate
(Products.PageTemplates.tests.testHTMLTests.HTMLTests) ...
okcheckI18nTranslateHooked
(Products.PageTemplates.tests.testHTMLTests.HTMLTests) ... ok
checkNotExpression
(Products.PageTemplates.tests.testHTMLTests.HTMLTests) ...
okcheckPathAlt (Products.PageTemplates.tests.testHTMLTests.HTMLTests)
... ok
checkPathNothing (Products.PageTemplates.tests.testHTMLTests.HTMLTests)
... ok
checkReplaceWithNothing
(Products.PageTemplates.tests.testHTMLTests.HTMLTests) ... ok
checkSimpleLoop (Products.PageTemplates.tests.testHTMLTests.HTMLTests)
... ok
checkStringExpressions
(Products.PageTemplates.tests.testHTMLTests.HTMLTests) ...
6,7r6,7
<  Error:x
<  Error:x
---
>  Error:'x'
>  Error:'x'
 
 
Failure in test checkStringExpressions
(Products.PageTemplates.tests.testHTMLTests.HTMLTests)
Traceback (most recent call last):
  File
"/home/jeremy/src/branches/Zope-2_7-branch/lib/python/Products/PageTemplates/tests/testHTMLTests.py", line 128, in checkStringExpressions
    self.assert_expected(self.folder.t, 'StringExpression.html')
  File
"/home/jeremy/src/branches/Zope-2_7-branch/lib/python/Products/PageTemplates/tests/testHTMLTests.py", line 81, in assert_expected
    util.check_html(expect, out)
  File
"/home/jeremy/src/branches/Zope-2_7-branch/lib/python/Products/PageTemplates/tests/util.py", line 88, in check_html
    assert s1==s2, "HTML Output Changed"
AssertionError: HTML Output Changed
 
checkUnicodeInserts
(Products.PageTemplates.tests.testHTMLTests.HTMLTests) ... Segmentation
fault (core dumped)

The bottom of the gdb stack trace shows that it's deep into the unicode
support inside Python.  It looks like it's constructor a unicode object
for "&amp;".  Inspecting C frame #8, in particular, indicates that it is
executing the first line of cgi.escape():

def escape(s, quote=None):
    """Replace special characters '&', '<' and '>' by SGML entities."""
    s = s.replace("&", "&amp;") # Must be done first!

#0  0x0809ee51 in unicode_resize (unicode=0x41c7f6e8, length=5)
    at ../Objects/unicodeobject.c:133
#1  0x0809efda in _PyUnicode_New (length=5) at
../Objects/unicodeobject.c:192
#2  0x080a3c64 in PyUnicodeUCS4_DecodeASCII (s=0x4043d9e4 "&amp;",
size=5,
    errors=0x81159d8 "strict") at ../Objects/unicodeobject.c:2515
#3  0x0809f88d in PyUnicodeUCS4_Decode (s=0x4043d9e4 "&amp;", size=5,
    encoding=0x814b1e0 "ascii", errors=0x81159d8 "strict")
    at ../Objects/unicodeobject.c:539
#4  0x0809f7ca in PyUnicodeUCS4_FromEncodedObject (obj=0x4043d9c8,
    encoding=0x0, errors=0x81159d8 "strict") at
../Objects/unicodeobject.c:511
#5  0x0809f686 in PyUnicodeUCS4_FromObject (obj=0x4043d9c8)
    at ../Objects/unicodeobject.c:447
#6  0x080aac24 in unicode_replace (self=0x41c7f568, args=0x41be5fbc)
    at ../Objects/unicodeobject.c:5457
#7  0x081080d2 in PyCFunction_Call (func=0x41ca6a0c, arg=0x41be5fbc,
kw=0x0)
    at ../Objects/methodobject.c:73
#8  0x080c078c in call_function (pp_stack=0xbfffba24, oparg=2)
    at ../Python/ceval.c:3439
#9  0x080bd071 in eval_frame (f=0x906d6b4) at ../Python/ceval.c:2116
#10 0x080bed3d in PyEval_EvalCodeEx (co=0x404430d8, globals=0x404d1674,
    locals=0x0, args=0x906dee8, argcount=1, kws=0x906deec, kwcount=0,

The code appears to be blowing up because of a reference count problem
(what else?).

#0  0x0809ee51 in unicode_resize (unicode=0x41c7f6e8, length=5)
    at ../Objects/unicodeobject.c:133
133         if (unicode == unicode_empty ||
(gdb) l
128             goto reset;
129
130         /* Resizing shared object (unicode_empty or single character
131            objects) in-place is not allowed. Use PyUnicode_Resize()
132            instead ! */
133         if (unicode == unicode_empty ||
134             (unicode->length == 1 &&
135              unicode->str[0] < 256 &&
136              unicode_latin1[unicode->str[0]] == unicode)) {
137             PyErr_SetString(PyExc_SystemError,
(gdb) p unicode->str[0]
$8 = -875836469
(gdb) p *unicode
$9 = {_ob_next = 0x0, _ob_prev = 0x0, ob_refcnt = 0, ob_type =
0x8130160,
  length = 1, str = 0x41903130, hash = -1, defenc = 0x0}

It looks like the unicode object in question was already freed, but
we're trying to dereference it.  The index into unicode_laten1[] fails,
because the index is a huge negative number.

Has anyone seen a failure like this before?  I tried reproducing it in
isolation, but couldn't.

Jeremy