[Zope3-dev] Unicode handling in Zope3 Page Templates
Sidnei da Silva
sidnei at awkly.org
Mon Aug 30 15:05:17 EDT 2004
Hi,
I've found some trouble with ZPT and unicode while using Zope 3 Page
Templates and Five with Zope2.
Basically, Zope 3 ZPT expects everything to return unicode, or some
string that can be converted to unicode using whatever the default
encoding of the system is.
Zope 2 OTOH, doesn't quite expect anything to spit unicode. (It may
even barf if you do so).
So, I've created a couple pieces of content, and everything worked
fine on the Zope 2 side. I can see the titles with the correct chars
both in the ZMI and in Plone.
However, if I try to view the same object in a Zope 3 ZPT, I get a
unicode error (UnicodeDecodeError I think).
If I use sys.setdefaultencoding('utf-8') in sitecustomize.py, I get a
bit further, but then when the special char is displayed, I get a '?'
char instead.
>From poking around, It seems like what's happening is that ZPT is
getting a 'latin-1' encoded string, and converting to unicode using
'utf-8', which is the system default encoding.
I've made some changes to tales, tal and pagetemplate and added a test
to confirm that if only encoded strings are returned it doesn't
break. And if only unicode strings are returned it also doesn't break.
However, this changes the current state, in which ZPT always returns
unicode to a state where if you get at least a method or attribute
returning unicode, you will get unicode output, whereas if you dont
have any unicode involved, you will get a string.
Any chances this can be integrated into Zope 3, or that someone can
shed more light on the subject?
--
Sidnei da Silva <sidnei at awkly.org>
http://awkly.org - dreamcatching :: making your dreams come true
http://www.enfoldsystems.com
http://plone.org/about/team#dreamcatcher
<bruce> how are the jails in israel?
<itamar> well, the one I was in was pretty nice
-------------- next part --------------
Index: src/zope/tal/talinterpreter.py
===================================================================
--- src/zope/tal/talinterpreter.py (revision 27353)
+++ src/zope/tal/talinterpreter.py (working copy)
@@ -562,7 +562,7 @@
if structure is self.Default:
self.interpret(block)
return
- text = unicode(structure)
+ text = isinstance(structure, basestring) and structure or unicode(structure)
if not (repldict or self.strictinsert):
# Take a shortcut, no error checking
self.stream_write(text)
Index: src/zope/pagetemplate/pagetemplate.py
===================================================================
--- src/zope/pagetemplate/pagetemplate.py (revision 27353)
+++ src/zope/pagetemplate/pagetemplate.py (working copy)
@@ -112,7 +112,7 @@
if self._v_errors:
raise PTRuntimeError(str(self._v_errors))
- output = StringIO(u'')
+ output = StringIO()
context = self.pt_getEngineContext(namespace)
TALInterpreter(self._v_program, self._v_macros,
context, output, tal=not source, strictinsert=0)()
Index: src/zope/pagetemplate/tests/input/nonascii.txt
===================================================================
--- src/zope/pagetemplate/tests/input/nonascii.txt (revision 0)
+++ src/zope/pagetemplate/tests/input/nonascii.txt (revision 0)
@@ -0,0 +1,2 @@
+In every census between 1960 and 2000, rural counties have constituted
+95 percent of those labeled ?persistently poor.?
Property changes on: src/zope/pagetemplate/tests/input/nonascii.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Index: src/zope/pagetemplate/tests/input/teeshop3.html
===================================================================
--- src/zope/pagetemplate/tests/input/teeshop3.html (revision 0)
+++ src/zope/pagetemplate/tests/input/teeshop3.html (revision 0)
@@ -0,0 +1,6 @@
+<html metal:use-macro="options/laf/macros/page">
+<div metal:fill-slot="body">
+<tal:block replace="options/data" />
+<tal:block replace="structure options/data" />
+</div>
+</html>
Property changes on: src/zope/pagetemplate/tests/input/teeshop3.html
___________________________________________________________________
Name: svn:eol-style
+ native
Index: src/zope/pagetemplate/tests/output/teeshop3.html
===================================================================
--- src/zope/pagetemplate/tests/output/teeshop3.html (revision 0)
+++ src/zope/pagetemplate/tests/output/teeshop3.html (revision 0)
@@ -0,0 +1,55 @@
+<html>
+<head>
+<title>Zope Stuff</title>
+<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
+<link rel="stylesheet" href="/common.css">
+</head>
+
+<body bgcolor="#FFFFFF" leftmargin="0" topmargin="0" marginwidth="0" marginheight="0">
+<table width="100%" border="0" cellspacing="0" cellpadding="0">
+ <tr bgcolor="#0000CC" align="center">
+ <td>
+ <table width="200" border="0" cellspacing="0" cellpadding="0">
+ <tr bgcolor="#FFFFFF">
+ <td><img src="/images/lside.gif" width="52" height="94"><img src="/images/swlogo.gif" width="150" height="89"><img src="/images/rside.gif" width="52" height="94"></td>
+ </tr>
+ </table>
+ </td>
+ </tr>
+</table>
+<br>
+<table width="300" border="0" cellspacing="0" cellpadding="0" align="center">
+ <tr align="center">
+ <td width="25%" class="boldbodylist">apparel</td>
+ <td width="25%" class="boldbodylist">mugs</td>
+ <td width="25%" class="boldbodylist">toys</td>
+ <td width="25%" class="boldbodylist">misc</td>
+ </tr>
+</table>
+<br>
+<br>
+<div>
+In every census between 1960 and 2000, rural counties have constituted
+95 percent of those labeled ?persistently poor.?
+
+In every census between 1960 and 2000, rural counties have constituted
+95 percent of those labeled ?persistently poor.?
+
+</div>
+<br><br>
+<table width="100%" border="0" cellspacing="1" cellpadding="3" align="center">
+ <tr>
+ <td align="center" bgcolor="#FFFFFF" class="bodylist">
+ Copyright © 2000
+ <a href="http://www.4-am.com">4AM Productions, Inc.</a>.
+ All rights reserved. <br>
+ Questions or problems should be directed to
+ <a href="mailto:webmaster at teamzonline.com">the webmaster</a>,
+ 254-412-0846.</td>
+ </tr>
+ <tr>
+ <td align="center"><img src="/images/zopelogos/buildzope.gif" width="54" height="54"></td>
+ </tr>
+</table>
+</body>
+</html>
Property changes on: src/zope/pagetemplate/tests/output/teeshop3.html
___________________________________________________________________
Name: svn:eol-style
+ native
Index: src/zope/pagetemplate/tests/test_htmltests.py
===================================================================
--- src/zope/pagetemplate/tests/test_htmltests.py (revision 27353)
+++ src/zope/pagetemplate/tests/test_htmltests.py (working copy)
@@ -68,6 +68,28 @@
out = t(laf = self.folder.laf, getProducts = self.getProducts)
util.check_html(expect, out)
+ def test_4(self):
+ self.folder.laf.write(util.read_input('teeshoplaf.html'))
+ data = util.read_input('nonascii.txt')
+ t = self.folder.t
+ t.write(util.read_input('teeshop3.html'))
+ expect = util.read_output('teeshop3.html')
+ data = unicode(data, 'latin-1').encode('utf-8')
+ out = t(laf = self.folder.laf, data = data)
+ expect = unicode(expect, 'latin-1').encode('utf-8')
+ util.check_html(expect, out)
+
+ def test_5(self):
+ self.folder.laf.write(util.read_input('teeshoplaf.html'))
+ data = util.read_input('nonascii.txt')
+ t = self.folder.t
+ t.write(util.read_input('teeshop3.html'))
+ expect = util.read_output('teeshop3.html')
+ data = unicode(data, 'latin-1')
+ out = t(laf = self.folder.laf, data = data)
+ expect = unicode(expect, 'latin-1')
+ util.check_html(expect, out)
+
def test_SimpleLoop(self):
t = self.folder.t
t.write(util.read_input('loop1.html'))
Index: src/zope/tales/tales.py
===================================================================
--- src/zope/tales/tales.py (revision 27353)
+++ src/zope/tales/tales.py (working copy)
@@ -706,7 +706,7 @@
text = self.evaluate(expr)
if text is self.getDefault() or text is None:
return text
- return unicode(text)
+ return isinstance(text, basestring) and text or unicode(text)
def evaluateStructure(self, expr):
return self.evaluate(expr)
More information about the Zope3-dev
mailing list