[Zope3-dev] Unicode handling in Zope3 Page Templates

[Jim Fulton]

Sidnei da Silva wrote:
> Hi,
> 
> I've found some trouble with ZPT and unicode while using Zope 3 Page
> Templates and Five with Zope2.
> 
> Basically, Zope 3 ZPT expects everything to return unicode, or some
> string that can be converted to unicode using whatever the default
> encoding of the system is.
> 
> Zope 2 OTOH, doesn't quite expect anything to spit unicode. (It may
> even barf if you do so).
> 
> So, I've created a couple pieces of content, and everything worked
> fine on the Zope 2 side. I can see the titles with the correct chars
> both in the ZMI and in Plone. 
> 
> However, if I try to view the same object in a Zope 3 ZPT, I get a
> unicode error (UnicodeDecodeError I think).
> 
> If I use sys.setdefaultencoding('utf-8') in sitecustomize.py,

I consider this evil.  I know Guido would be happy if this wasn't in
the language.  This is just not an option.



> I get a
> bit further, but then when the special char is displayed, I get a '?'
> char instead. 
> 
>>From poking around, It seems like what's happening is that ZPT is
> getting a 'latin-1' encoded string, and converting to unicode using
> 'utf-8', which is the system default encoding.
> 
> I've made some changes to tales, tal and pagetemplate and added a test
> to confirm that if only encoded strings are returned it doesn't
> break. And if only unicode strings are returned it also doesn't break.
> 
> However, this changes the current state, in which ZPT always returns
> unicode to a state where if you get at least a method or attribute
> returning unicode, you will get unicode output, whereas if you dont
> have any unicode involved, you will get a string.
> 
> Any chances this can be integrated into Zope 3,

Probably not. Certainly not in it's current form.

The problem is that you can't really predict what the
encoding will be in Zope 2.  IMO, it is better not to guess.

If you did guess, you'd probably want to guess latin 1.

I don't have any good ideas for a short-term hack.  Maybe someone
else does.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org