[ZCM] [ZC] 2204/ 3 Comment "UnicodeDecodeErrors in PageTemplates (Zope 2.10.0.c1)"

Wed Oct 4 06:43:43 EDT 2006

Issue #2204 Update (Comment) "UnicodeDecodeErrors in PageTemplates (Zope 2.10.0.c1)"
 Status Pending, Zope/bug critical
To followup, visit:
  http://www.zope.org/Collectors/Zope/2204

==============================================================
= Comment - Entry #3 by chrisw on Oct 4, 2006 6:43 am

ZPT currently tries to be agnostic to whether it's dealing with strings or unicodes.

This has lead to people happilly using encoded strings for everything, utf-8 encoded strings being a particular favourite.

However, Zope 3 has (rightly) opted to make everything unicode, so it will return unicode whenever it can.

As all good pythonistas should know, mixing unicodes and strings results in much pain and suffering unless the strings can be decoded using the python default decoder, which is usually us-ascii. 

A workaround is to change the default decoder to whatever encoding your strings have using site.py or site_customize.py.

However, ZPT's agnositicism is going to cause more and more problems as more of Zope 3 gets used in Zope 2...

Here was my encounter with a similar but not identical problem:

http://codespeak.net/pipermail/z3-five/2006q3/001534.html
http://codespeak.net/pipermail/z3-five/2006q3/001535.html
...and the rest of the thread.

cheers,

Chris
________________________________________
= Edit - Entry #2 by ajung on Oct 4, 2006 6:29 am

 Changes: submitter email, importance (medium => critical)
________________________________________
= Request - Entry #1 by eXt on Oct 4, 2006 6:27 am

Uploaded:  "test_unicode.py"
 - http://www.zope.org/Collectors/Zope/2204/test_unicode.py/view
This bug causes that it is very hard to use texts with national characters inside PageTemplates

Code like the one below placed at any_zpt.zpt file:
 <body>
  zaÅ¼Ã³Å‚Ä‡ gÄ™Å›lÄ… jaÅºÅ„ (text with national characters)
 </body>

prints when called:
Traceback (innermost last):

    * Module ZPublisher.Publish, line 115, in publish
    * Module ZPublisher.mapply, line 88, in mapply
    * Module ZPublisher.Publish, line 41, in call_object
    * Module Products.Five.browser.metaconfigure, line 404, in __call__
    * Module Shared.DC.Scripts.Bindings, line 313, in __call__
    * Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec
    * Module Products.PageTemplates.PageTemplateFile, line 121, in _exec
    * Module Products.PageTemplates.PageTemplate, line 89, in pt_render
    * Module zope.pagetemplate.pagetemplate, line 118, in pt_render
      Warning: Macro expansion failed
      Warning: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 99: ordinal not in range(128)
    * Module StringIO, line 271, in getvalue

 UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 99: ordinal not in range(128)

but this works well:

 <body>
    <span tal:replace="python:u'za\u017c\xf3\u0142\u0107 g\u0119\u015bl\u0105 ja\u017a\u0144\n'"></span> (text with national characters)
 </body>

UnitTest is attached.
==============================================================