[Zope3-dev] Unicode handling in Zope3 Page Templates

[Martijn Faassen]

Sidnei da Silva wrote:
[snip]
> | 
> | unicode(text)
> | 
> | with
> | 
> | isinstance(text, basestring) and text or unicode(text)
> 
> Yes, that's certainly wrong now that I look at it. I think that's what
> Jim was referring to?
> 
> | ..
> | 
> | plain unicode(text), like str(text), doesn't typically work in unicode 
> | agnostic code.
> | 
> | Trying to reconstruct the logic in more readable form (which is 
> | difficult, indicating that this code shouldn't be employed :), Sidnei's 
> | code looks like this, I think..:
> 
> Did you mean 'deployed' above? :)

no, I meant 'employed' as in 'to make use of'. :)

[snip]
> | Anyway, what would work better is the following:
> | 
> | if ininstance(text, basestring):
> |     result = text
> | else:
> |     result = str(text)
> 
> Changed to use this form.
> 
> | As long as str(text) == str(unicode(text)) is True (and doesn't fail 
> | with a unicode error), this will at least work correctly in both unicode 
> | mode (as it can deal with plain-ascii) as well as encoded mode (as it 
> | can deal with plain ascii).
> | 
> | For built-ins outside unicode strings, str(text) == str(unicode(text)) I 
> | think always applies. The problem remains with other objects which are 
> | not built-ins which may want to return unicode strings; i.e. custom 
> | objects which define __unicode__(). Perhaps i18n-ed strings? -- that's 
> | another good candidate for a test.
> 
> Not sure what's going to happen in this case. I think str() doesn't
> even look at __unicode__()? 

Don't think it does, and even if it does, you *want* unicode in that 
case, and str() is (conceptually) guaranteed to give a classic string as 
the result.

> Then we probably need your solution below.

in which case we do need to inform the engine explicitly what mode we're 
using.. And of course things are still odd, as an i18n string may only 
be able to present itself as unicode.

> I've added a test using i18n:translate, and then found that the base
> 'Context' object doesn't have a translate() method, even though the
> page template will accept and execute i18n:translate commands. I've
> added one that just returns the msgid. It should probably return a
> unicode string there? In which case we are hosed if i18n:translate
> tags are used in the template, as that's guaranteed to return unicode
> strings AFAICS.

I don't know enough about the i18n system, I just figured it might be a 
source of unicode data. :)

Regards,

Martijn