[Zope-dev] Re: RFV: Unicode in Zope 2

[Florent Guillaume]

Martijn Faassen wrote:
> Jim Fulton wrote:
> 
>> Martijn Faassen wrote:
> 
> [snip]
> 
>>> I'll volunteer to help brainstorm on this, but right now my 
>>> brainstorm is only very dark and full of lightning.
>>
>>
>>
>> You and I brainstormed this a few months ago.  I think this was on the
>> list.  I think that, for starters, we would arrange that all Zope 3
>> views used in Zope 2 would get unicode input.  If you like, I can try
>> to find this discussion. :)
>>
> 
> Ah, right, that is far less scary, indeed. Your post somehow gave me the 
> impression you wanted to change the way current Zope 2 does things, but 
> if you limit yourself to what happens with Zope 3 stuff in Zope 2, it's 
> less scary.
> 
> In fact Five already has hacks to make sure that unicode enters 
> Five-generated forms. Replacing these hacks with something solid would 
> be good.
> 
>>> Anyway, in some basics, Zope 2 does have an approach to unicode for 
>>> *output* that's fairly similar to Zope 3's: if you use unicode 
>>> strings your entire output (including page templates) will be unicode 
>>> (if you don't mix with non-unicode non-ascii strings..). Then the 
>>> response encoding setting is read and everything is transformed once 
>>> to unicode text. Silva uses this. It also struggles to make sure all 
>>> its input is transformed to unicode (among other ways using Formulator).
>>>
>>> In Plone, the situation is quite different -- its 
>>> PlacelessTranslationService monkeypatches into the page template 
>>> engine and puts in ways so that you can mix UTF-8 and unicode strings 
>>> together. This then goes on to break assumptions of code that uses 
>>> the page template engine in a unicode-pure environment (which is what 
>>> happened to Silva).
>>
>>
>> Ick.
>>
>> I'm not suggesting this is easy.  We may have some messy deprecation
>> and backward compatibility code.  But we *do* need to solve this problem
>> eventually, and the solution doesn't get any closer without taking steps.
> 
> 
> Yes. I'm optimistic about being able to do this for Five-related stuff. 
> If this is eventually going to be people's main development system, then 
> we can basically say we've solved the important unicode issues.
> 
> What I'm worried about doing this for old code, but some steps will 
> probably become clear during the brainstorming session. Migration tools 
> that turn strings in the ZODB into unicode ones magically (with the 
> ability to spell out exceptions and encoding)? Tricky...

I'd start by adding some deprecation warnings at the places where 
PlacelessTranslationService (or Localizer for that matter) do their 
workarounds so that normally incompatible types are mixed together.

Florent

-- 
Florent Guillaume, Nuxeo (Paris, France)   CTO, Director of R&D
+33 1 40 33 71 59   http://nuxeo.com   fg at nuxeo.com