[Zope-CMF] Re: [dev] encoding issues: showstoppers for the CMF 1.5 release?

[yuppie]

Hi Tres!


Tres Seaver wrote:
> yuppie wrote:
> 
>> A.) RSS Syndication fail on non-ascii
>> -------------------------------------
>> http://collector.zope.org/CMF/261
>>
> [...]
> 
> We could conceivably fix the first one by modifying the syndication 
> tool, making its 'getSyndicatableContent' method return a list of 
> dictionaries whose values were unicode strings.  Or, we might add a
> method, 'toUnicode', to the tool, and call that from within RSS.py.
> In either case, the tool would  for the context's 'default_encoding' 
> property;  if found, it would be used to decode the value.

- RSS.py fetches strings from different sources, so I'd propose to do 
the decoding there.

- The implicit encoding used by Page Templates is ascii, so I think it 
wouldn't hurt to set it explicitly as default 'default_encoding'.

Did I get you right that you'd prefer a context property over a site 
property? That makes things a bit more complex, but if there is a need 
to set the encoding depending on the context I'm fine with that.


>> B.) CMFSetup: import of type infos broken
>> -----------------------------------------
>> http://collector.zope.org/CMF/287
>>
> [...]
> 
> Note that both import and export contexts used by the CMFSetup tool have 
> a 'getEncoding' method, and that the importer plugins are expected to 
> use that value when manipulating XML data files. 

Well. The code tells me something different. Export contexts are 
completely unaware of encodings, and the assumption that all strings are 
ascii is hardcoded in many places.

But regarding imports you are right and that's all we need to fix #287.

> The way to achieve 
> what you want here is to supply and explicit encoding when setting the 
> profile on the tool (pass the 'encoding' argument to 
> 'setProfileDirectory').  We could make that default to 'ascii', but that 
> would be the only reasonable default.
> 
>  We *do* need to extend the tool's 'manage_udpateToolProperties' to take 
> an encoding argument, as well as the ZMI template which calls it.

Can't see how the encoding is related to the profile. I would propose to 
add a getEncoding method to the tool and use it in runImportStep / 
runAllImportSteps. But maybe I'm missing something.

> [...]
> 
>> What would be a good default encoding? I guess it might sometimes not 
>> be the same as the 'default_charset' for content. Should we restrict 
>> configuration data to ascii?
> 
> 
> That is the only reliable default.  the 'default_charset' hack is 
> intended to be a transitional thing;  I would rather not depend on it.

Don't think that exporting non-ascii configuration data will be 
implemented for CMF 1.5.0, so restricting imports as well to ascii 
wouldn't hurt much.

All I want for now is that imported values are converted to strings. 
Making the charset used for encoding configurable and using that charset 
also for exports can be done later.


Any comments?

Cheers,
	Yuppie