[Zope-CMF] Re: [RFC] [Patch] GenericSetup and encodings

[Yves Bastide]

yuppie wrote:
> Hi!
> 
> 
> Yves Bastide wrote:
>> yuppie wrote:
>>>
>>> 3.) GenericSetup is not tested with non-ASCII UTF-8 site settings. 
>>> AFAIK import works, but not export. I consider this a bug.
[...]
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xf4 in position 
>> 20: ordinal not in range(128)
> 
> This traceback just confirms that export does not work. Is import also 
> broken?

Differently: it may or may not raise ... And Zope treats properties as 
iso8859-15 anyway.

Fresh install of Zope trunk (after a long struggle; make instance now 
works but make install is broken?) and CMF trunk, with

~/src/CMF$ svn diff
Index: CMFDefault/profiles/default/properties.xml
===================================================================
--- CMFDefault/profiles/default/properties.xml  (revision 68514)
+++ CMFDefault/profiles/default/properties.xml  (working copy)
@@ -1,6 +1,6 @@
  <?xml version="1.0"?>
  <site>
- <property name="title">Portal</property>
+ <property name="title">Portàl</property>
   <property name="description"></property>
   <property name="email_from_address"
      type="string">postmaster at localhost</property>

Fails when CMFDefault.factory.addConfiguredSite calls createSnapshot.


Here's a minimal patch for GenericSetup not to raise on the previous 
case (Demonstration product. Not for sale.)

zeb at atone:~/src/CMF$ svn diff GenericSetup/
Index: GenericSetup/context.py
===================================================================
--- GenericSetup/context.py     (revision 68514)
+++ GenericSetup/context.py     (working copy)
@@ -475,7 +475,7 @@
          if isinstance( body, unicode ):
              encoding = self.getEncoding()
              if encoding is None:
-                body = body.encode()
+                body = body.encode('UTF-8')
              else:
                  body = body.encode( encoding )

Index: GenericSetup/utils.py
===================================================================
--- GenericSetup/utils.py       (revision 68514)
+++ GenericSetup/utils.py       (working copy)
@@ -625,6 +625,8 @@
              else:
                  if prop_map.get('type') == 'boolean':
                      prop = str(bool(prop))
+                elif isinstance(prop, str):
+                    prop = prop.decode('UTF-8')
                  elif not isinstance(prop, basestring):
                      prop = str(prop)
                  child = self._doc.createTextNode(prop)
zeb at atone:~/src/CMF$

With this applied, Portàl (u'Port\xe0l'), which becomes 'Port\xc3\xa0l', 
is displayed as Portà l ... Zope does input--output properties in utf-8, 
but stores them in iso8859.  Sigh.

> 
>> Thanks for setting me right. What's the usefulness of getEncoding()? 
>> As you say, exported files don't need to be other than utf-8 encoded.
> 
> I guess it just exists for historical reasons.

Might it be removed, or default'ed to utf-8? Do people already rely on it?

> 
>> Well, I think I can wriggle out of most of my problems using 
>> translation. And I'll try and write UTF-8 unit tests if nobody beats 
>> me to it.
> 
> That would be great.

Hmm, by adding to an existing test suite, or creating a new one?

> 
> 
> Cheers,
> 
>     Yuppie

Thanks,

yves