[Zope3-dev] test errors due to ascii defaultencoding assumption

[Dieter Maurer]

Tim Peters wrote at 2005-10-21 15:27 -0400:
> ...
>[Dieter Maurer]
>> Nowadays, life is hell without a sensible "setdefaultencoding":
>>
>>  Many isolated modules intersperse unicode in an otherwise
>>  "string" dominated world causing wide spread
>>  UnicodeDecodingErrors.

[Tim]
>And setdefaultencoding helps that?  What, you set it to latin1 just to
>squash the exceptions?

[Dieter]
No, because I know that I live in a "latin-1" subworld and that
almost all "str"s containing text (rather than binary data)
*ARE* indeed "latin-1" encoded.

[Tim]
> "Brrrrr", if so.  I suppose that's got a
>decent shot at working by accident, anyway.

[Dieter]
Up to now, I met a single "non accident" (you would add "<wink>"),
i.e. a single case where the "defaultencoding = 'latin-1'"
proved counter productive:

  Python's "xmlrpclib" tried to check whether it should
  convert the unicode data to an "str" with code like this:

	  try:
	    # try to return a simple 7bit string
	    return str(data)
	  except UnicodeError: return data

  Of course, this code only works as expected
  when "defaultencoding == 'ascii'".

  The XML-RPC called function would have correctly worked with Unicode
  and with an UTF-8 encoded "str" -- but it failed for a
  "latin-1" encoded "str".

  This problem was easily fixed by fixing the "xmlrpclib" code
  ("return data.encode('ascii')" rather than "return str(data)").
  
On the other hand, my "defaultencoding = 'latin-1'" saved me
dozens (if not hundreds) of "UnicodeDecodingError"s).
Thus, your "accident"s are appreciated incidents :-)

-- 
Dieter