[Zope-CMF] Charsets

[Charlie Clark]

Am 13.12.2008 um 18:40 schrieb Charlie Clark:

> Hi,
>
> I'm struggling with the way formlib forms handle decoding from forms.
> It looks like this gets set in BrowserView using an
> IUserPreferredCharsets adapter. The default adapter seems to be in
> zope.publisher.http and it looks like latin-1 will be set if there is
> no other charset and I'm having problems with the em-dash and en-dash
> (u'\u2013' and u'\u2013') characters automatically being converted
> from latin-1 when they are being entered as cp1252. For content that
> doesn't through this decoding I have no problems if zpublisher- 
> default-
> encoding is set to cp1252 and the default_charset is set to cp1252 as
> well: decoding with CMFDefault.utils.decode() works just fine.
>
> I suspect I'm missing something basic in the way charsets are handled
> but as it's a windows only IE6 environment, is the easiest solution
> writing an adapter that defaults to cp1252 if there is no
> HTTP_ACCEPT_CHARSET in the request header?


Overriding the adapter works fine. I'm still a bit confused by the  
original code:

         # Quoting RFC 2616, $14.2: If no "*" is present in an Accept- 
Charset
         # field, then all character sets not explicitly mentioned get a
         # quality value of 0, except for ISO-8859-1, which gets a  
quality
         # value of 1 if not explicitly mentioned.
         # And quoting RFC 2616, $14.2: "If no Accept-Charset header is
         # present, the default is that any character set is  
acceptable."
         if not sawstar and not sawiso88591 and header_present:
             charsets.append((1.0, 'iso-8859-1'))

So if Accept-Charset is '' then Zope will set the browser charset to  
Latin-1. This seems a little strange to me as the default is UTF-8 if  
the header is missing. And if the RFC does say the default is "any  
character set is possible" then I would have thought UTF-8 would be  
okay. Is this a possible bug?

Charlie
--
Charlie Clark
Helmholtzstr. 20
Düsseldorf
D- 40215
Tel: +49-211-938-5360
GSM: +49-178-782-6226