[ZCM] [ZC] 1490/ 6 Assign "default page encoding hardcoded to latin1"

[Collector: Zope Bugs, Features, and Patches ...]

Issue #1490 Update (Assign) "default page encoding hardcoded to latin1"
 Status Accepted, Zope/feature medium
To followup, visit:
  http://collector.zope.org/Zope/1490

==============================================================
= Assign - Entry #6 by chrisw on Oct 3, 2004 4:25 am

 Status: Pending => Accepted

 Supporters added: chrisw, htrd

The mailing list converation didn't really kick off, so putting Toby's comments in here, replying to them and assigning us both s owe can sort it out, probably on the next bug day...

Toby said:
>>>nor should it. manage_page_charset is a transition device for management
>>>> >   pages that are not unicode-aware. 
>>
>>> 
>>> Well, and if they are? Where do we config the charset we want
>>> to have in the browser? In each of these to be written
>>> unicode-aware products separately?
> 
> 
> 1. human language text should be unicode strings.
> 2. dtml renders unicode responses from unicode properties just nicely.

When you say "render", what do you mean, and what character encoding is used?

> 3. any unicode-aware 'page' needs to set its preferred encoding in 
> content-type header. 

OK, but what is you have loads of pages, but where the entire server serves content in a particular character encoding? Where should a global default policy be set?

> Typically this is a one-off standard_html_header for 
> content pages.

And page templates?

> Thats how it was intended to be used. Im out of touch with how many management 
> pages have been updated to work that way, but I guess 'too few'. (outside my 
> custom products, which happily migrated to all unicode around 2001). I agree 
> that this system is only so much bloat and hassle unles the majority are 
> upgraded to use it.

Hmm, your comments speak for themselves ;-)

>>> But for general use there should be a more or less
>>> straight forward way to configure by the user.
> 
> Any configurable charset is building a legacy problem for future versions.

How so? 

> The 
> only sane approach is for text to be stored in unicode strings, not plain 
> strings. No configuration is needed once that is done, its not hard to do, 
> and configurable charsets will make that transition harder.

?! The configurable characterset I'm talking about is setting the encoding ZPublisher uses when it sends the content to the browser. This has to happen regardless of what encoding is used to store text, even if that's unicode. 

This is currently set to latin1, which seems nastilly hard coded to me :-(
________________________________________
= Comment - Entry #5 by tino on Sep 2, 2004 8:56 am

> = Comment - Entry #4 by htrd on Sep 2, 2004 8:32 am
> 
> > A config option would be nice.
> 
> Grrrr

not so grr ;) There are many confusing options regarding 
charsets already in pleace. There is a locales
configuration which can include encoding too.
There are 2 config options for structured text 
(in and out).
> 
> > There is a manage_page_charset key which can be set
> > via property, 
> 
> Yes.
> 
> > but it is not respected by the publishing
> > process
> 
> Its not supposed to. The publishing process *does* respect the
>   Content-Type header. 
> > (which renders it quite useless)
> 
> for that purpose, yes. that is not its purpose.

Cool. You edit all content with that charset, meaning
properties, titles, content of templates and documents
and whatnot, but when you hit "view" you get a totally 
mess. Very nice ;)
> 
> > and you cannot add charset to pagetemplates content-type
> > (which is another bug imho).
> 
> agreed
>  
> > Also ZSQLMethods dont use mage_page_charset.
> 
> nor should it. manage_page_charset is a transition device for management
>   pages that are not unicode-aware. 

Well, and if they are? Where do we config the charset we want
to have in the browser? In each of these to be written
unicode-aware products separately?


> Urgh, this is all a mess. That documentation I linked _is_ out of date in
>   some respects. :-( What we need IMO is for the documentation (and code,
>   if necessary) to be updated by someone who understands how all this is
>   supposed to be used, rather than (no offense intended) those who
>   encounter problems because they dont understand, and feel an urge to fix.

I got some private patches which resolve most of the issues 
for me. But for general use there should be a more or less
straight forward way to configure by the user.

>   sadly I dont have time for this any more.
> 
> ________________________________________
> = Comment - Entry #3 by tino on Sep 2, 2004 8:20 am
> 
> A config option would be nice.
> There is a manage_page_charset key which can be set
> via property, but it is not respected by the publishing
> process (which renders it quite useless) and you cannot add charset to
>   pagetemplates content-type (which is another bug imho).
> 
> Also ZSQLMethods dont use mage_page_charset.
> ________________________________________
> = Comment - Entry #2 by htrd on Sep 2, 2004 6:47 am
> 
> > HTTPResponse.py contains a hard-coded default to latin1 in
> > line 436.
> 
> yes
>  
> > This should really come from a zope.conf option.
> 
> As architect of this original design, I say "Definitely Not".
> 
> The people who are in control of zope.conf already have good places to
>   express their preference. Having this value in this place always be
>   latin-1 is essential for robust Products. 
> 
> A seperate problem (that I no longer have effort to fight for) is that my
>   documentation patches explaining how to use this mechanism never got
>   merged :-( 
> http://zope.org/Members/htrd/howto/unicode
> http://zope.org/Members/htrd/howto/unicode-zdg-changes
> 
> 
> ________________________________________
> = Request - Entry #1 by chrisw on Sep 2, 2004 6:14 am
> 
> HTTPResponse.py contains a hard-coded default to latin1 in line 436.
> 
> This should really come from a zope.conf option.
________________________________________
= Comment - Entry #4 by htrd on Sep 2, 2004 8:32 am

> A config option would be nice.

Grrrr

> There is a manage_page_charset key which can be set
> via property, 

Yes.

> but it is not respected by the publishing
> process

Its not supposed to. The publishing process *does* respect the Content-Type header. 

> (which renders it quite useless)

for that purpose, yes. that is not its purpose.

> and you cannot add charset to pagetemplates content-type
> (which is another bug imho).

agreed
 
> Also ZSQLMethods dont use mage_page_charset.

nor should it. manage_page_charset is a transition device for management pages that are not unicode-aware. 

Urgh, this is all a mess. That documentation I linked _is_ out of date in some respects. :-( What we need IMO is for the documentation (and code, if necessary) to be updated by someone who understands how all this is supposed to be used, rather than (no offense intended) those who encounter problems because they dont understand, and feel an urge to fix. sadly I dont have time for this any more.

________________________________________
= Comment - Entry #3 by tino on Sep 2, 2004 8:20 am

A config option would be nice.
There is a manage_page_charset key which can be set
via property, but it is not respected by the publishing
process (which renders it quite useless) and you cannot add charset to pagetemplates content-type (which is another bug imho).

Also ZSQLMethods dont use mage_page_charset.
________________________________________
= Comment - Entry #2 by htrd on Sep 2, 2004 6:47 am

> HTTPResponse.py contains a hard-coded default to latin1 in
> line 436.

yes
 
> This should really come from a zope.conf option.

As architect of this original design, I say "Definitely Not".

The people who are in control of zope.conf already have good places to express their preference. Having this value in this place always be latin-1 is essential for robust Products. 

A seperate problem (that I no longer have effort to fight for) is that my documentation patches explaining how to use this mechanism never got merged :-( 
http://zope.org/Members/htrd/howto/unicode
http://zope.org/Members/htrd/howto/unicode-zdg-changes


________________________________________
= Request - Entry #1 by chrisw on Sep 2, 2004 6:14 am

HTTPResponse.py contains a hard-coded default to latin1 in line 436.

This should really come from a zope.conf option.
==============================================================