[Checkins] SVN: five.pt/trunk/ Encode string input for restricted Python to a byte string. This fixes encoding issues with non-ascii input (issue #60).

[Malthe Borch]

Log message for revision 122854:
  Encode string input for restricted Python to a byte string. This fixes encoding issues with non-ascii input (issue #60).

Changed:
  U   five.pt/trunk/CHANGES.txt
  U   five.pt/trunk/src/five/pt/expressions.py

-=-
Modified: five.pt/trunk/CHANGES.txt
===================================================================
--- five.pt/trunk/CHANGES.txt	2011-09-20 12:25:16 UTC (rev 122853)
+++ five.pt/trunk/CHANGES.txt	2011-09-20 12:42:32 UTC (rev 122854)
@@ -1,6 +1,12 @@
 Changelog
 =========
 
+In next release ...
+
+- Fixed encoding issue with restricted Python expression. The Python
+  2.4 AST parser does not accept unicode input and the expression
+  string must be explicitly encoded to a byte string.
+
 2.1.5 (2011-08-11)
 ~~~~~~~~~~~~~~~~~~
 

Modified: five.pt/trunk/src/five/pt/expressions.py
===================================================================
--- five.pt/trunk/src/five/pt/expressions.py	2011-09-20 12:25:16 UTC (rev 122853)
+++ five.pt/trunk/src/five/pt/expressions.py	2011-09-20 12:42:32 UTC (rev 122854)
@@ -224,10 +224,12 @@
 
     def parse(self, string):
         decoded = decode_htmlentities(string)
-        node = ast24_parse(decoded, 'eval').node
+        encoded = decoded.encode('utf-8')
+        node = ast24_parse(encoded, 'eval').node
         MutatingWalker.walk(node, self.rm)
         string = generate_code(node)
-        value = super(UntrustedPythonExpr, self).parse(string)
+        decoded = string.decode('utf-8')
+        value = super(UntrustedPythonExpr, self).parse(decoded)
 
         # Run restricted python transform
         self.rt.visit(value)