[Zope] Searching Catalog for Unicode Values of FieldIndex

[Mohsen Moeeni]

On Sun, 15 Aug 2004 22:35:00 +0200, Dieter Maurer <dieter at handshake.de> wrote:
> Mohsen Moeeni wrote at 2004-8-14 14:19 +0430:

> >While searching a catalog for a FieldIndex which has a value out of ASCII range,
> >one would encounter the following error:
> >
> >'ascii' codec can't decode byte 0xd9 in position 0: ordinal not in range(128)
> 
> Look at the traceback. Always do this when you get errors!

I did. For others who might want to see the traceback I wrote an URL to 
another message on the list, because tracebacks make an email ugly :-)

> 
> Almost surely, your index contains a non ASCII and non unicode value.
> When the index lookup compares the keys, the non ASCII value
> is converted into unicode (as there is another unicode operand)
> and this fails.

I must admit, the word ``Unicode`` is not very thoughtful in my email. It's a
normal but ``utf-8``-encoded string which contains non-ASCII characters.
So you are right.

> You can use a "Managable FieldIndex". It allows you to convert
> all values to unicode with your chosen encoding (specifying the
> encoding could be easier -- maybe in the next release).

Okay I tried for some hours to make ManagableIndex work for me. Firstly,
I read the documentation. I found out I have 3 points to do this conversion.
None of them worked:

* ``python: value.decode('utf-8')`` as the ValueProvider Normalizer raised no
 exception and I could find the object in catalog. However the object had
 no value under the interested index.

* ``python: value.decode('utf-8')`` as Term Prenomalizer and Normalizer raised
an ``AttributeError`` with value `` 'unicode' object has no attribute 'decode'``
upon creating the object instance. See end of the email for the traceback.

I do not know why the Index term is converted to ``Unicode`` when it comes to
Term Prenormalization. My code (which is an Archetypes product) does nothing
to make this happen. So I wonder what I am missing. 

I thought Term Prenomalizer would be the ideal place to make the
conversion happen.
Because, If I understand it right, the conversion will also apply to the terms
provided for searching the index (``_apply_index`` method if I am right).

Cheers,
Mohsen,


------
2004-08-17T18:40:24 ERROR(200) SiteError http://mysite.com/plone/createObject
Traceback (most recent call last):
 File "/usr/local/zope270/lib/python/ZPublisher/Publish.py", line 100,
in publish
   request, bind=1)
 File "/usr/local/zope270/lib/python/ZPublisher/mapply.py", line 88, in mapply
   if debug is not None: return debug(object,args,context)
 File "/usr/local/zope270/lib/python/ZPublisher/Publish.py", line 40,
in call_object
   result=apply(object,args) # Type s<cr> to step into published object.
 File "/usr/local/zope270/lib/python/Products/CMFFormController/FSControllerPythonScript.py",
line 88, in __call__
   result = FSControllerPythonScript.inheritedAttribute('__call__')(self,
*args, **kwargs)
 File "/usr/local/zope270/lib/python/Products/CMFFormController/Script.py",
line 141, in __call__
   return BaseFSPythonScript.__call__(self, *args, **kw)
 File "/usr/local/zope270/lib/python/Products/CMFCore/FSPythonScript.py",
line 104, in __call__
   return Script.__call__(self, *args, **kw)
 File "/usr/local/zope270/lib/python/Shared/DC/Scripts/Bindings.py",
line 306, in __call__
   return self._bindAndExec(args, kw, None)
 File "/usr/local/zope270/lib/python/Shared/DC/Scripts/Bindings.py",
line 343, in _bindAndExec
   return self._exec(bound_data, args, kw)
 File "/usr/local/zope270/lib/python/Products/CMFCore/FSPythonScript.py",
line 160, in _exec
   result = apply(f, args, kw)
 File "Script (Python)", line 16, in createObject
 File "/usr/local/zope270/lib/python/Products/CMFCore/PortalFolder.py",
line 363, in invokeFactory
   , kw
 File "/usr/local/zope270/lib/python/Products/CMFCore/TypesTool.py",
line 709, in constructContent
   ob = apply(info.constructInstance, (container, id) + args, kw)
 File "/usr/local/zope270/lib/python/Products/CMFCore/TypesTool.py",
line 401, in constructInstance
   return self._finishConstruction(ob)
 File "/usr/local/zope270/lib/python/Products/CMFCore/TypesTool.py",
line 299, in _finishConstruction
   ob.reindexObject(idxs=['portal_type', 'Type'])
 File "/usr/local/zope270/lib/python/Products/Archetypes/CatalogMultiplex.py",
line 60, in reindexObject
   c.catalog_object(self, self.__url(), idxs=lst)
 File "/usr/local/zope270/lib/python/Products/ZCatalog/ZCatalog.py",
line 513, in catalog_object
   update_metadata=update_metadata)
 File "/usr/local/zope270/lib/python/Products/ZCatalog/Catalog.py",
line 381, in catalogObject
   blah = x.index_object(index, object, threshold)
 File "/usr/local/zope270/lib/python/Products/ManagableIndex/ManagableIndex.py",
line 203, in index_object
   val= self._evaluate(obj)
 File "/usr/local/zope270/lib/python/Products/ManagableIndex/ManagableIndex.py",
line 441, in _evaluate
   return self._standardizeValue(v,object)
 File "/usr/local/zope270/lib/python/Products/ManagableIndex/ManagableIndex.py",
line 445, in _standardizeValue
   return self._standardizeTerm(value,object,1)
 File "/usr/local/zope270/lib/python/Products/ManagableIndex/ManagableIndex.py",
line 401, in _standardizeTerm
   value = self._prenormalizeTerm(value, object)
 File "/usr/local/zope270/lib/python/Products/ManagableIndex/ManagableIndex.py",
line 424, in _prenormalizeTerm
   return normalizer._normalize(value, object)
 File "/usr/local/zope270/lib/python/Products/ManagableIndex/Evaluation.py",
line 130, in _normalize
   return evaluator._evaluate(value,object)
 File "/usr/local/zope270/lib/python/Products/ManagableIndex/Evaluation.py",
line 86, in _evaluate
   v= EvalAndCall.inheritedAttribute('_evaluate')(self,value,object)
 File "/usr/local/zope270/lib/python/Products/ManagableIndex/Evaluation.py",
line 58, in _evaluate
   return expr(context)
 File "/usr/local/zope270/lib/python/Products/PageTemplates/ZRPythonExpr.py",
line 47, in __call__
   return eval(code, g, {})
 File "Python expression "value.decode('utf-8')"", line 1, in <expression>
AttributeError: 'unicode' object has no attribute 'decode'