[ZODB-Dev] Encoded strings as keys for OOBTree in 2.7.3

[Tim Peters]

[Chris Withers]
> Interesting problem with an OOBTree, in this case the _index attribute of
> a FieldIndex in a Zope 2.7.3 instance.
>
> doing self._index.get('tést') (you can get the é by doing Alt Gr - e ;-)
> resulted in a UnicodeDecodeError.
>
> The bizarre thing is that:
>
> t = OOBTree()
> t.get('tést')
>
> returns None as opposed to raising the exception, which is what I'd
> expect.

There's nothing exceptional about that.  You're indexing an OOBTree by a
string object, and the BTree couldn't care less about the content of the
string.  "Funny characters", control characters, NUL bytes, ..., they're all
fine in a Python str object.

> Looking at the keys of the defective btree showed keys that were a
> mixture of strings and unicodes.
>
> Any idea as to what could be causing this?

The specific BTree you have in hand is mixing objects of types str and
unicode as keys.  That's given as an example of something that should not be
done, here:

    http://www.zope.org/Wikis/ZODB/FrontPage/guide/node6.html

Read the "Total Ordering and Persistence" section, esp.

    It's important to realize that even if two types satisfy the rules
    on their own, mixing objects of those types may not.  For example,
    8-bit strings and Unicode strings both supply total orderings, but
    mixing the two loses trichotomy; e.g., 'x' < chr(255) and u'x' == 'x',
    but trying to compare chr(255) to u'x' raises an exception.
    Partly for this reason (another is given later), it can be dangerous
    to use keys with multiple types in a single BTree-based structure.
    Don't try to do that ...

I don't know why a FieldIndex would be mix key types; you'll probably have
better lucking pursuing that on a Zope list.