[Checkins] [zopefoundation/ZODB] a00d35: Python 3: pickle bytestrings using SHORT_BINSTRING

[GitHub]

  Branch: refs/heads/py3
  Home:   https://github.com/zopefoundation/ZODB
  Commit: a00d35fe1873357bc621fc29013b6e2ca1bd0b0f
      https://github.com/zopefoundation/ZODB/commit/a00d35fe1873357bc621fc29013b6e2ca1bd0b0f
  Author: Marius Gedminas <marius at gedmin.as>
  Date:   2013-03-05 (Tue, 05 Mar 2013)

  Changed paths:
    M setup.py
    M src/ZODB/ConflictResolution.py
    M src/ZODB/ExportImport.py
    M src/ZODB/FileStorage/tests.py
    M src/ZODB/_compat.py
    M src/ZODB/fsIndex.py
    M src/ZODB/serialize.py
    M src/ZODB/tests/PackableStorage.py
    M src/ZODB/tests/test_fsdump.py
    M src/ZODB/tests/testblob.py
    M src/ZODB/tests/testfsoids.py

  Log Message:
  -----------
  Python 3: pickle bytestrings using SHORT_BINSTRING

This uses bytes_as_strings=True option introduced in zodbpickle 0.2 for
this purpose.

This way pickles produced on Python 3 are nearly the same as on Python 2.
There are some slight differences (Python 3 seems to perform more
memoizations which grows the size of some pickles by a couple of bytes),
but they're immaterial.

Now we can use zodbpickle's noload() on Python 3 to scan pickles for
persistent references.  We couldn't do that before, because Python 3
normally pickles byte strings as calls to codecs.encode(u'latin1-data',
'latin-1'), and noload() doesn't interpret the REDUCE opcode involved in
that representation.

Note that when you're pickling byte strings using bytes_as_strings=True,
you have to load them using encoding='bytes' (which breaks instances, so
cannot be used) or using errors='bytes' (which mean some bytestrings may
get unpickled as unicode instead).  I've tried hard to discover every
place that unpickles OIDs and added conversion to bytes in those places.

Applications dealing with binary data be prepared to handle bytestrings
that unexpectedly become unicode on unpickling.  That's the price of
Python 2 compatibility.