[ZODB-Dev] ZODB alternate serialization format patch

[Shane Hathaway]

Hanno Schlichting wrote:
> Shane Hathaway wrote:
>> I have just created a patch for ZODB that makes the object serialization 
>> format pluggable, meaning that it should now be possible to write Python 
>> code that causes ZODB to store in formats other than Python pickles.
> 
> Cool :)
> 
>> In addition to making the data format pluggable, the patch puts most of 
>> ZODB's dependencies on the cPickle module in one place, so now if we 
>> decide to improve our usage of the cPickle module, we can make that 
>> change in one place rather than several.
> 
> I'm wondering if this would be a good opportunity to include a version
> marker in addition to the protocol format to the API's?

Maybe.  Starting with pickle protocol version 2, pickles do start with a 
version header.

> The ZODB still uses version one of the pickle protocol throughout and so
> far it has been complicated and cumbersome to change this in any way.

Is it cumbersome?  I tried changing to pickle protocol v2 while making 
this patch.  Only 4 of the 3000+ ZODB tests failed.  One of them failed 
because we have some pickle introspection code that does not yet 
understand protocol 2; that should be easy to fix.  Another failed 
because it seems to depend on the exact length of a generated pickle, 
which is a bit silly.  I didn't look at the rest of the test failures 
because I assume they are all similarly superficial.

> While the documentation of Protocol Buffers mention that they try to be
> stable and avoid incompatible versions, I don't trust any standard to be
> so generic that it can avoid incompatible changes over a period of many
> years.

The patch I made might cover this.  After this patch, ZODB will look for 
the serializer format name in curly braces at the beginning of each 
serialized object.  That format name could include a version number if 
necessary.

BTW, I've learned that Google's protocol buffers lack the concept of a 
mapping, and I think that might be a major problem for my current 
project.  We might switch to a different language agnostic format like 
Thrift or JSON.

Shane