[ZODB-Dev] ZODB alternate serialization format patch

[Hanno Schlichting]

Shane Hathaway wrote:
> Hanno Schlichting wrote:
>> Shane Hathaway wrote:
>>> In addition to making the data format pluggable, the patch puts most of 
>>> ZODB's dependencies on the cPickle module in one place, so now if we 
>>> decide to improve our usage of the cPickle module, we can make that 
>>> change in one place rather than several.
>> I'm wondering if this would be a good opportunity to include a version
>> marker in addition to the protocol format to the API's?
> 
> Maybe.  Starting with pickle protocol version 2, pickles do start with a 
> version header.

Reading pickles is backwards compatible. But specifying the version in
which to write pickles needs explicit configuration.

>> The ZODB still uses version one of the pickle protocol throughout and so
>> far it has been complicated and cumbersome to change this in any way.
> 
> Is it cumbersome?  I tried changing to pickle protocol v2 while making 
> this patch.

Without this patch it was cumbersome, as the number of places to change
the pickle version was many. Now it probably has become a lot easier ;)

> Only 4 of the 3000+ ZODB tests failed.  One of them failed 
> because we have some pickle introspection code that does not yet 
> understand protocol 2; that should be easy to fix.  Another failed 
> because it seems to depend on the exact length of a generated pickle, 
> which is a bit silly.  I didn't look at the rest of the test failures 
> because I assume they are all similarly superficial.

That sounds easy to fix.

>> While the documentation of Protocol Buffers mention that they try to be
>> stable and avoid incompatible versions, I don't trust any standard to be
>> so generic that it can avoid incompatible changes over a period of many
>> years.
> 
> The patch I made might cover this.  After this patch, ZODB will look for 
> the serializer format name in curly braces at the beginning of each 
> serialized object.  That format name could include a version number if 
> necessary.

Right. I was wondering if it would be a good idea to build this in from
the start. My impression is that every data format of any kind of
non-trivial complexity will have multiple incompatible versions of the
same spec at some point. Maybe it is YAGNI as the formats themselves
have a version mechanism in them, though.

Hanno