[ZODB-Dev] Build compression into ZODB 3.11?

[Laurence Rowe]

On 14 March 2012 17:47, Jim Fulton <jim at zope.com> wrote:
> I'm pretty happy with how zc.zlibstorage has worked out.
>
> Should I build this into ZODB 3.11?

+1

> BTW, lz4 compression looks interesting.
>
> The Python binding (at least from PyPI) is broken.
> I submitted an issue. Hopefully it will be fixed.

FWIW, I experimented with c_zlib from https://gist.github.com/242459
in order to use a zlib default dictionary - a 32KB string used to
pre-fill the compression buffer.

Using a ~75MB Data.fs from a Plone site that compressed down to ~30MB
with zc.zlibstorage normally, the most successful dictionary I tried
was the end of the Data.fs itself which saved only an additional 6%
over an empty dictionary. That feels like an unfair test to me,
probably deduplicating serialized catalog bucket values. The next best
was the last 32KB from another Plone Data.fs which only managed to
save an additional 2.5% and a fairly short dictionary with common
pickled classes saved an additional 2%.

None of those savings seem worthwhile pursuing further given the extra
brittleness involved.

Laurence