[ZODB-Dev] zeopack error

[Marius Gedminas]

On Thu, Feb 09, 2012 at 01:25:48AM +0200, Marius Gedminas wrote:
> On Wed, Feb 08, 2012 at 01:24:55PM +0100, Kaweh Kazemi wrote:
> > Recap: last week I examined problems I had packing our 4GB users
> > storage.
...
>     >>> unp = pickle.Unpickler(f)
>     >>> unp.persistent_load = lambda oid: '<persistent reference %r>' % oid
>     >>> pprint.pprint(unp.load())
>     {'data': {"<persistent reference ['m', ('game', '\\x00\\x00\\x00\\x00\\x00\\x00\\tT', <class '__main__.Tool'>)]>": 1,
>               "<persistent reference ['m', ('game', '\\x00\\x00\\x00\\x00\\x00\\x00\\x12\\x03', <class '__main__.EnergyPack'>)]>": 1}}
> 
> Those look like cross-database references to me.
> 
> The original error (aaaugh Mutt makes it hard for me to look upthread
> while I'm writing a response) was something about non-hashable lists?
> Looks like a piece of code is trying to put persistent references into a
> dict, which can't possibly work in all cases.
...
> > During my checks I realized that running the pack in a Python 2.7
> > environment (using the same ZODB version - 3.10.3) works fine, the
> > pack reduces our 4GB storage to 1GB. But our production server uses
> > Python 2.6 (same ZODB3.10.3) which yields the problem (though the test
> > had been done on OS X 10.7.3 - 64bit, and the production server is
> > Debian Squeeze 32bit).
> 
> I've no idea why running the same ZODB version on Python 2.7 instead of
> 2.6 would make this error go away.

Duh!  The code that fails is in the standard library -- in the cPickle
module:

> > Traceback (most recent call last):
...
> >   File "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/FileStorage/fspack.py", line 328, in findrefs
> >     return self.referencesf(self._file.read(dh.plen))
> >   File "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/serialize.py", line 630, in referencesf
> >     u.noload()
> > TypeError: unhashable type: 'list'

Since the bug is in the stdlib, it's not surprising that the newer
stdlib cPickle from Python 2.7 fixes it.

Looks like it was this one: http://bugs.python.org/issue1101399
(hg changeset 56532:d0f005e6fadd)

When I diff Modules/cPickle.c between the 2.6 and 2.7 branches in the hg
repository, I see that noload() handles the APPEND, APPENDS, SETITEM and
SETITEMS differently in 2.7.  hg blame attributes that change to 56532.
The SETITEM/SETITEMS opcode in the pickle now no longer tries to
actually stuff anything into any dicts, which is why the list never ends
up being hashed, which explains the error and why it is gone.

The fix is present in 2.7, but not in 2.6

That was fun.

Marius Gedminas
-- 
Critical sections are Windows's lightweight alternative to mutexes, thus
cleverly getting more use out of a previously well-defined phrase by using it
to mean something related to but slightly different from the earlier general
understanding
        -- Verity Stob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://mail.zope.org/pipermail/zodb-dev/attachments/20120209/61440804/attachment.sig>