[ZODB-Dev] Old transaction records with "bad" pickle data preventing packing

[Jim Fulton]

On Mon, Jun 14, 2010 at 3:43 AM, Jens Vagelpohl <jens at dataflake.org> wrote:
> Hi all,
>
> In a system using Zope 2.11 and ZODB 3.7.0b3 I'm having a problem with
> old transaction records containing pickle data that fails on
> cPickle.Unpickler.noload as employed by ZODB.serialize.referencesf, I
> get an AttributeError from inside the cPickle module:

It would be interesting to see the traceback.

> def referencesf(p, oids=None):
>    """Return a list of object ids found in a pickle
>
>    A list may be passed in, in which case, information is
>    appended to it.
>
>    Weak references are not included.
>    """
>
>    refs = []
>    u = cPickle.Unpickler(cStringIO.StringIO(p))
>    u.persistent_load = refs
>    u.noload()
>    u.noload()
>
>    # Now we have a list of referencs.  Need to convert to list of
>    # oids:
> <snip>
>
> The noload failure means I cannot pack the database right now. Since
> only historical transactions contain the problematic attribute and I
> don't care about those anymore

Note that:

- in ZODB 3.9, there's an option top pack without doing GC, and
- ZODB 3.9 ZEO servers can be used with older clients.


> I have introduced a simple change locally
> that ignores the failure:
>
> def referencesf(p, oids=None):
>    """Return a list of object ids found in a pickle
>
>    A list may be passed in, in which case, information is
>    appended to it.
>
>    Weak references are not included.
>    """
>
>    refs = []
>    u = cPickle.Unpickler(cStringIO.StringIO(p))
>    u.persistent_load = refs
>    try:
>        u.noload()
>        u.noload()
>    except AttributeError:
>        pass
>
>    # Now we have a list of referencs.  Need to convert to list of
>    # oids:
> <snip>

That's a bad idea.


> With this change I can pack the database and a few simple check
> (fsrefs.py etc) indicate the database is consistent after the pack.

Cool. You got lucky. :)

> My question: Is there any risk associated with ignoring the failure
> during packing?

Yes.  You might ignore references to objects causing them to be
removed while still used.

> The latest object versions for the persistent objects in
> question do not have the problematic attribute anymore, so the latest
> records are "safe" and will not cause the AttributeError, anyway.

Right, it worked out for your DB.

Jim


-- 
Jim Fulton