[Zope] Re: help debugging a "can't pickle" error deep within a catalog reindex

Ross Patterson me at rpatterson.net
Tue Jun 17 21:42:25 EDT 2008


Rob Miller <ra at burningman.com> writes:

> hi,
>
> i'm trying to perform a ZCatalog.refreshCatalog() on a catalog with
> over 29,000 indexed objects.  it churns for a good long time, and
> eventually fails with a long set of tracebacks, of which i've included
> a sample at the end of this message.
>
> i think i understand the gist of the issue... it's trying to write an
> object (probably a CatalogBrain) to the database, but this object's
> __dict__ contains a value that is of type instancemethod, which isn't
> allowed for persistent objects.
>
> the problem is that i can't figure out which specific objects are
> causing the problem.  i've used pdb.post_mortem to get a debug prompt
> way down in the traceback, but the code goes in and out of C modules,
> so i'm missing a lot of what's happening.  and when i interactively
> peek at the objects that are being indexed when the error happens,
> there doesn't seem to be anything wrong, and i can index the objects
> w/ no problem.  i've even tried dropping the subtransaction threshold
> down to 1, so it will try to commit a savepoint after every object,
> but none of the objects being indexed seemed to have any problems.
>
> i CAN verify that the instancemethod that is causing the problem
> renders like this:
>
> <bound method SessionDataManager.getSessionData of <SessionDataManager
> at /session_data_manager>>
>
> even that hasn't proven enough for me to concretely identify the
> source of the problem, though.
>
> i've been working on this for a full day already, and am not sure how
> to proceed.  does anyone have any debugging tips that might help me
> figure out what, exactly, is causing the reindex attempts to blow up?
>
> thanks!

Can you get a pdb.post_mortem prompt at the actual ZODB/serialize.py:416
error or only at transaction/_transaction.py:267 where some sort of
previous error is handled?

If the former, I often find it informative to step up to
ZODB/serialize.py:407 where obj.__getstate__() is called and find out
what obj is.  Is it the same object every time?  If not, is it of the
same type every time?

A next step can also be to put a pdb.set_trace() at
transaction/_transaction.py:340 in the register() method such that it's
is only called when the offending object is added to the transaction.

Hope some of that helps,
Ross

> Traceback (most recent call last):
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py",
> line 296, in refreshCatalog
>     self.catalog_object(obj, p, pghandler=pghandler)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CMFPlone/CatalogTool.py",
> line 367, in catalog_object
>     self._increment_counter()
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CMFPlone/CatalogTool.py",
> line 395, in _increment_counter
>     self._counter.change(1)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/BTrees/Length.py",
> line 55, in change
>     self.value += delta
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py",
> line 890, in register
>     self._register(obj)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py",
> line 900, in _register
>     self.transaction_manager.get().join(self)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py",
> line 273, in join
>     self._prior_operation_failed() # doesn't return
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py",
> line 267, in _prior_operation_failed
>     raise TransactionFailedError("An operation previously failed, "
> TransactionFailedError: An operation previously failed, with traceback:
>
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZServer/PubCore/ZServerPublisher.py",
> line 25, in __init__
>     response=b)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py",
> line 401, in publish_module
>     environ, debug, request, response)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py",
> line 202, in publish_module_standard
>     response = publish(request, module_name, after_list, debug=debug)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py",
> line 119, in publish
>     request, bind=1)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/mapply.py",
> line 88, in mapply
>     if debug is not None: return debug(object,args,context)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py",
> line 42, in call_object
>     result=apply(object,args) # Type s<cr> to step into published object.
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py",
> line 260, in manage_catalogReindex
>     self.refreshCatalog(clear=1, pghandler=handler)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py",
> line 296, in refreshCatalog
>     self.catalog_object(obj, p, pghandler=pghandler)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CMFPlone/CatalogTool.py",
> line 385, in catalog_object
>     update_metadata, pghandler=pghandler)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CacheSetup/patch.py",
> line 96, in catalog_object
>     uid, idxs, update_metadata, pghandler)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CacheSetup/patch_utils.py",
> line 6, in call
>     return getattr(self, PATTERN % __name__)(*args, **kw)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py",
> line 560, in catalog_object
>     transaction.savepoint(optimistic=True)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_manager.py",
> line 110, in savepoint
>     return self.get().savepoint(optimistic)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py",
> line 312, in savepoint
>     self._saveAndRaiseCommitishError() # reraises!
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py",
> line 422, in _saveAndRaiseCommitishError
>     t, v, tb = self._saveAndGetCommitishError()
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py",
> line 309, in savepoint
>     savepoint = Savepoint(self, optimistic, *self._resources)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py",
> line 737, in __init__
>     savepoint = savepoint()
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py",
> line 1046, in savepoint
>     self._commit(None)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py",
> line 555, in _commit
>     self._store_objects(ObjectWriter(obj), transaction)
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py",
> line 582, in _store_objects
>     p = writer.serialize(obj)  # This calls __getstate__ of obj
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/serialize.py",
> line 407, in serialize
>     return self._dump(meta, obj.__getstate__())
>   File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/serialize.py",
> line 416, in _dump
>     self._p.dump(state)
>   File "copy_reg.py", line 69, in _reduce_ex
>     raise TypeError, "can't pickle %s objects" % base.__name__
> TypeError: can't pickle instancemethod objects
>
> _______________________________________________
> Zope maillist  -  Zope at zope.org
> http://mail.zope.org/mailman/listinfo/zope**   No cross posts or HTML encoding!  **
> (Related lists - 
> http://mail.zope.org/mailman/listinfo/zope-announcehttp://mail.zope.org/mailman/listinfo/zope-dev )



More information about the Zope mailing list