[ZODB-Dev] Unpickler.noload, zc.zodbdgc and multi-database reference bug (IndexError)?

[Jim Fulton]

On Thu, Jan 30, 2014 at 12:40 PM,  <jason.madden at nextthought.com> wrote:
>
> On Jan 30, 2014, at 11:12, jason.madden at nextthought.com wrote:
>
>> So it seems that the behaviour of `noload` might have changed between 2.6.x and 2.7.x?
>
> Apologies for replying to myself, but I think I found the root cause.
>
> After some further investigation, I found issue 1101399 (http://bugs.python.org/issue1101399), complaining that noload is broken for subclasses of dict. The fix for this issue was applied to the cPython trunk in October of 2009 without any corresponding tests (http://hg.python.org/releasing/2.7.6/rev/d0f005e6fadd). In releases with this fix (if I'm reading the code correctly),

Probably because the original code had no tests (because we weren't
doing that then) and
no documentation either (my bad) because this was added for the
specific use case of
efficiently scraping out references in ZODB.

....

>
> This fix means that multi-database references are always going to be returned as an empty list under noload (again, if I'm reading the code correctly). This means that multi-references and noload don't work under Python 2.7.x or 3.x with zodbpickle and so consequently neither does an unmodified zc.zodbdgc.

Sigh.  We're still using Python 2.6 for  out database servers. :)

> I don't know what the best way forward is. Our solution to use `load` instead seems to work for us, but may not work for everyone.

It will probably work, but be slower, which isn't a huge deal since
zc.zodbdgc runs out of
process. We run it on ZRS secondaries, which are otherwise idle.

> Maybe zodbpickle could revert the fix in its branch and zc.zodbgc could depend on that? I'm happy to help test other ideas.

That may be the best way forward.

I can't speak with much confidence until I've had a chance to
wade in and refresh my memory on this stuff.

Something I wish I'd done differently in ZODB and contemplated changing on a
number of occasions is the handling of references. I wish I'd stored
them outside the
pickle so they could be analyzed without unpicking (or at least
without unpickling the
application data).

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton