[ZODB-Dev] Unpickler.noload, zc.zodbdgc and multi-database reference bug (IndexError)?

[jason.madden at nextthought.com]

On Jan 30, 2014, at 10:14, Jim Fulton <jim at zope.com> wrote:

> On Thu, Jan 30, 2014 at 9:58 AM,  <jason.madden at nextthought.com> wrote:

>> I was curious if anyone else had seen this
> 
> I haven't. I'm the author of zodbdgc and I use it regularly, including on
> large (for some definition) databases.

Hi Jim,

I really appreciate your input on this. Thanks for taking the time to respond.

>> Details:
>> 
>> I'm working under Python 2.7.6 and 2.7.3 with ZODB 4.0.0, zc.zodbdgc 0.6.1 and eventually zodbpickle 0.5.2. Most of my results were repeated on both Mac OS X and Linux.
> 
> Why are you using zodbpickle?  Perhaps that is behaving differently
> from cPickle in some fashion?

I didn't start out with zodbpickle, I just tried it as an alternative to cPickle after cPickle was failing. The hope was that they would actually behave differently in some fashion, but the observed behaviour is identical.


> I'd try using ZODB 3.10.  I suspect a ZODB 4 incompatibility of some sort.
> 
> Unfortunately, I don't have time to dig into this now.
> 
> This weekend, I'll at least see if I can make zodbdgc tests pass with ZODB 4.
> Perhaps that will shed light.

That's a very good point. I'm not sure what it will take to repeat the test with ZODB 3.10, but I can try (although, since the pickles are already written at this point, I probably wouldn't expect to see a difference? Nothing mutates the actual pickle data loaded from a storage, excluding transformers like zc.zlibstorage). 

Along those lines, I tried just downloading zc.zodbdgc and running its tests against ZODB 3.10 (under both Python 2.7.6/OS X and 2.7.3/Linux), and unfortunately, they immediately fail with the exact same IndexError I see when trying to GC our database:

    $ mkvirtualenv ZODB3
    $ workon ZODB3
    $ pip install ZODB3==3.10.5
    ....
    $ wget https://pypi.python.org/packages/source/z/zc.zodbdgc/zc.zodbdgc-0.6.1.tar.gz
    ...
    $ tar -xf zc.zodbdgc-0.6.1.tar.gz && cd zc.zodbdgc-0.6.1
    $ python ./setup.py test
    ...
    ======================================================================
    FAIL: zc.zodbdgc-0.6.1/src/zc/zodbdgc/README.test
    Doctest: README.test
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 2201, in runTest
        raise self.failureException(self.format_failure(new.getvalue()))
    AssertionError: Failed doctest test for README.test
      File "tmp/zc.zodbdgc-0.6.1/src/zc/zodbdgc/README.test", line 0

    ----------------------------------------------------------------------
    File "zc.zodbdgc-0.6.1/src/zc/zodbdgc/README.test", line 194, in README.test
    Failed example:
        bad = zc.zodbdgc.gc('config', days=2)
    Exception raised:
    Traceback (most recent call last):
      File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1289, in __run
        compileflags, 1) in test.globs
      File "<doctest README.test[86]>", line 1, in <module>
        bad = zc.zodbdgc.gc('config', days=2)
      File "zc.zodbdgc-0.6.1/src/zc/zodbdgc/__init__.py", line 105, in gc
        return gc_(close, conf, days, ignore, conf2, fs, untransform, ptid)
      File "zc.zodbdgc-0.6.1/src/zc/zodbdgc/__init__.py", line 209, in gc_
        for ref in getrefs(data, name, ignore):
      File "zc.zodbdgc-0.6.1/src/zc/zodbdgc/__init__.py", line 287, in getrefs
        ref = ref[1]
    IndexError: list index out of range

The tests failed the same way under ZODB 4.0.0 and ZODB3 3.9.0 (the minimum requirement listed in setup.py).

However, when I tried Python 2.6.8, the tests pass with ZODB 3.10.5 (they don't pass with ZODB 4, but it looks to be unrelated). So it seems that the behaviour of `noload` might have changed between 2.6.x and 2.7.x? Is it possible you're running on Python < 2.7?

Thanks,

Jason