[ZCM] [ZC] 2306/10 Comment "Pdata objects can not be Pickled once the chain is longer than 500"

Mon Sep 3 10:43:44 EDT 2007

Issue #2306 Update (Comment) "Pdata objects can not be Pickled once the chain is longer than 500"
 Status Pending, Zope/bug medium
To followup, visit:
  http://www.zope.org/Collectors/Zope/2306

==============================================================
= Comment - Entry #10 by tseaver on Sep 3, 2007 10:43 am

I'll note a couple of observations:

 - The chunking into individual 64k Pdata chunks can't work
   when you pass the file to the constructor:  the implementaiton
   relies on having a connection ('_p_jar') already associated
   with the File object, which will never be true ducring the
   constructor.  Therefore, you are likely to end up with a
   *single* Pdata chunk, which will be as large as the entire
   value passed to the constructor.  This is a Bad Thing (tm),
   and is why the 'manage_addFile' factory passes an empty
   string as data to the constructor, and then calls
   'manage_upload' with the real data.

 - Your test is spending a lot of effort getting around
   acquisition wrappers, which can't be pickled properly.
   I *suspect* that the recursion is due to an improperly-
   wrapped object somewhere.
________________________________________
= Comment - Entry #9 by tseaver on Sep 3, 2007 10:38 am

> tseaver:
> You are using Zope 2.10 - the bug is logged against Zope 2.8 and Zope 2.9
>   - maybe it's fixed in 2.10, who knows? Also, you have dumped the data
>   without using the Pickler class - maybe this affects it?

My test script behaves identically in Zope 2.8 and 2.9, and 
if I create a pickler wrapped around 'target' instead of calling
'dump' as a module scope function.

________________________________________
= Comment - Entry #8 by ajung on Sep 3, 2007 6:15 am

The error also occurs with the Python pickle module:

  File "/opt/python-2.4.4/lib/python2.4/pickle.py", line 695, in _batch_setitems
    save(v)
  File "/opt/python-2.4.4/lib/python2.4/pickle.py", line 338, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/python-2.4.4/lib/python2.4/pickle.py", line 433, in save_reduce
    save(state)
  File "/opt/python-2.4.4/

Also interesting is the behavior of _read_file() generating
only 1 chunk when called through the constructor.
________________________________________
= Comment - Entry #7 by ajung on Sep 2, 2007 6:38 pm

If you submit a bugreport including a test then the test should be as small as possible. I *did* look at the test just because I am also a Plone developer _but_ not everybody as a serveral Plone sandboxes at hand - and especially not if all 3rd-party dependencies like TextIndexNG (as used in your tests)...such tests aren't pretty helpful for us.

________________________________________
= Comment - Entry #6 by esrever_otua on Sep 2, 2007 12:17 pm

Uploaded:  "pdatatest.py"
 - http://www.zope.org/Collectors/Zope/2306/pdatatest.py/view
Attached is a single file implementing the tests as ZopeTestCase tests. I trust that you will be able to drop this into the correct folder and run.
________________________________________
= Comment - Entry #5 by esrever_otua on Sep 2, 2007 5:43 pm

Folks, I'm gobsmacked.

Fine, the testcase is a PloneTestCase - I will create a ZopeTestCase when I have a spare moment. Honestly, though, this is a trivial problem to duplicate. Upload a big file TTW, then try to run some code that pickles it.

Have either of you *actually run* the attached testcase? Does it fail? Can you offer some *constructive input* into why it is perfectly acceptable that it should fail?

tseaver:
You are using Zope 2.10 - the bug is logged against Zope 2.8 and Zope 2.9 - maybe it's fixed in 2.10, who knows? Also, you have dumped the data without using the Pickler class - maybe this affects it?
________________________________________
= Comment - Entry #4 by ajung on Sep 2, 2007 1:27 pm

I also have some doubts. We're storing large objects up to 150MB using Pdata chunking within the ZODB...never had any issues so far.

________________________________________
= Comment - Entry #3 by tseaver on Sep 2, 2007 7:19 am

I can't reproduce the described behavior at all, using the Python
API of File::

    $ cd ~/projects/Zope-CVS/Zope-2.10-branch
    $ PYTHONPATH=lib/python ../bin/python
    Python 2.4.4 (#1, Apr 30 2007, 11:13:06) 
    [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> x = 'X' * 1024 * 1024
    >>> data = open('/tmp/issue_2306', 'wb')
    >>> for i in range(34):
    ...     data.write(x)
    ... 
    >>> data.flush()
    >>> data.close()
    >>> data = open('/tmp/issue_2306', 'rb')
    >>> from OFS.Image import File
    >>> f = File('issue_2306', '', '')
    >>> f.manage_upload(data)
    >>> f.size
    35651584L
    >>> from cPickle import dump, load
    >>> target = open('/tmp/pickle', 'wb')
    >>> dump(f, target)
    >>> target.flush()
    >>> target.close()
    >>> source = open('/tmp/pickle', 'rb')
    >>> f2 = load(source)
    >>> f2 == f
    False
    >>> f2.data == f.data
    False
    >>> f2.size
    35651584L
    >>> str(f2.data) == str(f.data)
    True

________________________________________
= Comment - Entry #2 by ajung on Sep 1, 2007 5:05 am

Please submit a *simple* testcase on top of ZopeTestCase but not on top of PloneTestCase with lots of different dependencies from other products.
________________________________________
= Request - Entry #1 by esrever_otua on Apr 5, 2007 6:19 am

Uploaded:  "pdata_failing_testcase.tar.gz"
 - http://www.zope.org/Collectors/Zope/2306/pdata_failing_testcase.tar.gz/view
In OFS.Image there is a 'File' class and 'Pdata' wrapper class. Pdata objects are designed to operate efficiently with large amounts of data by allowing themselves to be chained together in a C linked-list, while presenting an interface as if to a single object at the head of the list.
The 'File' class (which is used for, EG, ATFile, CMFFile, etc) stores its data internally in a Pdata object. It provides a 'manage_upload()' interface for modifying or supplying data to be stored in the File. This function splits the data into 64KB chunks and assigns each chunk to a new Pdata object in the C linked-list Pdata chain.
Once a file more than 32MB has been stored via manage_upload(), the File object can no longer be dumped with the Pickler() class.
EG, cPickle.Pickler.dump(OFS.Image.File) fails when the data stored in OFS.Image.File is greater than 32MB and was stored via the supplied manage_upload() interface.
The error is a RuntimeError for maximum recursion being hit.

Attached is a simple ZopeTestCase that demonstrates the problem. 
==============================================================