[ZODB-Dev] large C extension objects / MemoryError

[Jeremy Hylton]

>>>>> "AD" == Andrew Dalke <adalke@mindspring.com> writes:

  AD> I didn't think it did, but I wasn't sure.  How do I make a C
  AD> extension behave nicely?  You say it relies on a containing
  AD> object that is Persistent.  In this case, the container is a
  AD> PersistentMapping.  That doesn't seem sufficient.  What should I
  AD> try next?

I believe BTrees is an example of a C extension type that is also
Persistent.  I not really familiar with the code, so I'm afraid the
best I can do is tell you to look at it.  A quick scan reveals the
following apparently relevant properties of BTrees objects:

    - they're based on ExtensionClass
    - they define the PERSISTENT_TYPE_FLAG
    - they implement _p_deactivate()

The last one is probably important for getting the ZODB cache to be
effective.  A ghost is an uninitialized Python object.  The C struct
for it has been allocated, but all of its data is zero.  An active
object becomes a ghost be calling its _p_deactivate() method.

For this approach to effectively reduce the amount of memory, the C
struct needs to have pointers to other objects that can be released
when its becomes a ghost.  If the memory is all allocated in one block
(like a Python string or tuple), then there's no advantage to becoming
a ghost.

Jeremy