[ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

[Dieter Maurer]

Jim Fulton wrote at 2006-10-6 16:02 -0400:
>Dieter Maurer wrote:
>> Jim Fulton wrote at 2006-10-6 15:08 -0400:
>>> ...
>>>>> You could implement your sticky attribute at the application level:
>>>>>
>>>>>
>>>>>     def _p_deactivate(self):
>>>>>         if getattr(self, '_p_sticky', False):
>>>>>            return
>>>>>         Persistent._p_deactivate(self)
>>>>>
>>>>> You could provide any policy you want, without making the policy part
>>>>> of ZODB.
>>>> But, such an object would never again be deactivated (unless
>>>> "_p_sticky" is reset). In my proposal, such objects may be
>>>> invalidated at transaction boundaries.
>>>>
>>>> This may not be a big problem, as there are probably not too many
>>>> objects that have "_p_sticky" defined permanently
>>>> ("Shared.DC.ZRDB.Connection" and
>>>> "Products.CMFCore.Skinnable.ObjectManager" would be examples).
>>> Good point.  I left off that you could register a callback on transaction
>>> commit to clear the sticky flag.
>> 
>> Not a nice perspective -- for too reasons:
>> 
>>   *  When would I register the callback?
>
>When you set the volatile data that needed to be sticky.

You miss my "Class level sticky declaration" use case...


Registering when I set the "_v_" variable would probably not be enough
at least not when the registration is valid only for one transaction --
what I expect (but am not sure of):

   The "_p_sticky" would need to be deactivated at any transaction
   boundary as long as the object is in the cache -- independent
   of whether or not I have set any "_v_" variable in the transaction.

> ...
>>      Sure, the callback should not need to look at all objects in
>>      the cache to find the sticky ones.
>> 
>>      This means, the callback would need to be registered for
>>      each object potentially relevant at transaction boundary time.
>
>Yes, any object that needed to be sticky would need to have a callback
>registered.

And this is difficult when the registration is only active for a single
transaction: the object may remain in the cache without me touching
it in the following transactions. Nevertheless, it should be possible
to garbage collect it at transaction boundaries.

If, on the other hand, the registration is active until explicitely canceled,
I explicitely need to cancel it when the object is flushed from
the cache. I would have to add respective code in "_p_deactivate"
(which I have already overridden) and in "_p_invalidate" (which
I would need to override as well).

>>      In principle, all objects in the cache can be relevant...
>
>All objects are sticky?

The "Class level sticky declaration" use case means that sticky
objects can come into existence without any explicit action.

Thus, while not all objects are sticky, any object in the cache could
be sticky.

>>   *  The examples above have their "_p_sticky" defined at
>>      class level -- as they make vital use of volatile attributes
>>      and need a garanteed lifetime for them until transaction boundary.
>
>Hm, so stickyness is a property of the class?  Why store _p_sticky on
>the instance then?

As explained in the proposal:

  We have 3 use cases for volatile attributes:

    1.  cache -- does not need a garanteed lifetime

    2.  uses like "Shared.DC.ZRDB.Connection" and
        "Products.CMFCore.Skinnable.SkinnableObjectManager":
	Require necessarily garanteed lifetime for volatile
	attributes.

	Most secure, if "_p_sticky" is defined on class level.

    3.  Additional control flow information as in the
        "_v_isc[o]p[y]" use by Archetypes.

	The "_p_sticky" is set temporarily on the instance
	to temporarily protect the "_v_" attribute.

	Setting it on class level is not adequate (as a potentially
	huge number of objects would become sticky).

>>      This means we need an additional callback activated after
>>      the cache garbage collection to get rid of the instance
>>      level "_p_sticky" again.
>
>I don't really follow this. I'm not sure I need to.
>
>I have a feeling that there's some deeper issue lurking here.
>I'll have to think about this some more.

Okay, will wait until then...

>Some observations:
>
>- This isn't unique to _v_ attributes.  An object could
>   have other non-persistent data that is precious.

No: only "_v_" attributes are lost at deactivation

(provided the object was informed about changes of its non-persistent data).

>- I wonder if an argument could be made than we shouldn't
>   implicitly deactivate an object that has been accessed in a
>   a transaction while the transaction is still running.

If we made this argument, then my sticky proposal would immediately be
void.

However, we explicitely call cache garbage collection at
savepoint boundaries in order to allow large transactions accessing
huge numbers of objects (such as catalog operations) to
keep their memory requirements small.

Moreover, Chris Withers is keen to see an aggressive cache replacement
policy where objects are deactivated immediately as soon the
configured cache limits are reached -- again in order to keep
memory consumption low even for large transactions.

And, finally, Toby (Dickenson) seems to call "incrgc" often in
his applications to keep transaction size bounded.

Keeping all accessed objects alive during a whole transaction
would go against all these use cases.



-- 
Dieter