[Zope3-dev] Synching objects with underlying datastore

[Kapil Thangavelu]

On Aug 21, 2004, at 10:10 AM, Garrett Smith wrote:

> Kapil Thangavelu wrote:
>> On Aug 20, 2004, at 8:45 AM, Garrett Smith wrote:
>>> I know when the RDB changes at the (Zope) application level. I can
>>> even get a list of objects that are effected by the RDB change. I
>>> want to avoid, however, synching all objects, since only a subset
>>> are actually cached (in memory). I.e. I don't want to cause an
>>> object to be loaded (__setstate__ called) if I don't have to.
>>>
>> its not clear at what level your gluing together these rdb attributes
>> with zodb persistent objects, ie if their stored in volatiles, or are
>> persisted along with the rest of the object, or how  object attributes
>> map onto rdb values, nor what application level infrastructure you
>> have available, or even what rdbms is being used. say for a simple
>> example, that these attributes are persisted along with the rest of
>> the object and that there is a 1-1 object to row ratio with the row
>> providing the rdb object attributes, and your using postgres. then if
>> you had some sort of 'object hub' which allowed resolution of zodb
>> objects based on their  row peer primary key, you could have an
>> daemon listening to postgres async notifications, possibly batching
>> them (depending on application sync requirements), and then sending
>> invalidation messages to an application method in zope (via xmlrpc or
>> some rest method) in the form of a set of rdb primary keys which the
>> application method in turn uses to resolve the zodb objects, and
>> update their attribute values.
>
> I do have a list of the objects that are invalidated, so I think I'm
> pretty close. If my thinking is correct, the update alogrithm is:
>
>   for object in invalidated_objects:
>     object._p_jar.sync()
>

i thought i went through this already in previous emails.. the _p_jar  
is likely the same for
all these objects if their zodb stored, so calling sync on the jar 
multiple times is pointless,
and calling sync will likely have no affect anyways since the zodb 
datamanager has no reason
invalidate the objects if their zodb state hasn't changed and the 
objects is otherwise unmodified.

you probably want to call, object._p_changed = 1 if you want to mark 
the object as needing invalidation.

> My concern is that iterating *all* invalidated objects is unncecessary
> and potentially very expensive.
>

you haven't really offered any additional information so its hard to 
say... ie any response to

>> its not clear at what level your gluing together these rdb attributes
>> with zodb persistent objects, ie if their stored in volatiles, or are
>> persisted along with the rest of the object, or how  object attributes
>> map onto rdb values, nor what application level infrastructure you
>> have available, or even what rdbms is being used.

since its not clear how and where your storing rdb attributes on these 
zodb objects its hard to give any more advice.

> - There'd be no point to calling calling _p_jar.sync() for unloaded
> objects -- they're really not invalidated.
>

> - By reading an object via the iteration, isn't each object's
> __setstate__ method called (if it hasn't already been loaded by the
> ZODB)?
>

yes.

> What I'd like, then, is:
>
>   for some_id in potentially_invalidated_objects:
>     if loaded_by_zodb(some_id):
>        get_object(some_id)._p_jar.sync()
>
> If this isn't possible, perhaps an alternative approach would be
> something like:
>
>   for data_manager in zodb:
>     data_manager.sync()
>
> Btw, thanks again for your input.
>
> Also, I realize this topic is better suited for the zodb list -- if it
> goes much longer, I'll move it over there :-)
>
>  --Garrett
>
>
Kapil Thangavelu <hazmat at objectrealms.net>       Vision Implemented
objectrealms.net <http://www.objectrealms.net>