[Zope3-dev] Synching objects with underlying datastore

[Kapil Thangavelu]

On Aug 20, 2004, at 8:45 AM, Garrett Smith wrote:

> Kapil Thangavelu wrote:
>> On Thu, 2004-08-19 at 13:00, Garrett Smith wrote:
>>> We're using a relational database to store _some_ of the attribute
>>> values for ZODB objects. Since it's possible for the RDBMS to be
>>> updated independently of Zope, I'd want to synchronize loaded ZODB
>>> objects.
>>>
>>> I know about _p_jar.sync() :-)
>>>
>>> Is there a way to enumerate the currently loaded ZODB objects. (By
>>> 'loaded' I mean those objects whose state is already in memory --
>>> maybe 'cached' is what I'm looking for here.) I obviously don't want
>>> to take the brute force approach and call _p_jar.sync() on all
>>> objects.
>>>
>>
>> _p_jar is the datamanager for a collection of persistent objects, ie
>> in a typical setup it is the zodb connection for a number of objects.
>> syncing on the data manager only needs to be called once for all the
>> objects in the collection.
>
> Would a 'collection' in many/most cases be a Folder? Or is 'collection'
> something internal to the ZODB?
>

the collection would be all objects loaded through the datamanager. say 
for a typical simple case, ie a web request with default zope settings, 
all persistent objects involved in the request would be from a single 
datamanager.


>> however, if your mixing attribute storage
>> for objects at an application level rather than at a datamanager level
>> calling sync() on the datamanager will have no effect, as there is no
>> mechanism for the datamanager to process these application level
>> invalidations. it sounds like you need to have some sort of queue and
>> notification management of database changes. if your using postgres or
>> oracle you can set it up to recieve notifications of rdb changes and
>> manually invoke some sort of application invalidation on the nesc
>> objects.
>
> I know when the RDB changes at the (Zope) application level. I can even
> get a list of objects that are effected by the RDB change. I want to
> avoid, however, synching all objects, since only a subset are actually
> cached (in memory). I.e. I don't want to cause an object to be loaded
> (__setstate__ called) if I don't have to.
>

its not clear at what level your gluing together these rdb attributes 
with zodb persistent objects, ie if their stored in volatiles, or are 
persisted along with the rest of the object, or how  object attributes 
map onto rdb values, nor what application level infrastructure you have 
available, or even what rdbms is being used. say for a simple example, 
that these attributes are persisted along with the rest of the object 
and that there is a 1-1 object to row ratio with the row providing the 
rdb object attributes, and your using postgres. then if you had some 
sort of 'object hub' which allowed resolution of zodb objects based on 
their  row peer primary key, you could have an daemon listening to 
postgres async notifications, possibly batching them (depending on 
application sync requirements), and then sending invalidation messages 
to an application method in zope (via xmlrpc or some rest method) in 
the form of a set of rdb primary keys which the application method in 
turn uses to resolve the zodb objects, and update their attribute 
values.

hth,

Kapil Thangavelu <hazmat at objectrealms.net>       Vision Implemented
objectrealms.net <http://www.objectrealms.net>