[ZODB-Dev] Missing loader for multidatabase refs?

[Jim Fulton]

On Mar 30, 2007, at 10:10 AM, Sidnei da Silva wrote:

> On 3/30/07, Jim Fulton <jim at zope.com> wrote:
>> > Say there's two databases, and the
>> > object in database '1' is the only one referencing some object in
>> > database '2'. Since those references would be treated like weak
>> > references, wouldn't the object in database '2' go away in a  
>> 'pack',
>> > thus causing a PosKeyError when loading the object from database  
>> '1'
>> > that still points to it?
>>
>> Yes. That is a limitation of cross-database references.
>
> What can be done to avoid that?

By "that" I assume you mean dangling referenced.

You could create extra references in the source database.  For  
example, if you make a cross-database reference, you could also add a  
corresponding reference in the source database.

> Not packing the databases at all?

That will work too. :)

> Would it be possible to make the loading of a cross-database reference
> that is gone return some sort of 'BrokenObject' instead of a
> PosKeyError, to work around the problem temporarily?

Sure.  I suppose the same could and perhaps should be done whenever  
you would get a POSKeyError.

>> Note that treating the references differently in a pack can't help
>> this as the references are in the referencing database, not the
>> referenced database.
>>
>> It would be cool to have:
>>
>> - A multi-database pack that took multiple databases into account.
>>    Such a feature is doable, but obviously, non-trivial.
>
> Specially if the databases happen to be on different ZEO servers  
> *wink*.

That is a complication, yes.  You would need some sort of distributed  
GC protocol.

>
>> - A non-GC pack that got rid of old records but didn't bother with  
>> GC.
>>    This would be advantagious for lots of folks independent of cross-
>> database reference issues.
>
> Sounds like this would be the easiest way to solve the above issue?

It is a fairly straightforward way, depending on the application.   
Unfortunately, it's probably non-trivial, as the FileStorage packing  
code is fairly intense. :)

I've been wanting to redo this code for some time to reduce the  
amount of disk I/O.  That would certainly be an opportunity to make  
GC optional. But maybe it wouldn't be too hard to disable GC in the  
current implementation -- I don't know.


>> > Right. I'm wondering how I did end up with a cross-database  
>> reference.
>> > Seems like a copy/paste through the ZMI caused it. My question  
>> is if
>> > that's expected/correct.
>>
>> Cross database references are pretty transparent and automatic.
>>
>> Maybe there should be an option to make them less so.
>
> Yes, such an option would be great.

Maybe.  It depends on what semantics you want.  In fact, I don't know  
how Zope 2 is setting up multi-databases.  If you don't want cross- 
database references at all, you could change the startup code to open  
separate databases without making them part of a multi-database.

Hm, brainstorming:

I suppose there could be could be an option that says you can't have  
a cross reference unless the source database has a root key equal to  
the object id of the source object with the value being the source  
object.  Then, an intentional cross reference would be made like this::

   # Make a reference to foo, which is in another DB:
   foo._p_jar.root()[foo._p_oid] = foo
   self.x = foo

The code that creates cross-database references would, if this option  
is available, do a check something like:

  if self.explicit_cross_database_references:
       if other._p_jar.root().get(other._p_oid) is not other:
           raise InvalidCrosDatabaseReference(...)

>> >> What version of ZODB are you using?
>> >
>> > The one included with Zope 2.9.6.
>>
>> Hm, are you using ZEO?
>>
>> I'm wondering what sort of release would be needed to help you out.
>
> Yes, I'm using ZEO. I don't need any sort of release particularly, a
> patch would be fine. As long as there's a Zope 2.9.x release sometime
> in the future for the people that can't live with patches, it should
> be ok.

I suspect you know what the patch is at this point -- at least to get  
the pack done.  Perhaps you can try it and let us know how it goes.

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org