[ZODB-Dev] Missing loader for multidatabase refs?

[Sidnei da Silva]

On 3/30/07, Jim Fulton <jim at zope.com> wrote:
> > What can be done to avoid that?
>
> By "that" I assume you mean dangling referenced.

I really meant avoiding the PosKeyError. ;)

> You could create extra references in the source database.  For
> example, if you make a cross-database reference, you could also add a
> corresponding reference in the source database.

Right. And removing this corresponding difference when the object in
the target database goes away.

> > Not packing the databases at all?
>
> That will work too. :)

Not an option!

> > Would it be possible to make the loading of a cross-database reference
> > that is gone return some sort of 'BrokenObject' instead of a
> > PosKeyError, to work around the problem temporarily?
>
> Sure.  I suppose the same could and perhaps should be done whenever
> you would get a POSKeyError.

That sounds like a candidate for a ZODB proposal.

> > Specially if the databases happen to be on different ZEO servers
> > *wink*.
>
> That is a complication, yes.  You would need some sort of distributed
> GC protocol.

Ouch. :)

> >> - A non-GC pack that got rid of old records but didn't bother with
> >> GC.
> >>    This would be advantagious for lots of folks independent of cross-
> >> database reference issues.
> >
> > Sounds like this would be the easiest way to solve the above issue?
>
> It is a fairly straightforward way, depending on the application.

The application is named Zope 2. You might have heard about it. :)

> Unfortunately, it's probably non-trivial, as the FileStorage packing
> code is fairly intense. :)
>
> I've been wanting to redo this code for some time to reduce the
> amount of disk I/O.  That would certainly be an opportunity to make
> GC optional. But maybe it wouldn't be too hard to disable GC in the
> current implementation -- I don't know.

Let's see. GC right now is done when the object has 0 references
pointing to it. A quick workaround would be making sure the code that
computes the references (referencesf?) always returns at least one
reference pointing to it?

> >> Cross database references are pretty transparent and automatic.
> >>
> >> Maybe there should be an option to make them less so.
> >
> > Yes, such an option would be great.
>
> Maybe.  It depends on what semantics you want.  In fact, I don't know
> how Zope 2 is setting up multi-databases.  If you don't want cross-
> database references at all, you could change the startup code to open
> separate databases without making them part of a multi-database.
>
> Hm, brainstorming:
>
> I suppose there could be could be an option that says you can't have
> a cross reference unless the source database has a root key equal to
> the object id of the source object with the value being the source
> object.  Then, an intentional cross reference would be made like this::
>
>    # Make a reference to foo, which is in another DB:
>    foo._p_jar.root()[foo._p_oid] = foo
>    self.x = foo
>
> The code that creates cross-database references would, if this option
> is available, do a check something like:
>
>   if self.explicit_cross_database_references:
>        if other._p_jar.root().get(other._p_oid) is not other:
>            raise InvalidCrosDatabaseReference(...)

That looks good. If not for solving the problem completely, at least
for catching where in Zope implicit references might be created, and
possibly fixing those places.

> I suspect you know what the patch is at this point -- at least to get
> the pack done.  Perhaps you can try it and let us know how it goes.

Packing worked fine, and packing the other database didn't generate a
PosKeyError... yet. That makes me really nervous though. I will try to
come up with a patch that creates a BrokenObject where a PosKeyError
would occur in the case of multi databases.

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214