[ZODB-Dev] ZEO and replication of BTree based objects

[Jeremy Hylton]

On Thu, 2003-06-05 at 13:53, Shane Hathaway wrote:
> Jeremy Hylton wrote:
> > I think it would be great to have a prefetching mechanism in ZEO,
> > probably coupled with a notification system to let the app know when an
> > object is loaded.
> 
> It seems like prefetching is actually easy to do.  You just create a 
> thread in the ZEO client that draws from some kind of OID queue, calling 
> storage.load() for each OID.  Each successful load() adds to the local 
> ZEO cache and possibly notifies the application.  This wouldn't require 
> any changes to ZODB.

It's a bit more complicated because ZEO only allows one synchronous call
at a time, which limits the benefits of pipelining.  I think it would be
better to have an async load message that gets produces an async
response.  Otherwise, as you have proposed.

> What isn't clear is how you fill the OID queue.  How can you know the 
> OIDs of the objects that are important enough to prefetch?

I agree that this is the harder problem.  The best bet is probably
heuristics based on references within an object.  If you access object
A, and it has references to B and C, it increases the likelihood that
you'll access B or C.

BTW, I blogged a little about this subject in April:
http://www.python.org/~jeremy/weblog/030418.html

Jeremy