[ZODB-Dev] Speeding up ZODB (was "redis cache for RelStorage")

[Shane Hathaway]

On 05/06/2011 10:18 AM, Paul Winkler wrote:
> On Fri, May 06, 2011 at 05:19:25PM +0200, Matthias wrote:
>> It would be cool if you could give a hint to ZEO somehow to prefetch a
>> certain set of objects along with their subobjects and then return
>> everything in one batch. This way you avoid all the round-trips when you
>> discover you want to retrieve a suboject.
>
> I've seen this proposed several times over the years, but AFAIK nobody
> has implemented it yet.
> For example:
> http://www.mail-archive.com/zodb-dev@zope.org/msg04107.html

 From my experience, most people who want ZODB to be faster want Zope 
catalogs in particular to be faster.  I don't think prefetching can make 
catalogs much faster, though.

However, there is a different class of problems that prefetching could 
help solve.  Let's say you have pages with a lot of little pieces on it, 
such as a comment page with a profile image for every comment.  It would 
be useful to tell ZODB to load all of the objects in a list before 
accessing them.  For example:

     app._p_jar.load_all(comments)
     app._p_jar.load_all([comment.profile for comment in comments])

If the storage cooperates, ZODB might implement load_all in an efficient 
way that hits the server only once (or not at all, if the objects are 
already loaded).

Shane