[ZODB-Dev] Does ZODB pipeline load requests?

[Dylan Jay]

On 20 Feb 2014, at 2:27 am, Jim Fulton <jim at zope.com> wrote:

> On Wed, Feb 19, 2014 at 9:57 AM, Dylan Jay <djay at pretaweb.com> wrote:
>> On 19 Feb 2014, at 10:44 pm, Jim Fulton <jim at zope.com> wrote:
> ...
>> yeah I figured it might be the case thats its hard to predict. In this case it's catalog indexing so I was wondering if something could be done with __iter__ on a btree? It's a reasonably good guess that you could start preloading more of those objects if the first few are loaded?
> 
> Iterators certainly seem like a logical place to start.

As an example I originally was doing a TTW zope reindex of a single index.
Due to conflict problems I used a modified version of this https://github.com/plone/Products.PloneOrg/blob/master/scripts/catalog_rebuild.py (which I'd love to integrate something similar into zcatalog sometime).
Both use iterators I believe.
I think even if there was an explicit api where you can pass in an iterator, a max buffer length and you'd get passed back another iterator. Then asynchronously objects will load to try and keep ahead of the iterator consumption. 
e.g.
for obj in async_load(myitr, 50):
   dox(obj)

I don't know how that would help with a loop like this however

for obj in async_load(myitr, 50):
   dox(obj.getMainObject())





> 
> Jim
> 
> -- 
> Jim Fulton
> http://www.linkedin.com/in/jimfulton