[ZODB-Dev] Does ZODB pipeline load requests?

[Jim Fulton]

On Wed, Feb 19, 2014 at 7:40 PM, Dylan Jay <djay at pretaweb.com> wrote:
>> Iterators certainly seem like a logical place to start.
>
> As an example I originally was doing a TTW zope reindex of a single index.
> Due to conflict problems I used a modified version of this https://github.com/plone/Products.PloneOrg/blob/master/scripts/catalog_rebuild.py (which I'd love to integrate something similar into zcatalog sometime).
> Both use iterators I believe.

> I think even if there was an explicit api where you can pass in an iterator, a max buffer length and you'd get passed back another iterator. Then asynchronously objects will load to try and keep ahead of the iterator consumption.
> e.g.
> for obj in async_load(myitr, 50):
>    dox(obj)

I like the idea of a wrapper. I think a) you're pushing the abstraction
to far, and b) this doesn't have to be a ZODB API, at least not initially.

In any case, if the lower-level API exists, it would be straightforward
to implement one like above.

> I don't know how that would help with a loop like this however
>
> for obj in async_load(myitr, 50):
>    dox(obj.getMainObject())

Well, this would simply be another custom iterator wrapper.

  for ob in main_iterator(myiter):
      ...

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton