[Zope3-dev] RFC: Make HTTP streaming of large data simpler

[Paul Winkler]

On Tue, Dec 20, 2005 at 10:48:24AM -0200, Sidnei da Silva wrote:
> On Tue, Dec 20, 2005 at 07:33:13AM -0500, Jim Fulton wrote:
> | I'd really like us to think about how we *want* this to work in Zope (2 and 
> | 3).
> | 
> | I'm uncomfortable with the subtle cues that seem to change the behavior of
> | request.write from non-streaming to streaming. 

Agreed. I had no idea about those "cues" until Sidnei pointed it out.

> | I'd like to see
> | a proposal for a clean API that supports:
> | 
> | - Simple traditional output (ie pages)
> | 
> | - Non-streaming large-file output.
> | 
> |   o Don't hold application thread to do I/O
> | 
> |   o Don't consume a lot of memory

> | - Streaming output
> | 
> |   o For applications that tale a lot of time to generate
> |     output.
> | 
> |   o Output is generated over a long period of time.
> | 
> |   o Perhaps extends life of application thread.
> | 
> |     (Maybe there are other alternatives.)

In zope 2 you can also do RESPONSE.setBody(foo), where foo 
provides ZPublisher.Iterators.IStreamIterator.
Works well for what you're calling "non-streaming large-file
output".  

However, it doesn't handle the "streaming output" case:
for two reasons:  1) there's a big "Don't Do That" note in the 
docstring about not using the database in your iterator code;
2) it mandates a precomputed content-length, which could
be fixed pretty easily: http://www.zope.org/Collectors/Zope/1962

> One thing that I had in mind for long-running streaming output is that
> 99% of the time it just needs a constant, read-only view of the
> database at the time the streaming started so instead of using a
> connection from the zodb connection pool, it could open a new,
> read-only connection and use that for delivering the content.

I agree that read-only is the more likely scenario.

However, the 1% may be more than 1%.  Suppose I'm writing a UI for
maintaining a gigantic site.  Some actions might be inherently slow and
inherently write to the ZODB - e.g. rebuilding a massive catalog.
I like to give feedback to the user early and often, if possible.

Another case that affects me today: using ZSyncer in pull mode.  In
current ZSyncer cvs, i use response.write() with http 1.1 chunking as
Sidnei described, and it works nicely for giving feedback as each item is
fetched or pushed.  Some of these operations involve writes,
and none of them have a known output size, so I can't currently
use an IStreamIterator.

-- 

Paul Winkler
http://www.slinkp.com