[Zope3-dev] RFC: Make HTTP streaming of large data simpler

[Jim Fulton]

Sidnei da Silva wrote:
> On Mon, Dec 05, 2005 at 01:24:54PM -0500, Paul Winkler wrote:
> | On Mon, Dec 05, 2005 at 11:13:31AM -0500, Jim Fulton wrote:
> | > BTW, your implementation also doesn't work because it doesn't
> | > set the content length.
> | 
> | Speaking of which, I'd love to see a way to make this work for
> | situations where you can't precompute a correct content length because
> | e.g. you're building a lot of data on the fly but you still want to
> | start streaming it as early as possible.
> | In this case, the HTTP spec says that you should NOT set the
> | content-length header but you must close the channel to signal "end of
> | data".  Or at least that's what Dieter says it says :-)
> | http://mail.zope.org/pipermail/zope/2005-November/162714.html
> | 
> | Apparently, according to Dieter, Zope 2 doesn't let you use this idiom;
> | does zope 3?  (I've very briefly looked at the Zope 2 code in medusa,
> | and I can't quite tell if Dieter is correct - there seems to be some
> | support for the "chunked" transfer coding, which should do the job, but
> | I'm not sure if/how one can currently take advantage of it.)
> 
> There are a couple conditions that must be met for 'chunked' to work
> with Zope 2.
> 
> 1. A Content-Length header must not be set.
> 2. The request must be HTTP 1.1
> 3. You must be streaming
> 4. No 'Connection: close' header has been set during the request
> 
> If all of that is met, then it works just fine. The signal for 'end of
> data' in chunked mode is '0\r\n\r\n', which Zope properly inserts when
> appropriate.

Is this documented anywhere?

If not (or perhaps even if so :), and given problems like:

Paul Winkler wrote:
 > On Tue, Dec 06, 2005 at 12:01:37PM -0200, Sidnei da Silva wrote:
 >
 >>On Tue, Dec 06, 2005 at 08:55:56AM -0500, Jim Fulton wrote:
 >>| Interesting.  I wasn't aware of this in Z2.  Zope 3 definately
 >>| doesn't have this.  Sindnei, have you verified that this actially
 >>| works in Z2?  I didn't think the Zope 2 publisher actually started
 >>| propducing output until the request was finished.
 >>
 >>Yes, I can confirm it works like I described. I've been using and
 >>relying on this (where this == chunked response producing output
 >>before the request is finished) on Enfold Desktop for a while.
 >
 >
 > But, as Sidnei pointed out to me offlist, it's not as modular as a first
 > glance at the code would make one hope :-) From an offlist mesage:
 >
 >
 >>Actually, chunked_producer doesn't seem to be used, afaict. Chunking
 >>is inlined in the body of HTTPResponse.write(). That could really get
 >>some loving. I've recently added 'gzip' content-encoding to streamed
 >>content by hacking around the 'ChannelPipe' and it was a real pain to
 >>do gzip+chunked. Updating the compressed producer to add the missing
 >>zlib header + checksum and actually using those instead of hardcoding
 >>it all in ZServer would be very interesting.
 >>
 >>BUT, in the light of Zope 3 moving to twisted, and possibly Zope 2 at
 >>the time of Zope 2.11, maybe that's not worth the effort.

I'd really like us to think about how we *want* this to work in Zope (2 and 3).

I'm uncomfortable with the subtle cues that seem to change the behavior of
request.write from non-streaming to streaming.  I'd like to see
a proposal for a clean API that supports:

- Simple traditional output (ie pages)

- Non-streaming large-file output.

   o Don't hold application thread to do I/O

   o Don't consume a lot of memory

- Streaming output

   o For applications that tale a lot of time to generate
     output.

   o Output is generated over a long period of time.

   o Perhaps extends life of application thread.

     (Maybe there are other alternatives.)

Note that there are some serious issues being discussed
on the Python Web SIG wrt threading and WSGI that have a bearing on
this discussion.  I'm worried that we won't get adequate threading
support from WSGI.  I hope that isn't the case.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org