[Zope3-dev] RFC: Make HTTP streaming of large data simpler

[Jim Fulton]

Sidnei da Silva wrote:
> On Tue, Dec 20, 2005 at 07:33:13AM -0500, Jim Fulton wrote:
> | >There are a couple conditions that must be met for 'chunked' to work
> | >with Zope 2.
> | >
> | >1. A Content-Length header must not be set.
> | >2. The request must be HTTP 1.1
> | >3. You must be streaming
> | >4. No 'Connection: close' header has been set during the request
> | >
> | >If all of that is met, then it works just fine. The signal for 'end of
> | >data' in chunked mode is '0\r\n\r\n', which Zope properly inserts when
> | >appropriate.
> | 
> | Is this documented anywhere?
> 
> In this email and on the source only it seems :( Could write a doctest
> for sure.

That's what I thought. :)

That's good. It means we're not bound. ;)

> | I'd really like us to think about how we *want* this to work in Zope (2 and 
> | 3).
> | 
> | I'm uncomfortable with the subtle cues that seem to change the behavior of
> | request.write from non-streaming to streaming.  I'd like to see
> | a proposal for a clean API that supports:
> | 
> | - Simple traditional output (ie pages)
> | 
> | - Non-streaming large-file output.
> | 
> |   o Don't hold application thread to do I/O
> | 
> |   o Don't consume a lot of memory
> | 
> | - Streaming output
> | 
> |   o For applications that tale a lot of time to generate
> |     output.
> | 
> |   o Output is generated over a long period of time.
> | 
> |   o Perhaps extends life of application thread.
> | 
> |     (Maybe there are other alternatives.)
> 
> If I understand correctly the newly introduced one would be the second
> bullet? I don't have an idea of how that can be achieved.

What newly introduced one?  You mean the IResult business?
The IResult api addresses the first use case, not the second.

> FWIW, I could live with returning a generator instead of using
> request.write for all applications where I use request.write today.

Cool. But that alone doesn't really address the core issue of how the
application thread should be managed. (I need to think about this
some more.)

> One thing that I had in mind for long-running streaming output is that
> 99% of the time it just needs a constant, read-only view of the
> database at the time the streaming started so instead of using a
> connection from the zodb connection pool, it could open a new,
> read-only connection and use that for delivering the content.

I don't see how that helps. Avoiding writes doesn't substantially
(if at all) reduce resource consumption.

Jim
-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org