[Zope3-dev] RFC: Make HTTP streaming of large data simpler

[Sidnei da Silva]

On Tue, Dec 20, 2005 at 07:33:13AM -0500, Jim Fulton wrote:
| >There are a couple conditions that must be met for 'chunked' to work
| >with Zope 2.
| >
| >1. A Content-Length header must not be set.
| >2. The request must be HTTP 1.1
| >3. You must be streaming
| >4. No 'Connection: close' header has been set during the request
| >
| >If all of that is met, then it works just fine. The signal for 'end of
| >data' in chunked mode is '0\r\n\r\n', which Zope properly inserts when
| >appropriate.
| 
| Is this documented anywhere?

In this email and on the source only it seems :( Could write a doctest
for sure.

| I'd really like us to think about how we *want* this to work in Zope (2 and 
| 3).
| 
| I'm uncomfortable with the subtle cues that seem to change the behavior of
| request.write from non-streaming to streaming.  I'd like to see
| a proposal for a clean API that supports:
| 
| - Simple traditional output (ie pages)
| 
| - Non-streaming large-file output.
| 
|   o Don't hold application thread to do I/O
| 
|   o Don't consume a lot of memory
| 
| - Streaming output
| 
|   o For applications that tale a lot of time to generate
|     output.
| 
|   o Output is generated over a long period of time.
| 
|   o Perhaps extends life of application thread.
| 
|     (Maybe there are other alternatives.)

If I understand correctly the newly introduced one would be the second
bullet? I don't have an idea of how that can be achieved.

FWIW, I could live with returning a generator instead of using
request.write for all applications where I use request.write today.

One thing that I had in mind for long-running streaming output is that
99% of the time it just needs a constant, read-only view of the
database at the time the streaming started so instead of using a
connection from the zodb connection pool, it could open a new,
read-only connection and use that for delivering the content.

| Note that there are some serious issues being discussed
| on the Python Web SIG wrt threading and WSGI that have a bearing on
| this discussion.  I'm worried that we won't get adequate threading
| support from WSGI.  I hope that isn't the case.

Me too.

-- 
Sidnei da Silva
Enfold Systems, LLC.
http://enfoldsystems.com