[Zope3-dev] Re: URGENT RFC: Is anyone using response.write in Zope 3?

[Shane Hathaway]

Jim Fulton wrote:
> You are confusing Zope 2 and Zope 3.  Zope 2's response.write
> does handle large output effciently, Zope 3's did not.

Last time I worked on it, Zope 2 and Zope 3 used the same strategy for 
handling large output.  It worked, I promise. :-)  Yet I hesitate to 
call the push-to-temporary-file strategy efficient, since it makes many 
extra copies of the data and only helps when a lot of conditions are 
met.  A pull strategy will be efficient for a lot more people.

Making code efficient interests me a lot.  When I worked on this stuff 
earlier, I thought of a way to get Zope to serve files and images out of 
the database more efficiently.  I'd like to know whether you think it's 
a good idea; if it is, we can plan to implement it when it's needed.

The idea is this: run a thread whose only responsibility is to stream 
image and file data out of the object database to browsers.  The thread 
will maintain a permanently open database connection.  Other threads 
will send open HTTP connections to this thread, asking it to finish the 
response.  The special thread will get the binary data from the database 
using its own connection, which would play well with the ZODB cache, 
since we'd only ever have one copy of the image/file data in memory at a 
time.  Contrast this with Zope 2's image and file serving, which loads 
image/file data into arbitrary database connections and immediately 
evicts them from the cache.  It seems like having a special connection 
just for serving images/files would be a lot more efficient, especially 
when concurrency is high.

However, this assumes that most images and files come out of the object 
database.  I suspect that current Zope 3 software gets most images and 
files from the filesystem, but that will change as Zope 3 takes on more 
CMS-like tasks.

Shane