[Zope3-dev] Re: URGENT RFC: Is anyone using response.write in Zope 3?

[Jim Fulton]

Shane Hathaway wrote:
> Jim Fulton wrote:
> 
>> You are confusing Zope 2 and Zope 3.  Zope 2's response.write
>> does handle large output effciently, Zope 3's did not.
> 
> 
> Last time I worked on it, Zope 2 and Zope 3 used the same strategy for 
> handling large output.  It worked, I promise. :-)

I believe you.  Uh, did you test it?


 >  Yet I hesitate to
> call the push-to-temporary-file strategy efficient, since it makes many 
> extra copies of the data and only helps when a lot of conditions are 
> met. 

Many extra copies?  Seems it makes just one.

I'll also note that making file copies is reasonably fast.

 > A pull strategy will be efficient for a lot more people.

I don't know what you mean by this.

> Making code efficient interests me a lot. 

Me too.

 > When I worked on this stuff
> earlier, I thought of a way to get Zope to serve files and images out of 
> the database more efficiently.  I'd like to know whether you think it's 
> a good idea; if it is, we can plan to implement it when it's needed.
> 
> The idea is this: run a thread whose only responsibility is to stream 
> image and file data out of the object database to browsers.  The thread 
> will maintain a permanently open database connection.  Other threads 
> will send open HTTP connections to this thread, asking it to finish the 
> response.  The special thread will get the binary data from the database 
> using its own connection, which would play well with the ZODB cache, 
> since we'd only ever have one copy of the image/file data in memory at a 
> time.  Contrast this with Zope 2's image and file serving, which loads 
> image/file data into arbitrary database connections and immediately 
> evicts them from the cache.  It seems like having a special connection 
> just for serving images/files would be a lot more efficient, especially 
> when concurrency is high.

This sounds too complicated to me.  Have you followed the ZODB blob work?
I think this will provide great benefits for people who choose to store large
(megabytes, not gigabytes or terabytes) files in the database.  I think the
people working on it could use some help finishing it, if you'd like to help.

Blobs are managed as temporary files on a ZEO client, so that they
can be served by the publisher without making copies (except, of course,
for the copy necessary to get them to the client).  They are also managed
outside the normal object and ZEO caches.

> However, this assumes that most images and files come out of the object 
> database.  I suspect that current Zope 3 software gets most images and 
> files from the filesystem, but that will change as Zope 3 takes on more 
> CMS-like tasks.

To the degree that Zope applications manage data as files, the new publisher
strategy will server them very well.  Of course, I hope that blobs will
make ZODB storage a resonable option in the future, as it provides a number
of advantages, such as simplicity, transactions, and ZEO distribution.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org