[Zope3-dev] Re: URGENT RFC: Is anyone using response.write in Zope 3?

[Jim Fulton]

Shane Hathaway wrote:
> Jim Fulton wrote:
> 
>> I could certainly find evidence that you tried, but the implementation 
>> was
>> actually buffering data in a string buffer until the request was 
>> finished.
>> This was the case at least as early as spring of 2004.
> 
> 
> Even with more than 1050000 bytes output over a slow connection?  That's 
> the default threshold; see 
> zope.server.adjustments.Adjustments.outbuf_overflow.  It certainly 
> worked at one time.  In any case, it appears to be unnecessary now.

Christian Theune and I looked at this in early 2004 and came to the conclusion
that data were being stored in memory. I think we used the debugger to determine this.
We weren't using a slow connection.  A casual scan of the code suggests the
data *should* be stored in a temporary file, which made me doubt my assertions. :)
I just verified though with the following view on X3.0:

class Big:

     def __init__(self, context, request):
         self.request = request

     def big(self):
         response = self.request.response
         response.setHeader('content-type', 'text/plain')
         response.setHeader('content-length', str(51*20*1000*100))
         for i in range(20*1000*100):
             response.write('x'*50+'\n')

That outputting lots of data causes memory to baloon.
In this case, on my machine, the virtual memory jumped
about 150 megs and the in-use memory jumped about 100 megs.

<shrug>

I then tried the following view with the trunk, whis is about the same
as 3.2:

import tempfile

class Big:

     def __init__(self, context, request):
         self.request = request

     def big(self):
         response = self.request.response
         response.setHeader('content-type', 'text/plain')
         f = tempfile.TemporaryFile()
         for i in range(20*1000*100):
             f.write('x'*50+'\n')
         return f

Here the memory temporarily jumped about 7 megs and the output
came back about an order of magnitude faster.

I tried this with both the Twisted and ZServer servers and
got about the same result (although I got an error in the log
for ZServer that I need to look into).

Jim


-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org