[Zope-dev] Re: Proposal for optimized Blob handling

[Chris McDonough]

Note that one micro-optimization for PUT requests is to not use a  
FieldStorage at all because the body is never mime-encoded anyway in  
practice.

I have a monkey patch to do this now, which I turned into a patch for  
the core, but took out because Phillipp whined at a sprint once.  ;-)

Here's the monkey patch...

def patch_httprequest_processinputs():
     """ Patch HTTPRequest.processInputs to not do any processing on a
     PUT request (it's pointless, and foils our on-the-fly encryption,
     as it creates a new tempfile via FieldStorage). """

     # note that OTF encryption support only works for PUT requests

     import re
     from ZPublisher.HTTPRequest import HTTPRequest
     oldProcessInputs = HTTPRequest.processInputs
     def newProcessInputs(
         self,
         # "static" variables that we want to be local for speed
         SEQUENCE=1,
         DEFAULT=2,
         RECORD=4,
         RECORDS=8,
         REC=12, # RECORD|RECORDS
         EMPTY=16,
         CONVERTED=32,
         hasattr=hasattr,
         getattr=getattr,
         setattr=setattr,
         search_type=re.compile('(:[a-zA-Z][-a-zA-Z0-9_]+|\\.[xy]) 
$').search,
         ):
         """Process request inputs

         We need to delay input parsing so that it is done under
         publisher control for error handling purposes.
         """
         method=self.environ.get('REQUEST_METHOD','GET')

         if method == 'PUT':
             # we don't need to do any real input processing if we  
are handling
             # a PUT request.  This is an optimization especially  
because
             # FieldStorage creates an additional tempfile if we  
allow it to
             # parse the body, and PUT uploads can tend to be large.
             self._file = self.stdin
             return

         return oldProcessInputs(self)
     HTTPRequest.processInputs = newProcessInputs




- C


On Mar 7, 2007, at 9:57 PM, Philipp von Weitershausen wrote:

> Christian Theune wrote:
>> Hi,
>> [modified slightly from a similar proposal to zope3-dev to match Zope
>> 2's publisher]
>> I'm writing up a proposal for the ZODB to make even more efficient  
>> Blob
>> handling possible.
>> This includes not copying the data from an uploaded file, but using a
>> `link` operation when possible.
>
> I think this is a great idea.
>
> Am I the only person here who immediately associated "link" with  
> the POSIX? Also, am I the only one who read "when possible" as  
> "when on a POSIX system where link is available", in other words,  
> "when not on Windows"? One starts to wonder...
>
>> However, the HTTPRequest class currently uses the default  
>> implementation
>> of the cgi module's FieldStorage.
>> I propose to create a small subclass to override the `make_file`  
>> method
>> to use `NamedTemporaryFile` instead of `TemporaryFile` to allow  
>> the file
>> being accessible from a filename so I can apply a `link` operation.
>
> +1
>
>
> -- 
> http://worldcookery.com -- Professional Zope documentation and  
> training
> _______________________________________________
> Zope-Dev maillist  -  Zope-Dev at zope.org
> http://mail.zope.org/mailman/listinfo/zope-dev
> **  No cross posts or HTML encoding!  **
> (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
> http://mail.zope.org/mailman/listinfo/zope )
>