[Zope] Serve large files efficiently from a pipe

Tino Wildenhain tino at wildenhain.de
Wed Nov 9 03:11:00 EST 2005


Am Dienstag, den 08.11.2005, 22:34 +0200 schrieb Roman Suzi:
> On Tue, 8 Nov 2005, Paul Winkler wrote:
> 
> > On 11/8/05, Roman Suzi <rnd at onego.ru> wrote:
> > > I don't know if writing to response one chunk at a time is proper
> >> solution? Will Zope store response body or sent it right away? I am not
> >> sure that it is the later...
> >
> > The classic way to do streaming data in Zope is like so:
> >
> > RESPONSE.setHeader('Content-length', N)
> > for chunk in some_iterable_yielding_data:
> >    RESPONSE.write(chunk)
> >
> > ... where N is the size in bytes of the data you are going to write.
> 
> OK... Then streaming from Unix pipe can't be done in Zope...

Sure it can. 

> > However, you can also publish any method that returns an
> > implementation of IStreamIterator as shown in
> > http://svn.zope.org/Zope/trunk/lib/python/ZPublisher/Iterators.py?view=markup
> >
> > The advantage is that this has less overhead, as once your method has
> > returned, the Zope app and ZODB are no longer involved, and your
> > iterator and ZPublisher do all the work.
> > This can become quite significant when you are streaming large data,
> > more than a few hundred kB. However, your iterator *must* be able to
> > do its work without a ZODB connection.
> >
> > You haven't said where the data you want to stream is going to come from,
> 
> Sorry if it was not clear: data comes from a pipe (that is, from another
> application, called by popen2...
> 
> > so I can't guess whether the IStreamIterator technique will work for you...
> 
>   - I do not have named file... only file handler. But maybe named pipe
> will do the trick... However, missing content-length (and unfortunately,
> resulting content-length is not known) is the show-stopper.

What makes you think a file is always a "named file"? After all,
what you get from popen2 is just a file. The only hurdle you 
need to take is the __len__ method on this object. So you
might need a little wrapper and/or investigate a bit what 
ZPublisher does with it and how to prevent it (since you dont have
the size, do you?)

Regards
Tino



More information about the Zope mailing list