[ZODB-Dev] Long upload times - "ZPublisher.Client" weakness (was: Re: Metadata in zope / binary data on FS,...)

[Dieter Maurer]

Dieter Maurer wrote at 2003-7-19 13:48 +0200:
 >  > ... uploading large files ...
 > 
 >   When I analysed the problem I recognized that the file
 >   was uploaded but it took 11 minutes. During this time
 >   the processors IO wait was between 50 and 90 percent.
 >   The Zope worker process started to see the request only
 >   after about 9 minutes.
 > ...
 > 
 > I expect that I will have the chance to come back to this
 > problem on a modern system in a few weeks.

I analysed the behaviour on an Athlon 1.4 GH (Linux 2)
with 670 MB RAM.

The observed behaviour was probably caused by "ZPublisher.Client":

  "ZPublisher.Client" requires about 4 times the amount of main memory
  than the file size.

Uploading a 100 MB file took 5 s to reach the Zope worker thread and
36 s for Zope to process and store into the ZODB (thus, the complete upload
took 41 s).
"ZPublisher.Client" needed about 400 MB main memory and Zope about
30 MB. With respect to Zope, that is not very bad behaviour.


Uploading a 200 MB file caused "ZPublisher.Client" to eat about 700 MB
virtual memory. Then, swap space was exceeded and Linux killed this
process.


Probably, the 11 min wait time resulted from excessive swapping
activitiy caused by "ZPublisher.Client"'s using an excessive amount
of virtual memory. However, the Solaris "top" did not indicate
excessive swapping, just excessive IO waits. I do not like
Solaris...


Maybe, I will enhance "ZPublisher.Client" to stream the HTTP requests
rather than build them completely in memory.


Dieter