[Zope3-dev] Performance Testing

[Nicola Larosa]

> Tarek has started some very interesting work on adding performance
> testing to the Zope 3 testing infrastructure and it so happens that Jim
> and I were discussing something very similar last week, so I'd like to
> suggest some functionality we might want to have (which I should be able
> to help implement).
> 
> 1) warn about regressions: the test runner will keep per-test,
> machine-independent records of how long tests take and will report
> regressions larger than a predefined percentage.  These records will be
> checked in so that if someone else makes changes (in a fresh checkout)
> that causes a particular test to slow down drastically, they will be
> warned.
> 
> 2) testbrowser should keep up with a (machine-independent) metric of how
> long the previous request took so performance assertions can be made
> inside tests.  E.g.
> 
>     >>> browser.open('http://localhost/foo')
>     >>> browser.last_request_time < 0.5
> 
> 3) the functional testing framework should be extended to allow the
> collection of total time (again, machine-independent) per request and
> the test runner should have an option to display the top "n" slowest
> requests.
> 
> Comments?

I suggest to extend the approach by adding monitoring of the resident
memory occupied by the process, alongside the execution time. Zope 3 is a
rather huge beast, doing all the things it does.

Some quick data: an instance freshly created from the trunk on a Linux
machine a few days ago (before the Twisted integration merge) used 35
Megabytes of resident memory at startup, and 50 MB after a few page views.

After removing all ZCML files from etc/package-includes (apart from two
indispensable ones), it used 22 MB at startup and 38 MB after a few page
views (but the instance was unusable, of course).

This, when a naked Python process uses about 2 MB, and a rather heavy
Twisted process uses about 10-12 MB.

The amount of resident memory used by a process is easily observable on
Linux: it's the second number in /proc/[pid]/statm, multiplied by 4K.

It's true that, in this day and age, RAM is rather cheap; nonetheless, when
deploying on virtual servers, hosting cost is usually proportional to
available memory, so minimization of its usage is convenient. The same is
true when employing multiple Zope 3 processes on the same machine, for
separation, resilience, or other reasons.

-- 
Nicola Larosa - nico at tekNico.net

...I sometimes think it would be nice to be a Tom Cruise or Mel Gibson
but like Rome, it's probably a nice place to visit but I wouldn't want
to live there. -- jgoldblog, May 2002