[Zope3-dev] Re: Performance Testing

[Tarek Ziadé]

Benji York wrote:

> Tarek has started some very interesting work on adding performance 
> testing to the Zope 3 testing infrastructure and it so happens that 
> Jim and I were discussing something very similar last week, so I'd 
> like to suggest some functionality we might want to have (which I 
> should be able to help implement).
>
> 1) warn about regressions: the test runner will keep per-test, 
> machine-independent records of how long tests take and will report 
> regressions larger than a predefined percentage.  These records will 
> be checked in so that if someone else makes changes (in a fresh 
> checkout) that causes a particular test to slow down drastically, they 
> will be warned.

-> About machine-independence: Stephan brought the pystone idea to take 
care of it

-> some thaught about percentage

this percentage (let's call it the tolerance), may vary a lot, depending 
on the test complexity.
So i think we can do the performance regression testing in two steps i 
would like to suggest:

1/ a first step would be to run the tester in a special mode to render 
an ordered list for each non-marked test:

    + a measure of the average complexity, including the number of calls
       and when possible the type of complexity (linear, exponential, 
logarithmic, etc...)

    + a measure of pystones

2/ This stats would then be used to mark all hot spots with a "max 
pystones allowed".
    The decorator we implemented last week fits well for this.

>
> 2) testbrowser should keep up with a (machine-independent) metric of 
> how long the previous request took so performance assertions can be 
> made inside tests.  E.g.
>
>     >>> browser.open('http://localhost/foo')
>     >>> browser.last_request_time < 0.5
>
> 3) the functional testing framework should be extended to allow the 
> collection of total time (again, machine-independent) per request and 
> the test runner should have an option to display the top "n" slowest 
> requests.
>
> Comments?

For the same reasons, it would be nice to have the same kind of 
regression test for memory taken by objects:

In the webmail i am coding, if i suddenly change the code and by doing 
this, i double the size of a mail object
in the ZODB, that can be quite bad, as i have thousands of instances of 
them.

So i would like to be able to do the same kind of marker for memory.

Regards,

Tarek