Performance myths provoked by performance tools' advertisement/docs
Submitted by Ainars Galvans on Mon, 13/03/2006 - 14:10.
non-functional testing | performance testing
I’m going to list my experiences with performance testing and management as answer to Neill and Alexander and many more on QAFORUMS
Critique the article by Robert .
I love search weaknesses in others’ argues. So I do this time. It can’t be that Robert and his article have no background at all... This will not be a single blog of “you are wrong” or “he is wrong”. This will be a list of different posts . But first of all I want to create some context – list my understanding of performance and non-functional test terminology. Afterwards you will be able to see why I called this post “myths provoked by performance tools advertisement/docs “
However I’m not going to provide my term list, instead I will simply list different goals for doing non-functional testing (at least all that I have ever experienced to be valuable):
1) Find (to fix) software performance issues – localized functioanlity that could be performance-improved dramatically by localized change in code (or database structure, indexing or something). Two sub-cases
1.1) Observed in single-thread (typically observed even in functional testing if data size is adequate)
1.2) Observed only for simultaneous execution (typically when two threads try to update the same data item)
2) Find (to fix) resource usage (leaking) issues. I mean any resource not only memory. Those are typically addressed by running some automated load tests for 24; 48; 72 or more hours.
3) Find load scalability issues – the solution inability to utilize hardware as it is increased. Typically absence of/inadequate load balancing, absence of support of asynchronous request processing, etc. Specific for this one is that fix is almost never localized, but rather change or architecture, or even change tools used for solution implementation. More over static testing (reviews) suites even better than dynamic (code execution)
4) Do hardware (and probably other resource) planning (how much do we need) /analysing (is it enough what we have)
5) See how the application will survive stress situation such as pike load (overload for a some time period)
6) Compare performance with another release (performance regression tests or analysing improvements
7) Tune 3rd party software used (thread, pool, memory sizes; etc.)
8) Tune DB indexes, etc. Also detect such DB specific issues as MS SQL page locks causing your application issues with scalability
9) Probably more that I have never experienced. You are welcome to extend the list
Let me now pick up the first learning point here. Typical performance test tool is great at (and it is impossible to do without it) the item 4 (and partially 5). That’s why their advertisements and even documentation is aimed to describe mostly this use case. Unfortunately test engineers and especially performance test engineers are learning to do performance testing exactly from this source. This is I believe source for the article criticised. I hope those tools representatives are not going to kill myself but I’m really going to tell you how all the goals listed above except 4 could be addressed without those tools and that addressing them without tolls even improve the results and effectiveness of doing them (mostly because you don’t suppose tool will do for you everything – you have to use your brains instead).
Critique the article by Robert .
I love search weaknesses in others’ argues. So I do this time. It can’t be that Robert and his article have no background at all... This will not be a single blog of “you are wrong” or “he is wrong”. This will be a list of different posts . But first of all I want to create some context – list my understanding of performance and non-functional test terminology. Afterwards you will be able to see why I called this post “myths provoked by performance tools advertisement/docs “
However I’m not going to provide my term list, instead I will simply list different goals for doing non-functional testing (at least all that I have ever experienced to be valuable):
1) Find (to fix) software performance issues – localized functioanlity that could be performance-improved dramatically by localized change in code (or database structure, indexing or something). Two sub-cases
1.1) Observed in single-thread (typically observed even in functional testing if data size is adequate)
1.2) Observed only for simultaneous execution (typically when two threads try to update the same data item)
2) Find (to fix) resource usage (leaking) issues. I mean any resource not only memory. Those are typically addressed by running some automated load tests for 24; 48; 72 or more hours.
3) Find load scalability issues – the solution inability to utilize hardware as it is increased. Typically absence of/inadequate load balancing, absence of support of asynchronous request processing, etc. Specific for this one is that fix is almost never localized, but rather change or architecture, or even change tools used for solution implementation. More over static testing (reviews) suites even better than dynamic (code execution)
4) Do hardware (and probably other resource) planning (how much do we need) /analysing (is it enough what we have)
5) See how the application will survive stress situation such as pike load (overload for a some time period)
6) Compare performance with another release (performance regression tests or analysing improvements
7) Tune 3rd party software used (thread, pool, memory sizes; etc.)
8) Tune DB indexes, etc. Also detect such DB specific issues as MS SQL page locks causing your application issues with scalability
9) Probably more that I have never experienced. You are welcome to extend the list
Let me now pick up the first learning point here. Typical performance test tool is great at (and it is impossible to do without it) the item 4 (and partially 5). That’s why their advertisements and even documentation is aimed to describe mostly this use case. Unfortunately test engineers and especially performance test engineers are learning to do performance testing exactly from this source. This is I believe source for the article criticised. I hope those tools representatives are not going to kill myself but I’m really going to tell you how all the goals listed above except 4 could be addressed without those tools and that addressing them without tolls even improve the results and effectiveness of doing them (mostly because you don’t suppose tool will do for you everything – you have to use your brains instead).