Please help me to select performance test stories to tell
Submitted by Ainars Galvans on Fri, 10/03/2006 - 14:04.
performance testing
With this post I want to start list of posts regarding performance testing. While I have plenty of ideas but little time to write about them I’ll first outline items that I’m willing to share. I hope to receive some feedback in form of comments so that I know which parts to concentrate on and which I could even skip. Thanks in advance.
So I have the following things to share:
1) Few bad and few not-so-bad stories about performance testing done late on project. With this I want to say that it is a risk with undefined impacts but moderate probability and the probability tend to increase with introduction of “more-layers”, code-reuse and especially code-generation. Also due to architecture approach “simplest sing that could work” .
2) Classify all performance issues I’ve ever seen which triggers those stories to be bad. Hint how to detect those issues. Probably those are performance testing heuristics I will share, but I’m not 100% sure I understand that term :)
3) Show that all of the issues in 2) could be demonstrated by manual testing. This denies the myth that no good performance testing could be done without support of load tools.
4) Show that you can’t still avoid automated load tests, because they are only option to detect stability (memory and other resource leaking) issues.
Well, there are context as well. I've been performance testing in-house as well as providing performance tests as service to telco and government, though during last few years I only manage performance tests among with functional, so I have lost some technical insight, but got big-picture. Though all my experience was never about product that has to compete to other products by means of performance. It means that performance improvements done in code which improves overall performance by 10% or even 20% are not significant, even more if it introduce some functional risk, e.g. regression risk, it is even not worth it. Only if improvement is evaluated in times or it could be any value depending on data size, only those improvements are worth to implement.
So I have the following things to share:
1) Few bad and few not-so-bad stories about performance testing done late on project. With this I want to say that it is a risk with undefined impacts but moderate probability and the probability tend to increase with introduction of “more-layers”, code-reuse and especially code-generation. Also due to architecture approach “simplest sing that could work” .
2) Classify all performance issues I’ve ever seen which triggers those stories to be bad. Hint how to detect those issues. Probably those are performance testing heuristics I will share, but I’m not 100% sure I understand that term :)
3) Show that all of the issues in 2) could be demonstrated by manual testing. This denies the myth that no good performance testing could be done without support of load tools.
4) Show that you can’t still avoid automated load tests, because they are only option to detect stability (memory and other resource leaking) issues.
Well, there are context as well. I've been performance testing in-house as well as providing performance tests as service to telco and government, though during last few years I only manage performance tests among with functional, so I have lost some technical insight, but got big-picture. Though all my experience was never about product that has to compete to other products by means of performance. It means that performance improvements done in code which improves overall performance by 10% or even 20% are not significant, even more if it introduce some functional risk, e.g. regression risk, it is even not worth it. Only if improvement is evaluated in times or it could be any value depending on data size, only those improvements are worth to implement.