Performance testing and coverage
Submitted by Ainars Galvans on Mon, 10/03/2008 - 14:46.
performance testing
I’m doing something like performance testing as consultant now. First thing I say to my customer is this: I will not replicate real application usage! I will only emulate simultaneous clicking in a few places of your application: tiny functional coverage; incomplete environment and data; no emulation of unexpected events… It reminds me of a joke about physicists analyzing a spherical horse in a vacuum.
This way I should have lost all my customers, shouldn’t I? Well… I have a little secret I want to share today. I gave a name stakeholder-guided (opposed stakeholder-signed) performance testing. Educating customers is a hard and ungrateful job. It is easier to abuse their belief in tools, requirements and methodologies, but I’m not looking for easy money.
Introduction: Best practices
For years I appose those advocates defining performance requirements as the best practice.
I’ve red some people saying that you must create environment as close as possible, you must run scenarios as realistic as possible. That’s bu**^#&@ and I’m not the only one who thinks so.
I have a few “best practices”. Let me illustrate on an example. Let’s assume that we have requirements for 200 user load and identified 5 different scenarios – each equally likely to be used, so we have to run 40 users of each scenario to emulate load. The first thing I will do is to run each scenario standalone with all 200 users doing this one scenario. Realistic – of course no! But if there are some issues about this specific scenario – this test is more likely to identify it. In functional tests we test boundary values even f they are not realistic – just because they are more likely to identify a bug.
It’s my best practice not because it’s so cool, but because it’s so easy to run! In my blogs you could find more of practices that are easy to do.
What’s the goal of performance testing?
Suppose you have run performance tests and have test results. But how valid they are? Will end-users experience the responsiveness that you have reported? 100% of the time? How about pikes, maintenance, unexpected events, etc.
You are lucky if validity is not your concern. However on my experience stakeholder goals have been one or more of the following: establishing benchmarks; sizing production environment; discovering bottlenecks; and of course - to ensure performance related problems would never appear in production…
Never ever? How about unexpected events: Christmas shopping; summer vacations;… You can’t predict and test everything.
There is always one more bug.. That’s what functional testing industry has realized decades ago. Term risk and risk-based testing is part of functional tester lexicon quite a while. Both functional and non-functional testing is only a way to minimize and evaluate a risk. Not a way to be sure of something.
Stakeholder-Guided Performance Testing
Suppose I’m a great driver used to manual transmission, off-road, etc., etc. But I don’t know the terrain. I have at least two options: get a map or a guide (person who knows terrain pretty well to sit next to me). I believe I would prefer guide. Even if he no driving licence or has no idea on how weather affects driving off-road. That’s even better – he will concentrate on guiding me through the terrain, not commenting driving details.
For years I’ve been trying to develop performance test plan templates and fill the plans with details so that stakeholders could sign them off. It would be like asking for a terrain map instead of a guide. Drawing a map takes time. The more details and the more accurate you what it the more time it takes. Map is static – it does not reflect changes caused by natural and human—interfered terrain transformations. Map all alone is not enough to select the fastest way through the city – if you are used to drive through it you know where the traffic lights will stop you and where jams.
In last years I’ve found it is more effective if a developers or architects knowing software internals guide me through. Of course – I could and would do the most testing decisions myself. I also need them to help finding workarounds and short-paths through the architecture as the problems arise.
Summary: just another buzzword…?
So what I propose is to recognize value of collaboration instead of spending time on documenting performance requirements that could be surrender anyway. I offer my customers interactive and dynamic change between tools, methodologies and activities – based on findings are developer’s input.
I may have called this an agile performance testing because what I’ve just wrote correlates closely with 4 agile principles. However my driver is not the same as it was on wopr 7 were we discussed how the buzzword "Agile” applies to performance testing. I’m only motivated to share my feelings. And I feel that stakeholders are rejecting my approach because they blindly believe in performance tools, requirements and methodology. My experience on the other hand says that those are the least things to trust.
This way I should have lost all my customers, shouldn’t I? Well… I have a little secret I want to share today. I gave a name stakeholder-guided (opposed stakeholder-signed) performance testing. Educating customers is a hard and ungrateful job. It is easier to abuse their belief in tools, requirements and methodologies, but I’m not looking for easy money.
Introduction: Best practices
For years I appose those advocates defining performance requirements as the best practice.
I’ve red some people saying that you must create environment as close as possible, you must run scenarios as realistic as possible. That’s bu**^#&@ and I’m not the only one who thinks so.
I have a few “best practices”. Let me illustrate on an example. Let’s assume that we have requirements for 200 user load and identified 5 different scenarios – each equally likely to be used, so we have to run 40 users of each scenario to emulate load. The first thing I will do is to run each scenario standalone with all 200 users doing this one scenario. Realistic – of course no! But if there are some issues about this specific scenario – this test is more likely to identify it. In functional tests we test boundary values even f they are not realistic – just because they are more likely to identify a bug.
It’s my best practice not because it’s so cool, but because it’s so easy to run! In my blogs you could find more of practices that are easy to do.
What’s the goal of performance testing?
Suppose you have run performance tests and have test results. But how valid they are? Will end-users experience the responsiveness that you have reported? 100% of the time? How about pikes, maintenance, unexpected events, etc.
You are lucky if validity is not your concern. However on my experience stakeholder goals have been one or more of the following: establishing benchmarks; sizing production environment; discovering bottlenecks; and of course - to ensure performance related problems would never appear in production…
Never ever? How about unexpected events: Christmas shopping; summer vacations;… You can’t predict and test everything.
There is always one more bug.. That’s what functional testing industry has realized decades ago. Term risk and risk-based testing is part of functional tester lexicon quite a while. Both functional and non-functional testing is only a way to minimize and evaluate a risk. Not a way to be sure of something.
Stakeholder-Guided Performance Testing
Suppose I’m a great driver used to manual transmission, off-road, etc., etc. But I don’t know the terrain. I have at least two options: get a map or a guide (person who knows terrain pretty well to sit next to me). I believe I would prefer guide. Even if he no driving licence or has no idea on how weather affects driving off-road. That’s even better – he will concentrate on guiding me through the terrain, not commenting driving details.
For years I’ve been trying to develop performance test plan templates and fill the plans with details so that stakeholders could sign them off. It would be like asking for a terrain map instead of a guide. Drawing a map takes time. The more details and the more accurate you what it the more time it takes. Map is static – it does not reflect changes caused by natural and human—interfered terrain transformations. Map all alone is not enough to select the fastest way through the city – if you are used to drive through it you know where the traffic lights will stop you and where jams.
In last years I’ve found it is more effective if a developers or architects knowing software internals guide me through. Of course – I could and would do the most testing decisions myself. I also need them to help finding workarounds and short-paths through the architecture as the problems arise.
Summary: just another buzzword…?
So what I propose is to recognize value of collaboration instead of spending time on documenting performance requirements that could be surrender anyway. I offer my customers interactive and dynamic change between tools, methodologies and activities – based on findings are developer’s input.
I may have called this an agile performance testing because what I’ve just wrote correlates closely with 4 agile principles. However my driver is not the same as it was on wopr 7 were we discussed how the buzzword "Agile” applies to performance testing. I’m only motivated to share my feelings. And I feel that stakeholders are rejecting my approach because they blindly believe in performance tools, requirements and methodology. My experience on the other hand says that those are the least things to trust.