Once again about the difference
Submitted by Alexander Podelko on Wed, 18/10/2006 - 04:03.
functional testing | non-functional testing | performance testing | performance testing patterns
Listening to Denise Arruda's presentation at CT CMG I found another point of view for my thoughts about performance testing. Simplified performance testing process flow looks like
1) Run test(s)
2) Meet requirements? Yes -> end
3) No-> modify the system and go to 1
Only 1) and 2) is really testing. The 3) represents performance analysis and system tuning (optimization). Functional testing doesn't have that part at all while it requires some specific skills (like some understanding of internals and debugging) and probably a different attitude (more help to look for the problem, not only report it).
And the main difference is if you get into this tuning cycle you are result-driven (with wide understanding of results including all monitoring results). In many cases you find what your next step is when you analyze results of previous. In most cases you don't have real test plan per se until you get out of that tuning cycle (well, you can put together a few thought to make it look like a plan, but you may throw it away after the next test). And that is probably is a fundamental difference from functional testing where we try to make a comprehensive (in a way) plan to cover functionality to make sure that software is good enough. And it is probably why the tuning principle "one change a time" looks unacceptable for people with functional testing background. Well, not literally one*, but it still make sense for me and I still believe that it is a big underlying difference with functional testing.
*Here is my current version of that statement:
Take a Systematic Approach to Changes
The tuning (and often diagnostic) process consists of making changes in the system and evaluating their impact on performance (or problems). It is very important to take a systematic approach to these changes. It could be, for example, the traditional approach of “one change at a time” (also often referred as “one factor at a time” - OFAT) or using design of experiments (DOE) theory. “One change at a time” here does not mean changing only one variable; it can mean changing several related variables to check a particular hypothesis.
The relationship between changes in the system parameters and changes in the product behavior is usually quite complex. Any assumption based on common sense can be wrong. A system’s reaction can be quite the opposite under heavy load. So changing several things at once without a systematic approach will not give an understanding how each change affects results. This could mess up the testing process and lead to incorrect conclusions. All changes and their impacts should be logged to allow rollback and further analysis.
1) Run test(s)
2) Meet requirements? Yes -> end
3) No-> modify the system and go to 1
Only 1) and 2) is really testing. The 3) represents performance analysis and system tuning (optimization). Functional testing doesn't have that part at all while it requires some specific skills (like some understanding of internals and debugging) and probably a different attitude (more help to look for the problem, not only report it).
And the main difference is if you get into this tuning cycle you are result-driven (with wide understanding of results including all monitoring results). In many cases you find what your next step is when you analyze results of previous. In most cases you don't have real test plan per se until you get out of that tuning cycle (well, you can put together a few thought to make it look like a plan, but you may throw it away after the next test). And that is probably is a fundamental difference from functional testing where we try to make a comprehensive (in a way) plan to cover functionality to make sure that software is good enough. And it is probably why the tuning principle "one change a time" looks unacceptable for people with functional testing background. Well, not literally one*, but it still make sense for me and I still believe that it is a big underlying difference with functional testing.
*Here is my current version of that statement:
Take a Systematic Approach to Changes
The tuning (and often diagnostic) process consists of making changes in the system and evaluating their impact on performance (or problems). It is very important to take a systematic approach to these changes. It could be, for example, the traditional approach of “one change at a time” (also often referred as “one factor at a time” - OFAT) or using design of experiments (DOE) theory. “One change at a time” here does not mean changing only one variable; it can mean changing several related variables to check a particular hypothesis.
The relationship between changes in the system parameters and changes in the product behavior is usually quite complex. Any assumption based on common sense can be wrong. A system’s reaction can be quite the opposite under heavy load. So changing several things at once without a systematic approach will not give an understanding how each change affects results. This could mess up the testing process and lead to incorrect conclusions. All changes and their impacts should be logged to allow rollback and further analysis.