Peformance Testing and Performance Engineering
Submitted by Alexander Podelko on Tue, 16/01/2007 - 04:21.
non-functional testing | performance testing
I completely agree with Wikipedia definitions that performance testing is a part of performance engineering.
The scope of performance testing is not well defined and varies from creating scripts and running tests in its narrow definition to all kinds of performance-related activities when synthetic workload is applied to the system in its wide definition. And the wide definition often includes performance analysis, performance troubleshooting / diagnostics, tuning, and capacity planning making the word "testing" to be rather disguising the substance of what it behind. The difference behind these definitions can partly explain the wide range of opinions about a path from functional testing to performance testing: while the path from automated functional testing to "narrow definition" performance testing is quite straightforward, skills required by other areas of "wide definition" performance testing are quite different.
If speak about the "wide definition" of performance testing, it is definitely highly interwoven with performance engineering. Performance testing is a source of data for any kind of modeling and capacity planning activities. It is way to calibrate and verify models (where models mean not only formal models, but any kind of perception of how the system is supposed to work). If any pro-active performance experiment consider as performance testing, only other source of data may be observation or log analysis of real work with the system, when you never know what exact level of load was applied and what else happened around (so data won’t be "clean"). A notable exception is data defining load (like throughput) which are parameters of performance testing; so they can’t come from its results, they should come from analyzing real systems or other sources.
Unique opportunity to exactly reproduce workload in performance testing is priceless during tuning and diagnostics of performance problems. You get exact impact of suggested change, while, for example, doing that in production you never know what other unknown factors may impact results.
While performance testing wasn’t discussed much by Dr. Connie U. Smith or Dr. Lloyd G. Williams (people who probably done most to promote software performance engineering), it is implied as an integral part of performance engineering. For example, the QSEM: Quantitative Scalability Evaluation Method paper really described performance testing to investigate system’s scalability.
Performance testing is not "sexy" from scientific point of view. While it has rather mature practice, there is no mature performance testing theory around. There is no classic textbook, for example, most education goes around vendor’s tools. It is difficult to put many formulas in it (I always enjoy performance management / capacity planning books where after some initial performance- and computer-related discussions usually there are many chapters of plain queuing theory and other part of mathematics that are supposed to be used).
Performance testing is a very good step to introduce performance engineering. At least it is the area where some low hanging performance fruits could be easily found and it is quite clear what to do it. While performance testing theory is almost non-existing, performance testing practice is pretty mature (although often grouped around tool vendors, their training and user communities). It looks like performance testing groups becomes the centers of performance engineering ("a performance center of excellence", for example) in the enterprise instead of performance analysis and capacity planning groups that were typical for corporations using mainframes.
The scope of performance testing is not well defined and varies from creating scripts and running tests in its narrow definition to all kinds of performance-related activities when synthetic workload is applied to the system in its wide definition. And the wide definition often includes performance analysis, performance troubleshooting / diagnostics, tuning, and capacity planning making the word "testing" to be rather disguising the substance of what it behind. The difference behind these definitions can partly explain the wide range of opinions about a path from functional testing to performance testing: while the path from automated functional testing to "narrow definition" performance testing is quite straightforward, skills required by other areas of "wide definition" performance testing are quite different.
If speak about the "wide definition" of performance testing, it is definitely highly interwoven with performance engineering. Performance testing is a source of data for any kind of modeling and capacity planning activities. It is way to calibrate and verify models (where models mean not only formal models, but any kind of perception of how the system is supposed to work). If any pro-active performance experiment consider as performance testing, only other source of data may be observation or log analysis of real work with the system, when you never know what exact level of load was applied and what else happened around (so data won’t be "clean"). A notable exception is data defining load (like throughput) which are parameters of performance testing; so they can’t come from its results, they should come from analyzing real systems or other sources.
Unique opportunity to exactly reproduce workload in performance testing is priceless during tuning and diagnostics of performance problems. You get exact impact of suggested change, while, for example, doing that in production you never know what other unknown factors may impact results.
While performance testing wasn’t discussed much by Dr. Connie U. Smith or Dr. Lloyd G. Williams (people who probably done most to promote software performance engineering), it is implied as an integral part of performance engineering. For example, the QSEM: Quantitative Scalability Evaluation Method paper really described performance testing to investigate system’s scalability.
Performance testing is not "sexy" from scientific point of view. While it has rather mature practice, there is no mature performance testing theory around. There is no classic textbook, for example, most education goes around vendor’s tools. It is difficult to put many formulas in it (I always enjoy performance management / capacity planning books where after some initial performance- and computer-related discussions usually there are many chapters of plain queuing theory and other part of mathematics that are supposed to be used).
Performance testing is a very good step to introduce performance engineering. At least it is the area where some low hanging performance fruits could be easily found and it is quite clear what to do it. While performance testing theory is almost non-existing, performance testing practice is pretty mature (although often grouped around tool vendors, their training and user communities). It looks like performance testing groups becomes the centers of performance engineering ("a performance center of excellence", for example) in the enterprise instead of performance analysis and capacity planning groups that were typical for corporations using mainframes.