Pitfalls of the "Waterfall" Approach to Performance Testing
Submitted by Alexander Podelko on Mon, 24/03/2008 - 22:23.
design & development | non-functional testing | performance testing | performance testing patterns
Looks like the pre-production validation approach to performance testing becomes typical for large corporations (if there is any at all):
-get the system ready
-develop all scripts requested (sometimes offshore)
-run them all together
-compare with the requirements provided
-allow some percentage of errors according to the requirements
-involve the development team if requirements were missed
It is some kind of the "waterfall" approach to performance testing. It looks like well established, mature process. But there are many serious pitfalls on this way. Here are several most serious:
- It assumes that the system is completely ready. At least for the functionality included in performance testing. It means that it happens pretty late in the development cycle and if any serious problems would be found, fixing them would be very expensive. There is no way to do such full-scope performance testing early in the development lifecycle. If we want to do something earlier it should be more agile / explorative process.
- Performance testing scripts (or whatever used to create load – perhaps a test harness) are software themselves. Using recording (typical way to create performance testing scripts) sometimes creates deceptive feeling that creating scripts is quick and easy; but correlation, parameterization, debugging, and verification may be pretty challenging tasks. Running a script for one user without errors doesn't prove anything. I have seen large-scale performance testing at a large corporation where none of the script actually get through logon (single sign-on token wasn't correlated) – but performance testing was declared successful and results were reported to management.
- Running all scripts together make it very difficult to tune and troubleshoot. It usually becomes a good illustration to the Shot-in-the-dark antipattern. Or you need to go back and disintegrate tests to find the exact part causing problems. Moreover, tuning and performance troubleshooting are iterative processes which difficult to place inside the "waterfall" approach. And, in most cases, it is not something you can do off-line – you need to tune system and fix the major problems before results will make sense.
- Running a single large test (or even a few of them) gives minimal information about the system behavior. You can not build any kind of model of it (either formal or informal), you will not see any relationship between workload and system behavior. In most cases the workload used in performance tests is only an educated guess, so you need to understand how stable the system will be and how consistent results would be if real workload would be somewhat different.
-get the system ready
-develop all scripts requested (sometimes offshore)
-run them all together
-compare with the requirements provided
-allow some percentage of errors according to the requirements
-involve the development team if requirements were missed
It is some kind of the "waterfall" approach to performance testing. It looks like well established, mature process. But there are many serious pitfalls on this way. Here are several most serious:
- It assumes that the system is completely ready. At least for the functionality included in performance testing. It means that it happens pretty late in the development cycle and if any serious problems would be found, fixing them would be very expensive. There is no way to do such full-scope performance testing early in the development lifecycle. If we want to do something earlier it should be more agile / explorative process.
- Performance testing scripts (or whatever used to create load – perhaps a test harness) are software themselves. Using recording (typical way to create performance testing scripts) sometimes creates deceptive feeling that creating scripts is quick and easy; but correlation, parameterization, debugging, and verification may be pretty challenging tasks. Running a script for one user without errors doesn't prove anything. I have seen large-scale performance testing at a large corporation where none of the script actually get through logon (single sign-on token wasn't correlated) – but performance testing was declared successful and results were reported to management.
- Running all scripts together make it very difficult to tune and troubleshoot. It usually becomes a good illustration to the Shot-in-the-dark antipattern. Or you need to go back and disintegrate tests to find the exact part causing problems. Moreover, tuning and performance troubleshooting are iterative processes which difficult to place inside the "waterfall" approach. And, in most cases, it is not something you can do off-line – you need to tune system and fix the major problems before results will make sense.
- Running a single large test (or even a few of them) gives minimal information about the system behavior. You can not build any kind of model of it (either formal or informal), you will not see any relationship between workload and system behavior. In most cases the workload used in performance tests is only an educated guess, so you need to understand how stable the system will be and how consistent results would be if real workload would be somewhat different.