Multiple Dimensions of Performance Testing
Submitted by Alexander Podelko on Wed, 09/04/2008 - 04:45.
non-functional testing | performance testing | performance testing patterns
Almost all experts agree that pre-deployment "waterfall" performance testing (which, with the record/playback method, confused by many as the performance testing itself) is not enough - too little, too late. Actually it is just one very specific way of performance testing - with a full spectrum of other approaches, which are used so infrequently (at least as intentional performance testing techniques) that I don't recall finding any good classification. Thinking about it, I see several dimensions of performance testing which, although definitely correlated, probably might be considered somewhat independently - of course, just a raw idea for the moment, just an effort to order thoughts a little.
1) System vs. unit performance testing
The unit here maybe any part of the system like a component, service, or device. There is absolutely no reason not to do it, but there are not many places where it is a standard practice. Why we should wait until the whole system would be assembled together to start performance testing? We don't wait in functional testing, why we should in performance? The pre-deployment performance test is an analogue of system or integration test, but usually it is done without prior testing of each part.
Here may be significant advantage of test-driven development – if you think how to break the system into components in such way that you may test them separately for performance, you will need to fix only integration issues when you put the system together.
Is it early? It definitely is comparing with pre-deployment performance testing. But other than that, it does not define other dimensions: it may be early or late (when the component is ready), investigation or just validation, agile or “waterfall”.
Ainars Galvans listed many useful techniques in his blog. We probably can put multi-user testing as described Karen Johnson's article in the same category. We also may consider hardware performance testing in this category too – hardware is, in some sense, a component of the whole system.
Unit testing is probably where we want to investigate how different variables – like amount of data, security, etc. - impact performance. In most cases we have simpler test cases and shorter tests in unit performance testing; we usually very limited in the number of variable combinations we can test in during system performance testing - and have drastically more combinations.
The main problem here is that you need to change mentality from record/playback performance testing to performance engineering (the terminology is vague here, perhaps it may be better wording for the idea). You need to translate end-user usage into component/unit-level usage, end-user requirements into component/unit-level requirements, etc. You need to go from record/playback to some kind of programming – both to generate workload and create stubs to isolate the component from other parts of the system. You need to go from black/dark grey box testing to, say, "light grey".
2) Doing performance testing in an agile way vs. "waterfall" approach
Actually I advocated for a long time that performance testing is agile by definition and that you actually can use the "waterfall" approach only in some trivial cases (like testing a small upgrade for a mature system) – when you don’t expect any performance issues or tuning cycles.
Performance testing is iterative by its nature, you adjust your plans based on results you get. You run a simple test and get a lot of information about the system. To be efficient you need to incorporate the feedback, make modifications to the system if necessary, and adjust you plans.
For example, you plan to run 20 different tests. You run the first and find that you have bottleneck somewhere (for example, the number of threads in Web Server). Actually running other 19 would be just wasting of time (if the all use Web Server) until you find and eliminate the bottleneck. Rather you may need to modify your test to trace down the problem.
Why I use word agile here and not just talk about exploration and investigation - these terms look indeed somewhat correlated? Because it may be no special investigation there. Even if you limit you goal to validation / verification, approaching testing with an agile, iterative approach you meet your goals faster and more efficiently and, of course, learn more about the system on the way.
For example, when I get a script ready (whatever it can be – it definitely is not limited to a load testing tool script), I am running one, a few, and many users (many depends on the system), analyze results (including, of course, system monitoring), trying to sort out any errors. The source of errors can be quite different – script error, functional error, or a consequence of a performance bottleneck. It doesn't make much sense to add load / scripts until you figure out what is going on. Even with one script you can find many system problems and make an iteration of system's tuning. Running scripts separately allows you to make some kind of system's "model". I don't mean any kind of formal model – something like workload A creates noticeable load on components X and Y and it quite could be that cpu would be a bottleneck, while component Z is hardly touched. As you run more and more complex tests you verify results you get against your "model", your understanding how the system behaves – and if they don't match, you need to figure out what is wrong.
Using the "waterfall" approach doesn’t change the nature of performance testing; it just means that you probably do a lot of extra work and come back to the same point – performance troubleshooting and tuning - much later in the cycle. Not to mention that large tests using multiple use cases are usually a bad point to start performance troubleshooting and tuning - symptoms you see may be cumulative effects of multiple issues.
3) Doing exploration / investigation vs. validation / verification
I believe that this terminology in this context was introduced into performance testing by Scott Barber in earlier versions of Investigation vs. Validation. Scott’s point is that we need both: "Investigate performance early; validate performance last."
The terminology somewhat deviates from the traditional use, not sure how important it is. I am also not sure if investigation here is an application of exploratory testing to performance or we have any difference here.
Performance testing isn't exact science. It is a way to decrease the risk, not to eliminate it completely. Results are as meaningful as the test and environment you created – how close it is to the reality is a separate question (in some cases you base your test design / performance requirements on pretty reliable data, in some cases it is a pure guess). Usually performance testing has tiny functional coverage, no emulation of unexpected events, etc. In many cases incomplete environment and data (still not the reason not to try to make it as close as possible – you will get some differences even in this case).
All this contradict to the straightforward approach to performance testing – test X users simulating use cases A and B. This way we leave aside a lot of questions, for example: How many users the system can handle? What happens if we add scenarios C and D? Do ratios of use cases matter? What if some administrative activities happen in parallel? Looks like all these questions get into the investigation category.
Perhaps you even need to do some investigation to get some understanding of the system before you start creating performance testing plans.
4) Early vs. late performance testing
"Early", I believe, is when the system or component is not ready yet.
Mike Kelly's post about testing in different environments probably get in this category. Ainars Galvans also listed some useful techniques in his blog.
We probably can talk here about Software Performance Engineering as taught by Dr. Connie Smith and Dr. Lloyd Williams. They do not elaborate much on testing, but actually you can not implement it without testing / measurements behind the scene to get numbers and verify models.
One specific case here is doing performance testing in agile projects. This is, I guess, what is Antony Marcano's vision is about. Actually I don't see any particular reason not to do it this way right now (except using visual tool which looks like something that won't be available in near future) if performance is a real concern. Having some kind of working system on each iteration makes performance testing easy and natural – you don't need to go for all these tricks of getting performance testing early in traditional methods of development when you don't heave working system until the very end of the development lifecycle. The only reason not to do it I can think of is that performance isn't a serious enough concern to bother.
See also Scott Barber’s An Explanation of Performance Testing on an Agile Team. Part I and Part II.
Returning to the "traditional" way of performance testing and looking where it is in these four dimensions we see that it matches the following intersection: 1) system 2) "waterfall" 3) validation/verification 4) late.
1) System vs. unit performance testing
The unit here maybe any part of the system like a component, service, or device. There is absolutely no reason not to do it, but there are not many places where it is a standard practice. Why we should wait until the whole system would be assembled together to start performance testing? We don't wait in functional testing, why we should in performance? The pre-deployment performance test is an analogue of system or integration test, but usually it is done without prior testing of each part.
Here may be significant advantage of test-driven development – if you think how to break the system into components in such way that you may test them separately for performance, you will need to fix only integration issues when you put the system together.
Is it early? It definitely is comparing with pre-deployment performance testing. But other than that, it does not define other dimensions: it may be early or late (when the component is ready), investigation or just validation, agile or “waterfall”.
Ainars Galvans listed many useful techniques in his blog. We probably can put multi-user testing as described Karen Johnson's article in the same category. We also may consider hardware performance testing in this category too – hardware is, in some sense, a component of the whole system.
Unit testing is probably where we want to investigate how different variables – like amount of data, security, etc. - impact performance. In most cases we have simpler test cases and shorter tests in unit performance testing; we usually very limited in the number of variable combinations we can test in during system performance testing - and have drastically more combinations.
The main problem here is that you need to change mentality from record/playback performance testing to performance engineering (the terminology is vague here, perhaps it may be better wording for the idea). You need to translate end-user usage into component/unit-level usage, end-user requirements into component/unit-level requirements, etc. You need to go from record/playback to some kind of programming – both to generate workload and create stubs to isolate the component from other parts of the system. You need to go from black/dark grey box testing to, say, "light grey".
2) Doing performance testing in an agile way vs. "waterfall" approach
Actually I advocated for a long time that performance testing is agile by definition and that you actually can use the "waterfall" approach only in some trivial cases (like testing a small upgrade for a mature system) – when you don’t expect any performance issues or tuning cycles.
Performance testing is iterative by its nature, you adjust your plans based on results you get. You run a simple test and get a lot of information about the system. To be efficient you need to incorporate the feedback, make modifications to the system if necessary, and adjust you plans.
For example, you plan to run 20 different tests. You run the first and find that you have bottleneck somewhere (for example, the number of threads in Web Server). Actually running other 19 would be just wasting of time (if the all use Web Server) until you find and eliminate the bottleneck. Rather you may need to modify your test to trace down the problem.
Why I use word agile here and not just talk about exploration and investigation - these terms look indeed somewhat correlated? Because it may be no special investigation there. Even if you limit you goal to validation / verification, approaching testing with an agile, iterative approach you meet your goals faster and more efficiently and, of course, learn more about the system on the way.
For example, when I get a script ready (whatever it can be – it definitely is not limited to a load testing tool script), I am running one, a few, and many users (many depends on the system), analyze results (including, of course, system monitoring), trying to sort out any errors. The source of errors can be quite different – script error, functional error, or a consequence of a performance bottleneck. It doesn't make much sense to add load / scripts until you figure out what is going on. Even with one script you can find many system problems and make an iteration of system's tuning. Running scripts separately allows you to make some kind of system's "model". I don't mean any kind of formal model – something like workload A creates noticeable load on components X and Y and it quite could be that cpu would be a bottleneck, while component Z is hardly touched. As you run more and more complex tests you verify results you get against your "model", your understanding how the system behaves – and if they don't match, you need to figure out what is wrong.
Using the "waterfall" approach doesn’t change the nature of performance testing; it just means that you probably do a lot of extra work and come back to the same point – performance troubleshooting and tuning - much later in the cycle. Not to mention that large tests using multiple use cases are usually a bad point to start performance troubleshooting and tuning - symptoms you see may be cumulative effects of multiple issues.
3) Doing exploration / investigation vs. validation / verification
I believe that this terminology in this context was introduced into performance testing by Scott Barber in earlier versions of Investigation vs. Validation. Scott’s point is that we need both: "Investigate performance early; validate performance last."
The terminology somewhat deviates from the traditional use, not sure how important it is. I am also not sure if investigation here is an application of exploratory testing to performance or we have any difference here.
Performance testing isn't exact science. It is a way to decrease the risk, not to eliminate it completely. Results are as meaningful as the test and environment you created – how close it is to the reality is a separate question (in some cases you base your test design / performance requirements on pretty reliable data, in some cases it is a pure guess). Usually performance testing has tiny functional coverage, no emulation of unexpected events, etc. In many cases incomplete environment and data (still not the reason not to try to make it as close as possible – you will get some differences even in this case).
All this contradict to the straightforward approach to performance testing – test X users simulating use cases A and B. This way we leave aside a lot of questions, for example: How many users the system can handle? What happens if we add scenarios C and D? Do ratios of use cases matter? What if some administrative activities happen in parallel? Looks like all these questions get into the investigation category.
Perhaps you even need to do some investigation to get some understanding of the system before you start creating performance testing plans.
4) Early vs. late performance testing
"Early", I believe, is when the system or component is not ready yet.
Mike Kelly's post about testing in different environments probably get in this category. Ainars Galvans also listed some useful techniques in his blog.
We probably can talk here about Software Performance Engineering as taught by Dr. Connie Smith and Dr. Lloyd Williams. They do not elaborate much on testing, but actually you can not implement it without testing / measurements behind the scene to get numbers and verify models.
One specific case here is doing performance testing in agile projects. This is, I guess, what is Antony Marcano's vision is about. Actually I don't see any particular reason not to do it this way right now (except using visual tool which looks like something that won't be available in near future) if performance is a real concern. Having some kind of working system on each iteration makes performance testing easy and natural – you don't need to go for all these tricks of getting performance testing early in traditional methods of development when you don't heave working system until the very end of the development lifecycle. The only reason not to do it I can think of is that performance isn't a serious enough concern to bother.
See also Scott Barber’s An Explanation of Performance Testing on an Agile Team. Part I and Part II.
Returning to the "traditional" way of performance testing and looking where it is in these four dimensions we see that it matches the following intersection: 1) system 2) "waterfall" 3) validation/verification 4) late.