Performance testing techniques OR against realistic load
Submitted by Ainars Galvans on Fri, 30/06/2006 - 12:11.
performance testing
Alexander Podelko have been asking me why I differentiate performance testing and investigation. I’m now ready to answer: I won’t differentiate them any more. Instead I will differentiate performance testing and performance validation, which I will call performance acceptance (PA).
Yes I compare this to UAT, which I love to call UA with “testing” because in UA(T) they apply real-life patterns, while in other types of testing we use testing techniques to identify those (probably not-so-typical) scenarios that triggers error.
Main task in PA is to gather as detailed performance requirements as you could (including realistic user scenarios, etc) and to play-back them as close as possible. Main task in performance testing is to detect as much “bugs that matter” as possible.
So there are techniques for performance testing, those that helps to identify bugs that matter despite of how detailed our requirements are. I’ve just realized that my recent blogs about simple performance investigation are actually the techniques or almost like them. I will provide one more technique “equivalence partitioning“ in this blog as well.
I’m lucky to be tasked with investigation of a “product”
You may have noticed that my practices and my passion is behind what Scott Barber called Investigation, i.e. "collecting information" with goal of determining and improving the product quality. It is opposed to Validation of set/presumed “expectation” (requirements). Validation requires realistic load to be emulated, which is more like business challenge. It seems to be quite an exciting challenge as much as I have red about performance in books, magazines and blogs. I have also done that in several projects and even talked about it in Latvia test conference. But I’m happy that I have been recently moved from application performance testing to product performance testing. While those are terms have different definitions I suggest you to read discussion about functional testing difference . Or I could give short summary: project has a goal to solve single customer single issue (automation of a specific business), while product is a software that is aimed to help solve certain type of issue(s) for any (or limited type of) customers. There may be out-of-box solution, or the solution may need more development effort.
I’m happy about it because there is no “realistic load” during product implementation. I only have possible customers or business opportunities. I have to analyze “sample customer” cases and come up with generic load scenarios. Why I like this? Probably thanks to my math. background joined with the fact that I don’t like modeling and statistics (yes I have an experience in both, that’s why I know I don’t like them). My hobby is combinatorics (I teach undergraduates who are going to International Mathematic Olympiads).
An illustrations against validation type of testing
To better illustrate why I don’t like validation I would like to stress the word quality in the investigation description above. Suppose you are validating the requirements that say each page response time should be below 5 seconds for 100 users load. Now suppose that developers created application capable to have 0-2 seconds response times, but found that 100 virtual users with recoded think times makes CPU 100% busy, which end up with few transactions having even 10 second response times. Developers add 2 seconds delay on server side for any request. Now CPU is only ~70% utilized and response times are 2-4 seconds so the validation tests are passed. But are you satisfied with the quality?
Typical product scenarios
As I mentioned above if you have a generic product there are no typical scenarios; no available test data; no defined hardware. And I’m tasked to do a “performance testing”= reduce risk of any customer reporting performance issues that can’t be addressed by adding hardware or the additional hardware cost is unreasonable high.
What I’ve been tasked with to do a lot of investigations to provide data to understand if the quality (in terms pf performance and reliability) should be improved at this point. My primary goal is to detect issues/defects in the code. There are at least two options. The first is to select few sample customers, to learn what hardware they has, get their data and usage scenarios and test those hoping that they will be representative enough. Even if I would have the access to that type of information I prefer another choice.
I’m not going to take sample customers (e.g. applying something like equivalence partitioning of the customers). I will not take sample use cases, which are not real for any particular customer but an approximation of customer group. Not sure if I make any sense so far with this description so I will better describe this on example below:
Performance testing technique: Equivalence Partitioning
Now to illustrate what I mean by modeling generic load: suppose I record search script. I don’t care for what is the input values, I care about result-set size. I have in my script test steps Search_For_One, Search_For_100, Search_For_1000. I don’t really care about the specific SQL as SQL tuning depend on custom implementation. So I’m only going to search by item ID and nothing more. I fill in database so that for each ID in some range there are an item per ID and write following Search_For_100 script. ID_From = Random(1-1000). ID_To= ID_From+100. DoSearch (FromID= ID_From;ToID=ID_TO).
Yes, I know there are going to be those “unrealistic” database queries executed. But again DB performance is not my concern at this point. My concern is hot our application process the data received from DB.
Yes I compare this to UAT, which I love to call UA with “testing” because in UA(T) they apply real-life patterns, while in other types of testing we use testing techniques to identify those (probably not-so-typical) scenarios that triggers error.
Main task in PA is to gather as detailed performance requirements as you could (including realistic user scenarios, etc) and to play-back them as close as possible. Main task in performance testing is to detect as much “bugs that matter” as possible.
So there are techniques for performance testing, those that helps to identify bugs that matter despite of how detailed our requirements are. I’ve just realized that my recent blogs about simple performance investigation are actually the techniques or almost like them. I will provide one more technique “equivalence partitioning“ in this blog as well.
I’m lucky to be tasked with investigation of a “product”
You may have noticed that my practices and my passion is behind what Scott Barber called Investigation, i.e. "collecting information" with goal of determining and improving the product quality. It is opposed to Validation of set/presumed “expectation” (requirements). Validation requires realistic load to be emulated, which is more like business challenge. It seems to be quite an exciting challenge as much as I have red about performance in books, magazines and blogs. I have also done that in several projects and even talked about it in Latvia test conference. But I’m happy that I have been recently moved from application performance testing to product performance testing. While those are terms have different definitions I suggest you to read discussion about functional testing difference . Or I could give short summary: project has a goal to solve single customer single issue (automation of a specific business), while product is a software that is aimed to help solve certain type of issue(s) for any (or limited type of) customers. There may be out-of-box solution, or the solution may need more development effort.
I’m happy about it because there is no “realistic load” during product implementation. I only have possible customers or business opportunities. I have to analyze “sample customer” cases and come up with generic load scenarios. Why I like this? Probably thanks to my math. background joined with the fact that I don’t like modeling and statistics (yes I have an experience in both, that’s why I know I don’t like them). My hobby is combinatorics (I teach undergraduates who are going to International Mathematic Olympiads).
An illustrations against validation type of testing
To better illustrate why I don’t like validation I would like to stress the word quality in the investigation description above. Suppose you are validating the requirements that say each page response time should be below 5 seconds for 100 users load. Now suppose that developers created application capable to have 0-2 seconds response times, but found that 100 virtual users with recoded think times makes CPU 100% busy, which end up with few transactions having even 10 second response times. Developers add 2 seconds delay on server side for any request. Now CPU is only ~70% utilized and response times are 2-4 seconds so the validation tests are passed. But are you satisfied with the quality?
Typical product scenarios
As I mentioned above if you have a generic product there are no typical scenarios; no available test data; no defined hardware. And I’m tasked to do a “performance testing”= reduce risk of any customer reporting performance issues that can’t be addressed by adding hardware or the additional hardware cost is unreasonable high.
What I’ve been tasked with to do a lot of investigations to provide data to understand if the quality (in terms pf performance and reliability) should be improved at this point. My primary goal is to detect issues/defects in the code. There are at least two options. The first is to select few sample customers, to learn what hardware they has, get their data and usage scenarios and test those hoping that they will be representative enough. Even if I would have the access to that type of information I prefer another choice.
I’m not going to take sample customers (e.g. applying something like equivalence partitioning of the customers). I will not take sample use cases, which are not real for any particular customer but an approximation of customer group. Not sure if I make any sense so far with this description so I will better describe this on example below:
Performance testing technique: Equivalence Partitioning
Now to illustrate what I mean by modeling generic load: suppose I record search script. I don’t care for what is the input values, I care about result-set size. I have in my script test steps Search_For_One, Search_For_100, Search_For_1000. I don’t really care about the specific SQL as SQL tuning depend on custom implementation. So I’m only going to search by item ID and nothing more. I fill in database so that for each ID in some range there are an item per ID and write following Search_For_100 script. ID_From = Random(1-1000). ID_To= ID_From+100. DoSearch (FromID= ID_From;ToID=ID_TO).
Yes, I know there are going to be those “unrealistic” database queries executed. But again DB performance is not my concern at this point. My concern is hot our application process the data received from DB.