Consider: Lessons Learned in performance testing.
Submitted by Ainars Galvans on Fri, 29/12/2006 - 12:12.
books | context-driven testing | performance testing | perspectives
I had an intention to blog few (~10) "lessons learned in performance testing", copying style of a book Lessons Learned in Software Testing. A context-driven approach Hoping that community could extend this with more of them. I’ve been postponing this so many times that I gave up eventually. I hope by this blog to inspirit someone (probably with wider experience in performance testing than me) to take it over.
Prologue
This year I’ve been inspirited a lot by performance testing. Participating WOPR 7 was a peak . Things changes however and I feel willing to spend more time on considering functional testing issues again. Because I still love testing so much. I feel for community which is still not sure about what exactly the testing should be and how to teach it. Even exploratory testing ideas still need to get wide recognition .
(perhaps) Context is even more important in performance testing
I’m proponent of a context-driven testing. Context is essential to decide the process to follow, the methodology/techniques/etc. to implement. It applies to performance testing a lot the same as functional testing: resources (how detailed how quality performance tests they are willing to pay for), skills (is it a functional tests asked to do performance testing without any tools/methodology training?), level of tests (demonstrate SLA agreement is met or help find any issues before production/as far as possible) etc.
There is however at least one significant key aspect of context specific for performance testing: architecture/technology. How simple it will be to use capture and play-back tools or any other tools to emulate load? Is it pure HTTP/HTML traffic, or there are legacy architecture used, secured unstructured (e.g. serialized objects) data transferred etc...
Based on my experience the most significant (and most complicated) aspect in a context is following: the target audience of the software. Let me illustrate this as items with sub-items discussing performance testing specific for each.:
1. Batch-processes, one-user applications and a hardware tests. There are no idea of load, so the only tests you do is compare processing time depending on data amount to process.
1.1. Batch processes/single-threaded server processes. We are typically looking for how they are affected by hardware noises and what level on noise they create on the hardware (memory usage, CPU usage, etc.).
1.2. One-user applications. User experience is what is essential. You can’t simply record response times and report them – you better ask real users working with it and explain the experience in terms of performance.
1.3. Hardware-based testing. Such as testing scanners, embedded systems, etc. I have no real experience but there is a lot of (too much of) configurations to tests as far as I know.
2. Corporate applications. Context: limited to internal usage by company stuff, each person is authorized and trained to perform certain activities, usage may be administratively influenced, etc. The goal of the application is to reduce number of manual work and save related costs ()
2.1. Workflow/BPM applications. I’ve recently blogged about this, lem me quote a little bit ...concerned about performance in terms of business entities (process instances) processed per day... typical user will not only do those actions required to process the business item, they will do actions like search for certain items
2.2. Document/Content Storage. Typical actions are complex searching (including full-text-search), adding and updating content. Optionally issues due to versioning, custom indexing etc. Also issues if several users try to edit the same item simultaneously or optionally – two siblings (e.g. add new child/remove existing to the same item simultaneously).
3. Public applications such as e-shops, home-pages, blogs, etc. Unlimited and hard to forecast number of visitors, scenarios, etc. The application itself directly or indirectly provides/supports income.
3.1. Static or internally interactive HOME-Pages, such as informative homepages. There may be user authentication and even authorization implemented, but the trick is that any activities any user performs are local to this user and does not influence content the other users will see. Those applications don’t typically have scalability issues and the only trick is to forecast the load in order to forecast the hardware required to maintain it.
3.2. “Multi-active” web pages. Such as forums, where one user activity (e.g. add comment) change the content the other user will see (retrieve all comments). As content changes all the content storage issues (see above) may apply. I’m not particularly experience with this type of application tests however.
3.3. [Probably more. I do not have significant experience with public application performance testing. I hope someone who has could update this.]
Each application may consist of one or several items of that tree. For example
corporate applications with public interfaces could mix all the types.
Lessons learned outline – few items for beginning
Few titles for lessons learned as examples if anyone (or a group) consider he might want to write some post, article or even book about lessons he (they) learned during performance testing experiences.
1)Concentrate on emulating real user scenarios instead of simply call functions (loading pages) in a loop.
2)Don’t concentrate to on emulating real user scenarios only. (a href to my previous logs)
3) Knowledge or system architecture is critical to write good test scripts and prepare test data for them
4)Knowledge of test tools and understanding of what they does in general is essential to write good test scripts and prepare test data for them
5)You should use think times that accord to real user scenarios to realize how system will behave in production
6)You could reduce or even ignore think times to gather significant information about system
Prologue
This year I’ve been inspirited a lot by performance testing. Participating WOPR 7 was a peak . Things changes however and I feel willing to spend more time on considering functional testing issues again. Because I still love testing so much. I feel for community which is still not sure about what exactly the testing should be and how to teach it. Even exploratory testing ideas still need to get wide recognition .
(perhaps) Context is even more important in performance testing
I’m proponent of a context-driven testing. Context is essential to decide the process to follow, the methodology/techniques/etc. to implement. It applies to performance testing a lot the same as functional testing: resources (how detailed how quality performance tests they are willing to pay for), skills (is it a functional tests asked to do performance testing without any tools/methodology training?), level of tests (demonstrate SLA agreement is met or help find any issues before production/as far as possible) etc.
There is however at least one significant key aspect of context specific for performance testing: architecture/technology. How simple it will be to use capture and play-back tools or any other tools to emulate load? Is it pure HTTP/HTML traffic, or there are legacy architecture used, secured unstructured (e.g. serialized objects) data transferred etc...
Based on my experience the most significant (and most complicated) aspect in a context is following: the target audience of the software. Let me illustrate this as items with sub-items discussing performance testing specific for each.:
1. Batch-processes, one-user applications and a hardware tests. There are no idea of load, so the only tests you do is compare processing time depending on data amount to process.
1.1. Batch processes/single-threaded server processes. We are typically looking for how they are affected by hardware noises and what level on noise they create on the hardware (memory usage, CPU usage, etc.).
1.2. One-user applications. User experience is what is essential. You can’t simply record response times and report them – you better ask real users working with it and explain the experience in terms of performance.
1.3. Hardware-based testing. Such as testing scanners, embedded systems, etc. I have no real experience but there is a lot of (too much of) configurations to tests as far as I know.
2. Corporate applications. Context: limited to internal usage by company stuff, each person is authorized and trained to perform certain activities, usage may be administratively influenced, etc. The goal of the application is to reduce number of manual work and save related costs ()
2.1. Workflow/BPM applications. I’ve recently blogged about this, lem me quote a little bit ...concerned about performance in terms of business entities (process instances) processed per day... typical user will not only do those actions required to process the business item, they will do actions like search for certain items
2.2. Document/Content Storage. Typical actions are complex searching (including full-text-search), adding and updating content. Optionally issues due to versioning, custom indexing etc. Also issues if several users try to edit the same item simultaneously or optionally – two siblings (e.g. add new child/remove existing to the same item simultaneously).
3. Public applications such as e-shops, home-pages, blogs, etc. Unlimited and hard to forecast number of visitors, scenarios, etc. The application itself directly or indirectly provides/supports income.
3.1. Static or internally interactive HOME-Pages, such as informative homepages. There may be user authentication and even authorization implemented, but the trick is that any activities any user performs are local to this user and does not influence content the other users will see. Those applications don’t typically have scalability issues and the only trick is to forecast the load in order to forecast the hardware required to maintain it.
3.2. “Multi-active” web pages. Such as forums, where one user activity (e.g. add comment) change the content the other user will see (retrieve all comments). As content changes all the content storage issues (see above) may apply. I’m not particularly experience with this type of application tests however.
3.3. [Probably more. I do not have significant experience with public application performance testing. I hope someone who has could update this.]
Each application may consist of one or several items of that tree. For example
corporate applications with public interfaces could mix all the types.
Lessons learned outline – few items for beginning
Few titles for lessons learned as examples if anyone (or a group) consider he might want to write some post, article or even book about lessons he (they) learned during performance testing experiences.
1)Concentrate on emulating real user scenarios instead of simply call functions (loading pages) in a loop.
2)Don’t concentrate to on emulating real user scenarios only. (a href to my previous logs)
3) Knowledge or system architecture is critical to write good test scripts and prepare test data for them
4)Knowledge of test tools and understanding of what they does in general is essential to write good test scripts and prepare test data for them
5)You should use think times that accord to real user scenarios to realize how system will behave in production
6)You could reduce or even ignore think times to gather significant information about system