Exploratory testing and Test Cases
Submitted by Ainars Galvans on Fri, 17/11/2006 - 08:02.
exploratory testing | test management
I found that my Why do we write test cases sometimes appears in popular content. I’m happy to extend it with details about “case 2) Test Cases are for team internal use only.” More about documentation for full life-cycle: new feature test planning and execution and such a beast as exploratory regression tests.
Read how I’m using Test Case Management as a tool and test cases as documents to improve my Exploratory testing efficiency and visibility (reporting). I’m lucky not to be concerned about test credibility. If one of your main concerns is credibility – I rather recommend Heursitic and other articles by James Bach.
Test preparation and test execution steps
No matter how scripted your tests are there will always be test preparation steps. While doing exploratory testing there are a little actual difference between those two categories. My idea is simple – we don’t have to test carefully those features that will be invoked in test preparation anyway. While thinking of what to test we should concentrate on those features that are “leafs” in a tree of all test scenarios we are going to perform. Although each test typically starts with login, the set of test scenarios however is not going to always look like a single tree. A bush in best case, a total maze at worst. With this blog I’m trying to make some order in there.
Exploratory testing: 4 categories of features
I divide all features into following sets: core features (main trunk), horizontal features (other trunks), vertical features (main trunk branches) and diagonal features (other trunk branches). Core features are those that you will be forced to use while doing other feature tests. The most typical core feature is login. You can’t test anything without being logged in. So any reason to write standalone test case for login?! There are much more core and more-or-less core features – those that you use to prepare data or state for other tests.
Vertical features are those which are the last step on any test scenario – you are not going to use them in test preparation on a purpose, but you could use them accidentally during preparation. Those are typically covered by test design text-books(boundary values, etc.). An example could be entering wrong password during login. The idea here is that there are a good chance that you will invoke them time-to-time – as frequent as the customers will. Unless your test project context is about users who does a lot of errors, for example web-app with typical user a person who see this page for first time) – but you as a professional tester could invoke them on a purpose in this case.
The next set of features is horizontal ones. Horizontal features are those features that you could and will bypass using while preparing for other tests. You can’t invoke them occasionally by simply mistyping or typing specific values and. For example you’ve got two different login types – one requiring typing in user name and password and another reusing your windows credentials. If all testers use only one, which is more suitable, you will never test the second one. If the choices are limited and number of testers big enough you could ask each to use own configuration, but don’t abuse this idea. I’m afraid those are the features that you will need to deal with in systematic way – for example writing test cases. Before I go into details of writing test cases let me remind of the last set – diagonal features. Let me start with example this time - if your testers use type in password configuration and your test case is about single-login (reusing windows credentials) how about testing such issues as your username of password in windows don’t fit to application’s restrictions (for example too long password)? Diagonal features are those features that are vertical to horizontal ones, that’s why I call them diagonal. What I do is simple - when I write test cases for horizontal features I never write them too specific, so that tester who will run it have a chance to do something different and step into diagonal feature.
Planning tests? Actually snap-shooting the current vision
My testing is not 100% exploratory. Although specifications are almost always blur, design incomplete and outdated, there are a lot of sources for information gathering. You could read and try talking to anyone involved (managers, customers, developers) – if you don’t abuse their time and do your homework(reading) carefully they are going to cooperate. Or at least I’ve been lucky to have them most of the time.
This list is my test plan to be updated later. It does not describe what I will be testing, instead it describes what the tem believes at this moment need to be tested. This is a snapshot of this moment, at which development is not completed yet and things are going to change, but this is the moment when whole team is thinking about “the big picture” and not yet concerned with implementation details. That’s why this is the best time to write down the ideas and call this initial test plan. Not because we need a formal document to follow.
So I’ve typically been able to write down the initial list of horizontal features I want to test based on this conversation. The list is never correct or complete – I know I’m going to remove some impossible ones and add more that my explorations will illuminate. However I need the initial list for two purposes: to control my exploration and to make sure that my exploration has not failed to find significant items I’ve read/heard about.
Test Execution and documenting regression test cases
So test plan only document the horizontal out of 4 domains of tests. Guess what – the same does my test cases. They are actually update version of the initial snap-shot with the complete list of horizontal features – after coding and exploratory testing is completed. Frequently I don’t write any test case at all – simply update the test plan document with detailed list. Here I probably need to comment that my test plans and test cases are created as Lotus Notes documents with rather rich content – colors, fonts, tables, lists, etc. I may want to create test cases if the test plan becomes too large, however. By too large I mean either physically long or too long to execute – if you divide them into test cases you will could use your progress-reporting feature of your test case management database! That’s why I mentioned visibility in the beginning of this blog.
So even if I write test case the document contain the list of idea or the goals of test, not the exact steps to execute it. The test case may start with a configuration hits (I never call them setup steps). The typical test cases consist of a table with following columns: Feature/Idea to test; Expected results (generic, such as “should process this error gracefully”); actual results (if differ); data-attachments (reusable data set to import in order to boost the next execution); hints (screen-shots from the last execution) and comments (also references to documentation). Rows of the table list ALL actual horizontal features and even ideas that tester should not forget to test. Optionally there is a column that describes regression risk/execution priority (e.g. saying only test this during full functional testing).
This is reason also why I’ve given the name to my test case type a checklist.
Few Issues
If you think you like my idea a lot and you want try it out – here are some warnings.
Issue #1: gaining credibility of this approach. Your credibility once you take this approach will be the biggest issue. I earned credibility through doing excellent work instead of arguing my approach. I’m maybe still too young or something but I believe that job done is what I’m judged by not the credibility of how I did it. Managers don’t want to know why it was impossible to find all those defects – they are going to blame you anyway, so the best you could do to find them. Only if you can’t – you should think of a ways to protect yourself, but you can’t win a war if your only strategy is offence.
Issues #2 Requires stable team. Some people even in my company don’t like this approach, because I don’t describe exact test steps. If you give this test case to a different tester (not one who have written it) this will take a lot of time to execute it. Yes, they are right, just the same as you can’t expect to give a new developer in a team to read architecture and design documents and assume he immediately get up to speed. I’ve seen developers in a project team working for year and still supposed as a novice to only get simple tasks.
Read how I’m using Test Case Management as a tool and test cases as documents to improve my Exploratory testing efficiency and visibility (reporting). I’m lucky not to be concerned about test credibility. If one of your main concerns is credibility – I rather recommend Heursitic and other articles by James Bach.
Test preparation and test execution steps
No matter how scripted your tests are there will always be test preparation steps. While doing exploratory testing there are a little actual difference between those two categories. My idea is simple – we don’t have to test carefully those features that will be invoked in test preparation anyway. While thinking of what to test we should concentrate on those features that are “leafs” in a tree of all test scenarios we are going to perform. Although each test typically starts with login, the set of test scenarios however is not going to always look like a single tree. A bush in best case, a total maze at worst. With this blog I’m trying to make some order in there.
Exploratory testing: 4 categories of features
I divide all features into following sets: core features (main trunk), horizontal features (other trunks), vertical features (main trunk branches) and diagonal features (other trunk branches). Core features are those that you will be forced to use while doing other feature tests. The most typical core feature is login. You can’t test anything without being logged in. So any reason to write standalone test case for login?! There are much more core and more-or-less core features – those that you use to prepare data or state for other tests.
Vertical features are those which are the last step on any test scenario – you are not going to use them in test preparation on a purpose, but you could use them accidentally during preparation. Those are typically covered by test design text-books(boundary values, etc.). An example could be entering wrong password during login. The idea here is that there are a good chance that you will invoke them time-to-time – as frequent as the customers will. Unless your test project context is about users who does a lot of errors, for example web-app with typical user a person who see this page for first time) – but you as a professional tester could invoke them on a purpose in this case.
The next set of features is horizontal ones. Horizontal features are those features that you could and will bypass using while preparing for other tests. You can’t invoke them occasionally by simply mistyping or typing specific values and. For example you’ve got two different login types – one requiring typing in user name and password and another reusing your windows credentials. If all testers use only one, which is more suitable, you will never test the second one. If the choices are limited and number of testers big enough you could ask each to use own configuration, but don’t abuse this idea. I’m afraid those are the features that you will need to deal with in systematic way – for example writing test cases. Before I go into details of writing test cases let me remind of the last set – diagonal features. Let me start with example this time - if your testers use type in password configuration and your test case is about single-login (reusing windows credentials) how about testing such issues as your username of password in windows don’t fit to application’s restrictions (for example too long password)? Diagonal features are those features that are vertical to horizontal ones, that’s why I call them diagonal. What I do is simple - when I write test cases for horizontal features I never write them too specific, so that tester who will run it have a chance to do something different and step into diagonal feature.
Planning tests? Actually snap-shooting the current vision
My testing is not 100% exploratory. Although specifications are almost always blur, design incomplete and outdated, there are a lot of sources for information gathering. You could read and try talking to anyone involved (managers, customers, developers) – if you don’t abuse their time and do your homework(reading) carefully they are going to cooperate. Or at least I’ve been lucky to have them most of the time.
This list is my test plan to be updated later. It does not describe what I will be testing, instead it describes what the tem believes at this moment need to be tested. This is a snapshot of this moment, at which development is not completed yet and things are going to change, but this is the moment when whole team is thinking about “the big picture” and not yet concerned with implementation details. That’s why this is the best time to write down the ideas and call this initial test plan. Not because we need a formal document to follow.
So I’ve typically been able to write down the initial list of horizontal features I want to test based on this conversation. The list is never correct or complete – I know I’m going to remove some impossible ones and add more that my explorations will illuminate. However I need the initial list for two purposes: to control my exploration and to make sure that my exploration has not failed to find significant items I’ve read/heard about.
Test Execution and documenting regression test cases
So test plan only document the horizontal out of 4 domains of tests. Guess what – the same does my test cases. They are actually update version of the initial snap-shot with the complete list of horizontal features – after coding and exploratory testing is completed. Frequently I don’t write any test case at all – simply update the test plan document with detailed list. Here I probably need to comment that my test plans and test cases are created as Lotus Notes documents with rather rich content – colors, fonts, tables, lists, etc. I may want to create test cases if the test plan becomes too large, however. By too large I mean either physically long or too long to execute – if you divide them into test cases you will could use your progress-reporting feature of your test case management database! That’s why I mentioned visibility in the beginning of this blog.
So even if I write test case the document contain the list of idea or the goals of test, not the exact steps to execute it. The test case may start with a configuration hits (I never call them setup steps). The typical test cases consist of a table with following columns: Feature/Idea to test; Expected results (generic, such as “should process this error gracefully”); actual results (if differ); data-attachments (reusable data set to import in order to boost the next execution); hints (screen-shots from the last execution) and comments (also references to documentation). Rows of the table list ALL actual horizontal features and even ideas that tester should not forget to test. Optionally there is a column that describes regression risk/execution priority (e.g. saying only test this during full functional testing).
This is reason also why I’ve given the name to my test case type a checklist.
Few Issues
If you think you like my idea a lot and you want try it out – here are some warnings.
Issue #1: gaining credibility of this approach. Your credibility once you take this approach will be the biggest issue. I earned credibility through doing excellent work instead of arguing my approach. I’m maybe still too young or something but I believe that job done is what I’m judged by not the credibility of how I did it. Managers don’t want to know why it was impossible to find all those defects – they are going to blame you anyway, so the best you could do to find them. Only if you can’t – you should think of a ways to protect yourself, but you can’t win a war if your only strategy is offence.
Issues #2 Requires stable team. Some people even in my company don’t like this approach, because I don’t describe exact test steps. If you give this test case to a different tester (not one who have written it) this will take a lot of time to execute it. Yes, they are right, just the same as you can’t expect to give a new developer in a team to read architecture and design documents and assume he immediately get up to speed. I’ve seen developers in a project team working for year and still supposed as a novice to only get simple tasks.