Story: Choose Ad Hoc over formal regression testing
Submitted by Ainars Galvans on Thu, 12/04/2007 - 14:48.
test management
When someone use term Ad Hoc testing it mostly mean lack of formalities: no formal process, no formal reporting, no formal test documentation, etc. However I want to share you the story where we’ve got all the formalities and test tools like test case management, but still reserved time for the Ad Hoc testing process.
What’s the most interesting we did that at the very end of the project – testing the final build – the build to be shipped to all customers. And that was the only tests we did and never regret that! And we still use this practice.
Context: reducing regression testing time
I’ve red a lot about statistical ways to predict which test need to be run in each regression test cycle. I’ve red about risk evaluation meetings that size test effort to be put in each component. No – my story is a different one. I use something that you may call intuition or something that I’m going to call “informed guess” in my further blogs, but let me tell you the story.
Few years ago I was managing a large test project. We’ve got quite detailed tests cases, grouped and indexed quite well. Content was good – sometimes support even used our test cases instead of product documentation. The size - hundreds of test cases overall requiring some 500-1000 person-days to run for a single test environment, but as you may guess we’ve got multiple environments. Not enough invested in test automation (there serious reasons for that) so test team lead did a fantastic work of risk-based test planning and testers of a fast test execution and spend less than 1500 person-days per release for all (numerous) cycles of regression tests, achieving reasonable environment coverage. (this is by the way why I laugh at automation ROI calculations who would say that I need about 3(=number of test cycles)*20(number of unique configurations)*1000 person-days to do testing. You could calculate ROI for test team lead... :)
The stage “Forget the test cases, do it Ad-Hoc now!”
Selecting the right test cases for regression tests is not what I want to share here. You know what – even with all the team leads’ job done the last few days before release was the most critical. And you know what we did – the last regression test cycle was Ad-Hoc regression testing. The team of about 10 experienced testers did that within a day (+/- depending on change scope since last formal regression test cycle). Isn’t it incredible? We delegated risk estimation to each tester. And they never failed. Yes – as a test manager I was never told of any regression problem implemented since last formal regression tests cycle and slipped through Ad Hoc regression test cycle. Incredible indeed. We still use the same approach in other projects and the small it is the better that works. There are two limitations however: you need a stable test team, each tester did each test case at least several times, you need tester to know all the changes done since the last formal regression test cycle in component he is testing and you need those changes to be a few.
I was thinking of how this is possible and why those limitations applies and I believe I know the answer now, but it is another blog I’m going to publish soon (still discussing it with some nice people and special thanks to James Bach who triggered my attention in that direction).
What’s the most interesting we did that at the very end of the project – testing the final build – the build to be shipped to all customers. And that was the only tests we did and never regret that! And we still use this practice.
Context: reducing regression testing time
I’ve red a lot about statistical ways to predict which test need to be run in each regression test cycle. I’ve red about risk evaluation meetings that size test effort to be put in each component. No – my story is a different one. I use something that you may call intuition or something that I’m going to call “informed guess” in my further blogs, but let me tell you the story.
Few years ago I was managing a large test project. We’ve got quite detailed tests cases, grouped and indexed quite well. Content was good – sometimes support even used our test cases instead of product documentation. The size - hundreds of test cases overall requiring some 500-1000 person-days to run for a single test environment, but as you may guess we’ve got multiple environments. Not enough invested in test automation (there serious reasons for that) so test team lead did a fantastic work of risk-based test planning and testers of a fast test execution and spend less than 1500 person-days per release for all (numerous) cycles of regression tests, achieving reasonable environment coverage. (this is by the way why I laugh at automation ROI calculations who would say that I need about 3(=number of test cycles)*20(number of unique configurations)*1000 person-days to do testing. You could calculate ROI for test team lead... :)
The stage “Forget the test cases, do it Ad-Hoc now!”
Selecting the right test cases for regression tests is not what I want to share here. You know what – even with all the team leads’ job done the last few days before release was the most critical. And you know what we did – the last regression test cycle was Ad-Hoc regression testing. The team of about 10 experienced testers did that within a day (+/- depending on change scope since last formal regression test cycle). Isn’t it incredible? We delegated risk estimation to each tester. And they never failed. Yes – as a test manager I was never told of any regression problem implemented since last formal regression tests cycle and slipped through Ad Hoc regression test cycle. Incredible indeed. We still use the same approach in other projects and the small it is the better that works. There are two limitations however: you need a stable test team, each tester did each test case at least several times, you need tester to know all the changes done since the last formal regression test cycle in component he is testing and you need those changes to be a few.
I was thinking of how this is possible and why those limitations applies and I believe I know the answer now, but it is another blog I’m going to publish soon (still discussing it with some nice people and special thanks to James Bach who triggered my attention in that direction).