BBST Practical Lessons
Submitted by John McConda on Sat, 10/01/2009 - 06:27.
The Black Box Software Testing series offered for free to AST members is starting up again, and I’m excited to be a student and volunteer instructor. Developed by Cem Kaner, Rebecca Fiedler, and James Bach, these classes offer university-grade course material in a unique online format.
The most important measure for me when I take part in testing training is how many lessons I can actually use in my daily job. The Foundations course has provided me many of these, a few of which I’d like to share in this and subsequent posts.
The lesson I’ve found myself using time and time again this past year is the impossibility of complete testing. There is a myth that pervades most of the contexts I’ve been a part of, and that is the idea that somehow, we can be “safe”, and just “test everything”. “Sure, it will be a sacrifice and take many hours, but we just can’t afford to take any chances.” Any tester with any experience at all will tell you that this is dangerous thinking. First and foremost, it gives the impression to the rest of the team and management that there is a known quantity of testing that can be done to ensure a zero or low defect rate. That means even if the team does run “all the tests”, if something slips past production, the blame still falls on the test team who obviously didn’t run all the right tests or this wouldn’t have happened.
The implication of this principle, and the idea I find myself using the most, is opportunity cost. You want us to run a manual regression suite that takes 4 weeks because of a browser patch? Sure, what tests that we were planning for our new release would you like us to cut out to make time for that?
Unlike developers, testers are not building a product that can be quantified as complete, because there is no complete testing. There is only the amount of testing that we can finish in the time period we are given. That means testers and those who manage them must constantly make tough decisions about which activities are most important to the mission, and universal rules rarely apply. Every task takes time away from another task that may or may not be more important to our mission, so we’d better take a hard look at every test run, every document created, every automated test maintained, every meeting attended, etc., and decide if it‘s worth the opportunity cost of everything else we could be doing.
The most important measure for me when I take part in testing training is how many lessons I can actually use in my daily job. The Foundations course has provided me many of these, a few of which I’d like to share in this and subsequent posts.
The lesson I’ve found myself using time and time again this past year is the impossibility of complete testing. There is a myth that pervades most of the contexts I’ve been a part of, and that is the idea that somehow, we can be “safe”, and just “test everything”. “Sure, it will be a sacrifice and take many hours, but we just can’t afford to take any chances.” Any tester with any experience at all will tell you that this is dangerous thinking. First and foremost, it gives the impression to the rest of the team and management that there is a known quantity of testing that can be done to ensure a zero or low defect rate. That means even if the team does run “all the tests”, if something slips past production, the blame still falls on the test team who obviously didn’t run all the right tests or this wouldn’t have happened.
The implication of this principle, and the idea I find myself using the most, is opportunity cost. You want us to run a manual regression suite that takes 4 weeks because of a browser patch? Sure, what tests that we were planning for our new release would you like us to cut out to make time for that?
Unlike developers, testers are not building a product that can be quantified as complete, because there is no complete testing. There is only the amount of testing that we can finish in the time period we are given. That means testers and those who manage them must constantly make tough decisions about which activities are most important to the mission, and universal rules rarely apply. Every task takes time away from another task that may or may not be more important to our mission, so we’d better take a hard look at every test run, every document created, every automated test maintained, every meeting attended, etc., and decide if it‘s worth the opportunity cost of everything else we could be doing.