…If chance and risk are synonyms, then risk-based testing should not be too far from chance-based testing, should it? No I’m serious. Or do you think following quotes are from people who knows too little about test techniques?
“How can we maximize the chance of finding such a problem in the limited time we have to test?”
“I was lucky enough in my random selection of order for retesting the fixes that the system was in a state for me to get some simple extra tests in for the previous feature.”
Casinos makes money on
roulette because they knows statistics. Testing is not as random as dice roll, but we could make a few conclusions if we analyze statistics. One of the conclusions that I make is following: it makes a lot of sense to vary tests (minefield analogy) when we do it first few times. But as we do regression again and again the effect is lost.
I guess a lot when testing
I can’t say for everyone but for myself only. As a tester I actually asses risks thousands of times per day. When retesting a bug – what and if should I test around the bug? How much? I typically guess, based on experience, skills and knowledge of the product, of the developer who made the fix, etc. But I still guess.
Or another illustration. There is a well know
pairwise testing idea. Let me quote Michael “How can we maximize
the chance of finding such a problem in the limited time we have to test?”
Well, someone would ask – why don’t we automate all the cases? First of all if you have 16 fields with 5 options each you will have 150 billions of cases. Even automated test can’t test them all.
More over if one attribute hide, disable or change the validation of another attribute then even order in which they are changed are significant and it is hard to write universal validation.
That’s why I don’t like all-pairs testing, but use guess-based instead. For example I know that first 3 and last 4 values appears to be dependant groups and others seems to be standalone. But wait a bit – how do I know which are independent? I think I know. I take risk to guess that they are independent. There is a great chance that I am right. I think the chance that I am right depends on experience, skills and knowledge of the product. I prefer Exploratory Testing, because I have more knowledge of the product to guess.
So there is a chance to make a wrong guess. And either miss a bug or waste your time as a result. And if you waste your time you don’t have time to find more bugs, so you miss bugs eventually.
Other chances
There are chance to mistype or make any other mistake while testing. This is not bad actually as it really means a chance to extend test coverage. We should not be afraid of that. This is a chance to find more bugs.
There are chance to trigger not-repeatable bugs. I hope you know what I mean by
not repeatable bug and what it has to do with a chance. All bugs that first seems not-repeatable are actually found by chance, because the test that we executed may have passed without triggering the bug.
Have you studied statistics?
Let’s assume there is a bug on a dice. i.e. there is a crack on one of the six faces. When you roll a dice and the buggy face is goes bottom you don’t see the crack – you miss a bug. There is an approximately 17% chance that you will miss the bug. If you roll it twice the change is only 3% now.
Now, let’s assume there is so poor illumination that you need to have face up to notice a crack. One roll gives you 17% chance to discover the bug. Two rolls – 31%, three rolls 42%, 52%, 60%, 67%, … In increases slower and slower. The chance to miss the bug is 5/6 in power on n where n is number of rolls.
So how does statistics applies to testing?
disclaimer: All the consideration below is generic and only somewhat apply to practice. My point it to back-up my practices, not to calculate actual bug-finding metrics.
Let’s recall the first chance in testing I described above. I make a lot of guessing while testing features. If my guess is wrong I will miss a bug. Let’s assume for the matter of example that there is 17% chance for me to miss a bug as a result of wrong guess. If I could guess twice it would be only 3%... But if there is another tester – she could try to guess independently. That’s why I advocate any form of
pair-testing.
Now, let’s move to not-repeatable bugs. Let’s assume again for the matter of example that there is a 17% chance to occasionally trigger a bug while executing the designed test. If we would do one more independent execution the chance would go up to 31%, for one more it goes a bit more up. However if your testing is repeatable (scripted or even automated) the regression testing is no more independent and the chance don’t goes up too fast (or at all).
Conclusions
The chance for each individual roll, i.e. for each tester’s guess depends on the testers ability to guess right. It depends on skills, experience, knowledge of the product, context, etc. So improving and evaluating the ability is essential (I’ll blog about it sometimes soon).
However, for given test team, if you want to find as much bugs as possible you should allow tests to vary. But due to statistics it makes no sense to vary them too often.
If one tester tests the same features, if they follow scripts or even run automated tests, the chance that tests will vary small.
So the best way is to chose between manual and automated as well as between scripted and free-style (Exploratory or Ad-Hoc) testing. I would recommend to have both either automated or scripted-in-details tests on a project, but sometimes add a free-style testing into a cocktail.
