The Hoffman Heuristic: Why not try them all?

Mike: "Who uses this product?" James: "I do. I use it to train testers." I know it's a sample application. This tells me that there is no specific user I should be keeping in mind as I'm testing. This affirms in my mind that this is an analysis exercise. I think of James' SFDPO heuristic and ask more questions. Mike: "Tell me about the structure." James: "HTML and JavaScript." Not very helpful. I do a view source and look at the code. Very simple. I do notice that the display values for time are converted to integers for the error checking. Mike: "All it does is take the two inputs and performs error checking?" James: "Correct." Man of few words. Mike: "Where is the data stored?" James: "The data doesn't get stored anywhere." There is no real user, so I skip over operations. Mike: "Are you interested in platform based defects?" James: "I'm interested in seeing how you test it and what problems you find." I start with some basic quick tests. I change resolution and browser size, I notice that on low resolution the time dropdowns fall off the screen and I can't select all the values. I tell James. James: "Ok. What else?" He's looking for something deeper. The easy stuff over, I do my analysis of the functionality. There were two equivalence classes, AM and PM. There are also some interesting values to try: noon, midnight, and at least one half-hour value for each class. Error handling appears to fire when the start time is later in the day then the end time. I run a couple of tests. Once it's been 30 minutes, I don't want to spend more time on this problem. I tell James my results. Mike: "Well, it doesn't like it when I select midnight to end my work day. It thinks that's the start of the day. Other then that, everything appears to be fine. When I select a start time before my end time, it works. Oh, the error dialog doesn't have a caption. I might report that." James: "Are you sure everything appears to work? What have you tried?" I show him how I modeled the problem. I tell James my thoughts during my analysis and some of the tests I had executed based on my equivalence classes. James: "Are there any other tests you want to try? Why not try more?" Oh! James is looking for me to talk about the value of a test! He wants me to tell him that I no longer see value in testing and that I should now move on the next application. Mike: "I suppose I can, but I'm not sure there's value in it. At this point I'm relatively confident that the next test won't reveal new information. I should move on to the next assignment." James: "Really? Why?" Uncertainty... Mike: "Well, I've spent 30 minutes on it. I've completed the testing based on my analysis. I can keep testing, but why?" James: "Why didn't you just test all the values?" Surprise... Mike: "Why didn't I test all the values? There are two fields. Each field has 48 values. That's 48 x 48 tests! I'm not testing all the values, it's a waste of time." James: "Why didn't you just script testing all the values?" I murmur... Mike: "I didn't think about it." James: "Why?" Mike: "I don't know. I guess it seemed like too simple a problem. Based on everything we've covered up to this point I figured my analysis would be enough. Why waste time writing a script when I can test it manually faster." James: "Take a look at this..." At this point, James writes a quick Perl script. He copies the selection values from the HTML and uses a regular expression to read them in (so he doesn't need to waste time formatting the data), and he runs the test. Total time to write and execute the script, about 10 minutes. It turns out there is a defect I missed around a specific conversion in the JavaScript that shows up with a value I never tried because of my equivalence classes.

• Failures can lurk in incredibly obscure places
• You could need 4 billion computations to find 2 errors caused by a bug
• Even covering all of the input and result values isn't enough to be certain