What's special about a good Exploratory Tester?

Hi Julian. I'll answer you on my blog.

Thank you to both Mike and James for your helpful and insightful replies.

I'll read some of the material you recommend (and I have the entire set of STQE / Better Software so I can find James' article there :-), sadly I'm unlikely to have time to read all the references at the moment...

Mike mentioned traditional test techniques being used as tools. At the risk of sounds stupid, or stating the bleeding obvious, I have some questions for you both (and anyone else who'd like to comment).

- Do you expect an Exploratory Tester to be well versed in these techniques? Do you check that they are competent in them, etc?

- If a tester is weak in the area of traditional test techniques do you train them, and/or refer them to suitable references, etc?


Thanks again, I may have more questions later as I learn more about the subject.

Exploratory testing has been studied for years. The only problem is that people in the software testing field haven't read about any of those studies. For a shining example, I refer you to the ENTIRE BODY OF WORK of Nobel prize winner Herbert Simon. His book The Sciences of the Artificial is more or less the scientific basis for my testing methodology. I would also urge you to read, carefully and cover-to-cover, Jerry Weinberg's book An Introduction to General Systems Thinking. If you read those books, I believe you will have a much better basis to think about and experiment with the exploratory approach to software testing.

There exists a whole world of work-- mostly in psychology and philosophy and hardly at all in computer science-- that has been done on the subject of exploratory approaches to intellectual work.

To answer one of your questions: "How do you tell if someone is doing exploratory testing?" Here's how. If a tester is doing testing, and the testing they are doing is based on ideas that they get while doing the testing, then that is exploratory testing. If the testing is based only on ideas that preceded that test session, then it isn't exploratory, it's scripted testing. Another way of saying this is that exploratory testing is simultaneous learning, test design, and test execution.

Exploratory and scripted testing exist on a continuum. Almost all of our testing is partly exploratory and partly scripted. But until about fifteen years ago, there was virtually no discussion, lore, or training about exploatory approaches. Instead, we were all supposed to pretend that our test designs are better when they are created before learning and without experimenting as we go.

There's no scientific or logical justification for the idea that extensively scripted testing is better than unscripted testing. At least, there has been no justification since about the turn of the twentieth century and the rise of psychologically sophisticated theories of learning and of doing science. But it's an easy medicine to prescribe, so consultants continue to recommend it.

A few of us, including my brother Jonathan, Cem Kaner, Michael Bolton, and I, have been working specifically on the problem of how to coach and assess testers. We've done a fair amount of this. Cem has done the most.

What we do is basically pose real testing problems to testers and then observe how they cope with them. What we look for is based on our model of what testing is. I use several models. The main one is the Heuristic Test Strategy model, which you can see at my website. Another more general model is the five-fold system, published in Lessons Learned in Software Testing.

By doing this with a variety of testing problems, we can easily tell who can do exploratory testing effectively.

[textile] James talks about this in his STQE article "Inside the Mind of an Exploratory Tester" - Nov/Dec 2003 (Vol. 5, Issue 6). I don't know of an online version, but I'm happy to fax you a copy if you like. I also recommend "Exploratory Testing Explained" (http://www.satisfice.com/articles/et-article.pdf) if you have not read that. I tend to find cognitive skills to be the most important. Excellent cognitive skills allow a tester to process the sensory information we collect and to recall the information to develop more complex and meaningful tests. Our cognitive skills are our:

• ability to analyze
• ability to evaluate
• ability to retain information
• ability to recall experiences
• ability to make comparisons
• ability to determine action

Although cognitive skills have an innate component, the bulk of cognitive skills are learned. Like sensory and motor skills, cognitive skills can be practiced and improved with the right training. "Inside the Mind of an Exploratory Tester" covers some of these skills. It's these skills, not "traditional testing skills" (or test techniques rather) that make an exploratory tester good. Those techniques are simply tools that we use while applying our cognitive abilities. I would think the next biggest factor is intuition. This is where experience comes in. Like chess masters who use intuition (experience and a database of memorized game scenarios) to determine moves in a complex environment, exploratory testers us intuition as they apply their skills. Their intuition guides them in terms of what they learn about the software and the problem, and in terms of what tests they design and execute. That's my initial response, but there are people who can probably give you a better answer. Great first post! -Mike [/textile]