Interview hint: be careful using term Exploratory Testing
Submitted by Ainars Galvans on Tue, 22/01/2008 - 12:42.
exploratory testing
Term Exploratory Testing (ET) is gaining popularity at least in my region. I welcome and promote the popularity. As a test manager I use the term to indicate a bigger story for my team, to hypnotize upper management and for other good reasons.
But there is a something where I avoid this term. I have quite an experience with tester hiring and performance reviews. I don’t ask anyone about ET – not any more. Well… unless interviewee uses this term first. In most cases that’s a terrible mistake.
A mistake: generic example
My official title is senior project manager, although I always think of myself as a test team lead . However I do participate hiring and performance review often (almost always for our company testers). This is a story which tends to repeat again and again:
• …
• Me: so tell me any of your advanced testing experience
• Tester: … and then I’m used to do an exploratory testing
• Me: so what does it mean to do ET?
• Tester: our specifications are not detailed enough, so we have to explore the software we are testing to learn how it works
• (ME thinking to myself) I wonder if he/she is talking about learning a product or testing it, but how could I ask to figure it out? Should I ask - what do you mean by “how it works”… I hate those terminology swamps!
Do you see the mistake? I’ve red a lot about Exploratory Testing, seen a presentation by Jonathan Kohl (at Oredev conference) and videos by Cem Kaner. I know what they mean by ET. But do you – an employee or candidate know what you think when you mention this term? Are you sure the boss in front of you think the same? Maybe you think it’s cool, but the boss equates ET to unskilled Ad Hoc testing that his 8 years old child is best at. Or you think it’s fresh, original, and innovative – then you need to know its 23 year old term now. I don’t recommend trying to hypnotize with terms someone who thinks to be more senior, more experienced than you are.
Exploring: not only functionality but also its quality
What’s your first impression about a term exploratory testing? My first association for explorer comes from strategy games with so called fog of war. When you start game you don’t know the map of “the world” so you train/buy/purchase units to explore terrain around you. This is very bad association if I want to describe ET however. I’m not sure how archeologists are working, but I bet it would make a better example. I think that they have techniques to guess places to dig and their techniques are based on their findings are they progress.
We’ve explored numerous test automation tools, source control systems, defect reporting systems with goal to understand if they fit our purpose. That was exploratory learning process. But we never really tested it – we were not asking it’s functional quality yet. Or perhaps we only did happy-path testing: no error handling, no special cases, etc. At least for me testing is something more. So what’s testing and what’s ET for me?
Context-driven terminology
I don’t like arguing over definitions. For example, I don’t care how you define term walking. It is a way too hard to. For example if you say it is a way to get from point A to point B using your feet, then there are following issues: a) it also covers running, which it should not b) it does not include walking round with my 2 moth old child in my arms to get him asleep, which it should, or maybe it shouldn’t? How about walking up and down stairs? Does your definition apply to walking on a moon with so little gravity and do you care at all weather you include it?
I’ve seen people frequently missing context during a job interviews. If a job description includes prerequisite “used to walking” and you believe to fulfill it – be ready to clarify your experience. Person A used to walk to bathroom every day may claim he is practicing walking on regular basis. Person B may be found of Beachcombing. Neither of them may fit to my team designed to (with a goal to loose weight) have 1 hour walking exercise in a fitness center. Both would probably drop the second day- first because of his legs aching the second – because of boredom/time wasting.
As a boss, I don’t care how well you know term definitions or theory. If you are in university and a professor asks you to what’s ET he wants to see if you have learned theory. As a boss I don’t care. I want to know if I could assign you to do ET. For example - will you enjoy it to do it good enough? Because when I compare ET to scripted testing I compare motivation and discipline, empowerment and control, commitment and order. I need some disciplined testers on my team to do some formal stuff; just the same way as I need motivated to find more bugs faster. That’s difference between two testing approaches in my context.
Summary
Our industry can’t agree on terms – everyone use it their own way. Exploratory testing is only an example. I have seen even seasoned testers ignoring to study best ideas documented about Exploratory Testing. I don’t blame them. I don’t blame youngsters who believe to be in love. What’s love for me is not the same thing as it was not the same thing for me 10 years ago.
What’s love anyway? Does it matter? To whom does it matter? I love my wife and I love my children, I’m not sure if this is the same love. I don’t care if you understand what exactly I feel about them… I really don’t.
This blog is only a warning message.
further conversation
I will soon post a blog about test case review (it is still in review itself). I recently realized that I my own understanding of exploratory and scripted testing difference is closly related to - how does test design and (both test and requirement) review interacts, not how test design and execution (+learning +...) interacts.
But there is a something where I avoid this term. I have quite an experience with tester hiring and performance reviews. I don’t ask anyone about ET – not any more. Well… unless interviewee uses this term first. In most cases that’s a terrible mistake.
A mistake: generic example
My official title is senior project manager, although I always think of myself as a test team lead . However I do participate hiring and performance review often (almost always for our company testers). This is a story which tends to repeat again and again:
• …
• Me: so tell me any of your advanced testing experience
• Tester: … and then I’m used to do an exploratory testing
• Me: so what does it mean to do ET?
• Tester: our specifications are not detailed enough, so we have to explore the software we are testing to learn how it works
• (ME thinking to myself) I wonder if he/she is talking about learning a product or testing it, but how could I ask to figure it out? Should I ask - what do you mean by “how it works”… I hate those terminology swamps!
Do you see the mistake? I’ve red a lot about Exploratory Testing, seen a presentation by Jonathan Kohl (at Oredev conference) and videos by Cem Kaner. I know what they mean by ET. But do you – an employee or candidate know what you think when you mention this term? Are you sure the boss in front of you think the same? Maybe you think it’s cool, but the boss equates ET to unskilled Ad Hoc testing that his 8 years old child is best at. Or you think it’s fresh, original, and innovative – then you need to know its 23 year old term now. I don’t recommend trying to hypnotize with terms someone who thinks to be more senior, more experienced than you are.
Exploring: not only functionality but also its quality
What’s your first impression about a term exploratory testing? My first association for explorer comes from strategy games with so called fog of war. When you start game you don’t know the map of “the world” so you train/buy/purchase units to explore terrain around you. This is very bad association if I want to describe ET however. I’m not sure how archeologists are working, but I bet it would make a better example. I think that they have techniques to guess places to dig and their techniques are based on their findings are they progress.
We’ve explored numerous test automation tools, source control systems, defect reporting systems with goal to understand if they fit our purpose. That was exploratory learning process. But we never really tested it – we were not asking it’s functional quality yet. Or perhaps we only did happy-path testing: no error handling, no special cases, etc. At least for me testing is something more. So what’s testing and what’s ET for me?
Context-driven terminology
I don’t like arguing over definitions. For example, I don’t care how you define term walking. It is a way too hard to. For example if you say it is a way to get from point A to point B using your feet, then there are following issues: a) it also covers running, which it should not b) it does not include walking round with my 2 moth old child in my arms to get him asleep, which it should, or maybe it shouldn’t? How about walking up and down stairs? Does your definition apply to walking on a moon with so little gravity and do you care at all weather you include it?
I’ve seen people frequently missing context during a job interviews. If a job description includes prerequisite “used to walking” and you believe to fulfill it – be ready to clarify your experience. Person A used to walk to bathroom every day may claim he is practicing walking on regular basis. Person B may be found of Beachcombing. Neither of them may fit to my team designed to (with a goal to loose weight) have 1 hour walking exercise in a fitness center. Both would probably drop the second day- first because of his legs aching the second – because of boredom/time wasting.
As a boss, I don’t care how well you know term definitions or theory. If you are in university and a professor asks you to what’s ET he wants to see if you have learned theory. As a boss I don’t care. I want to know if I could assign you to do ET. For example - will you enjoy it to do it good enough? Because when I compare ET to scripted testing I compare motivation and discipline, empowerment and control, commitment and order. I need some disciplined testers on my team to do some formal stuff; just the same way as I need motivated to find more bugs faster. That’s difference between two testing approaches in my context.
Summary
Our industry can’t agree on terms – everyone use it their own way. Exploratory testing is only an example. I have seen even seasoned testers ignoring to study best ideas documented about Exploratory Testing. I don’t blame them. I don’t blame youngsters who believe to be in love. What’s love for me is not the same thing as it was not the same thing for me 10 years ago.
What’s love anyway? Does it matter? To whom does it matter? I love my wife and I love my children, I’m not sure if this is the same love. I don’t care if you understand what exactly I feel about them… I really don’t.
This blog is only a warning message.
further conversation
I will soon post a blog about test case review (it is still in review itself). I recently realized that I my own understanding of exploratory and scripted testing difference is closly related to - how does test design and (both test and requirement) review interacts, not how test design and execution (+learning +...) interacts.