testingReflections.com - exploratory testing

Chance-based testing

Mon, 12 Jan 2009 15:59:34 +0000

…If chance and risk are synonyms, then risk-based testing should not be too far from chance-based testing, should it? No I’m serious. Or do you think following quotes are from people who knows too little about test techniques?

“How can we maximize the chance of finding such a problem in the limited time we have to test?”

“I was lucky enough in my random selection of order for retesting the fixes that the system was in a state for me to get some simple extra tests in for the previous feature.”

Casinos makes money on roulette because they knows statistics. Testing is not as random as dice roll, but we could make a few conclusions if we analyze statistics. One of the conclusions that I make is following: it makes a lot of sense to vary tests (minefield analogy) when we do it first few times. But as we do regression again and again the effect is lost.

Do you write test cases to kill time?

Mon, 08 Dec 2008 09:52:45 +0000

I wouldn’t believe myself until saw this really happening. Testers keep writing detailed test cases up-front with the primary goal to justify the time they spend early on the project. And right they are (are they?). At least they believe so. Managers see evidence that testers have red and understood all the requirements, covered them by test cases. Testers don’t have to think how to prove time spent on reviewing requirements even if they don’t find any issues there. And if developers delay a milestone we have more time to polish the test cases, do one more review or them, write more details, etc. Test lead doesn’t have to worry that on the next project testers will be added much later on the project schedule. Everything is fine but quality.

Could a novice tester do Exploratory Testing?

Tue, 04 Nov 2008 17:01:29 +0000

I think that the question is wrong itself. The question is what a novice „should”. I do believe that for most of the novices this is not the best way to learn testing art/skills, but it depends on a person. However! To learn testing by following scripts (I mean detailed instructions for manual testing) written by others is even worse, much worse!

ET thoughts: The Seeker (CKA) heuristic

Sun, 20 Apr 2008 08:18:28 +0100

Some people are more into mnemonics than others. I can recall walking along with James Lyndsay one day. This in itself was unusual because we are normally on opposite sides of the earth. We were discussing how there are many great mnemonics for test ideas, but neither of us was able to recall the items, just the mnemonics! I think this strongly influences our exploratory testing approaches. Alan Richardson has blogged about this here . Because it is general, It doesn’t really need a mnemonic, but I will give it an acronym. Seeker, CKA, Challenge Key Assumptions. While any tester can do this, the experienced tester will be more skilled at identifying key assumptions, and be more effective at using it. If you have less experience, or can remember mnemonics (!), maybe try the standard ones! You’ll find some links by searching for “test ideas” at my testingspot.net site

Discovery vs. Confirmation

Sat, 29 Mar 2008 06:44:58 +0000

When I hear debates about scripted vs. exploratory testing... or even debates such as "automate all tests" vs. "you can't automate all tests"... I don't think I'm hearing the real debate.

I think the debate that I'm hearing is "testing is about confirmation" vs. "testing is about discovery".

This distinction in the underlying intent of a given approach seems to not be emphasised in these discussions.

In simple terms, I think of the intent behind testing as having two facets:

Confirmation - Does the system do what we anticipated it should do?
Discovery - Does the system do (or not do) anything we did not anticipate it would (or should do)?

Ad-hoc vs. Exploratory Testing

Wed, 12 Mar 2008 12:52:28 +0000

In various discussions I've been having lately I've tried to help people distinguish between ad-hoc testing (or as some people say - having a play) and exploratory testing...

Ad-hoc testing in comparison to exploratory testing is like the difference between someone who only runs when they are late for their train and a professional athlete that competes in 100 metre sprints.

The person that runs for the bus simply runs instinctively - yet the professional athlete, with the sports-science consultant, studies ways to optimise their running, trains and practices to improve both their power and technique (such as with the Pose method).

The minute you even read about exploratory testing, you are stepping away from ad-hoc (having a play) testing. The mere fact that you appreciate that it isn't only instinctive but realise that there is a method to what may at a glance seem easy to do is the day you are no longer just "having a play".

Do you keep notes about “the first below the line” tests?

Tue, 12 Feb 2008 15:25:26 +0000

We can’t test everything. So it would be logical to divide tests into must and may (if time permit)” test. To have some tests reserved... Nevertheless, in continuous integration projects I have “it’s now or never” rule as testing new features. I may postpone whole feature testing, but specific test ideas – never. I’m not sure if I’m right – that’s why I’m writing this.

How do I know when I'm done... (again)

Wed, 06 Feb 2008 17:08:47 +0000

In the context of an XP-style Story/Acceptance Test Driven Development process, that uses exploratory testing to generation intra-iteration feedback... The question “how do we know we are done” is often asked and I'm not sure I always give or get the best answer. One of the problems is that there is so much context... so, here's one way you might know (based on my better experiences)...

That sweet check-in...

ET thoughts: Prediction and the all players all-in poker bug

Tue, 05 Feb 2008 03:27:34 +0000

One of the key aspects of exploratory testing is Prediction. By background research (specialist knowledge, interviewing others or looking at specifications, etc) or by observing software, we understand the models involved and can predict other behavior based on a model or model interactions. While some of this involves predicting positive behaviors, a skilled exploratory tester is often able to leverage failure models to find bugs. Once a bug is found, further investigation may lead to other variations of the bug often with a higher impact.

Test Case Review

Thu, 31 Jan 2008 17:03:15 +0000

For years I was wondering why we disagree with people about exploratory testing. Recently (working close with them) I’ve realized part of their thinking: To review test case you have to document it. You have to document it in advance to test execution; otherwise you will run unapproved test cases. You can’t review test cases in exploratory testing because there are no test cases written in advance to test execution.

Interview hint: be careful using term Exploratory Testing

Tue, 22 Jan 2008 11:42:48 +0000

Term Exploratory Testing (ET) is gaining popularity at least in my region. I welcome and promote the popularity. As a test manager I use the term to indicate a bigger story for my team, to hypnotize upper management and for other good reasons.
But there is a something where I avoid this term. I have quite an experience with tester hiring and performance reviews. I don’t ask anyone about ET – not any more. Well… unless interviewee uses this term first. In most cases that’s a terrible mistake.

The experience of an ET experience

Sun, 16 Dec 2007 05:16:20 +0000

A good training or conference session should be very informative, interesting, insightful and entertaining. A session can be made more appealing through the use of practical examples. I was privileged to get comments like these on feedback forms for my “Mission Possible: an exploratory testing experience” session at STARWest in California. It was an unusual session for me, no slides, just me testing small applications inviting the audience to provide test ideas while introducing various attacks and other heuristics Near the end of the session. I invited Michael Bolton to test some unseen applications at the end of the session, and he discovered some intersting bugs. I had minimal notes, and had just practiced with the software so I would be familiar with what I was testing. I wanted to test lots of simple easy to understand examples in an hour session.

Exploratory Testing Excercise - how I solved it...

Sat, 08 Dec 2007 22:14:37 +0000

In a previous post I presented a challenge as an exploratory testing exercise

why is it an exploratory testing exercise?

At first, when my friend posed the question, I was just intrigued and quickly obtained the answer so I could answer her question. Then I realised that what I was doing was applying some of the skills I use when performing inspective exploratory testing... (arguably, we do the same with prospective testing too).

Exploratory Testing Excercise

Sat, 08 Dec 2007 14:24:57 +0000

A friend sent me this game... and asked me, 'how does it know?'... As I solved it, I realised that the skills I applied were much the same as the skills I apply when exploratory testing...

So, play this game... and see if you can answer the same question... how does it know?

Try to solve it... then think about how you solved it...

The approach I took, is in the next post on my blog... don't read that post until you've tried the game and attempted to solve it yourself.

ET thoughts: Wolfgang Strigel and Werner Forßmann

Tue, 06 Nov 2007 23:17:56 +0000

Recently at STANZ 2007, Wolfgang Strigel spoke about a scientific study that contrasted exploratory testing against scripted testing. While many of us in the ET community already know how effective ET is, many traditionalists want to see studies of this type done. Wolfgang found that ET was 10 times as effective as scripted testing at finding bugs. I hope he publishes more details soon.