Skip navigation.
Home » blogs » Antony Marcano's blog

Discovery vs. Confirmation

Submitted by Antony Marcano on Sat, 29/03/2008 - 07:44. exploratory testing | perspectives | test driven development

When I hear debates about scripted vs. exploratory testing... or even debates such as "automate all tests" vs. "you can't automate all tests"... I don't think I'm hearing the real debate.

I think the debate that I'm hearing is "testing is about confirmation" vs. "testing is about discovery".

This distinction in the underlying intent of a given approach seems to not be emphasised in these discussions.

In simple terms, I think of the intent behind testing as having two facets:

  • Confirmation - Does the system do what we anticipated it should do?
  • Discovery - Does the system do (or not do) anything we did not anticipate it would (or should do)?

Confirmation

The goal of scripted tests that are written for repeatability (be they manual or automated) is confirmation.

We are trying to confirm that the system does what we have anticipated it should do. The important word there is 'anticipate', because we have to anticipate a test for it to be scripted or automated. How we anticipate the test is another issue and will vary depending on what process you are using.

Indeed, we may discover where the system doesn't do what we expect it to (e.g. during regression tests) but the emphasis in such tests is on confirmation.

Discovery

Inspective exploratory testing (and ad-hoc testing) where we are not robotically executing pre-defined tests but designing the test in the instant before we execute it - emphasises discovery over confirmation.

We are trying to discover where the system does things we didn't anticipate that it would do - and find things that the system doesn't do (or doesn't do very well). In essence we are seeking tests-worth-repeating - such as tests that find bugs.

Indeed, in the course of performing exploratory testing we might construct an hypothesis and perform tests that are designed to confirm that hypothesis but nonetheless the over-arching intent is discovery.

How does this manifest itself in practice?

Let me explain how this relates to areas of interest to me...

In Test-Driven Development (TDD) and Acceptance Test Driven Development (ATDD) - whilst we are deciding what test is next and indeed while we are trying to write it, we are trying to 'discover' what the item under development should do. Once the test is written we are saying that is what we currently believe the system should do and the automated test's purpose is then to 'confirm' when the item under development actually does it. This automated test then continues to confirm that the system actually does it.

During an iteration, all the tests we anticipate may then be running and passing... now it's time to 'discover' the tests we didn't anticipate. I would employ exploratory testing for this purpose. As we attempt test after test (that we'd previously not anticipated) - we might find a bug.... or maybe just a 'gap in our original thinking'. That is now a test-worth-repeating. This might be captured in a bug-tracking system or, for me, more often captured as an automated test (probably done after the exploratory testing session). This test might be best applied as a unit test or as an acceptance test... it tends to depend (what it depends on is beyond the scope of this blog post).

Some closing thoughts

Some processes don't try to anticipate tests and rely more on exploratory testing (i.e. discovery-oriented). Some processes do try to anticipate tests but not in a way that is practical (resulting in scripted or automated tests being deprecated) - again, relying more on exploratory testing. In such cases, we are simply relying on discovery-oriented-testing to drive the identification of confirmation-oriented-tests. Confirmation-oriented tests might be manual (in a test-script or bug-report) or automated... I believe test-driven methods allow a balanced application of both discovery and confirmatory testing.

Next time you find yourself in a debate about scripted vs. exploratory or automate-everything vs. automate-some-tests stop and think for a moment - is that what we are really debating here? You may just have two very different perspectives on the intent of the testing you are most concerned with... it might be that you are actually debating confirmation vs. discovery when perhaps your conversation would better serve your project if you discussed how to exploit both confirmation and discovery oriented approaches to testing.

» Antony Marcano's blog | login or register to post comments

Familiar Discussion...

Submitted by sbarber on Wed, 23/04/2008 - 22:59.
Antony/James,

I have had many discussions that feel (to me) very similar to this one related to the an article I wrote a few years back about performance testing; Investigation vs. Validation.

One of the key points I was trying to make then (which, I think, is part of Antony's point) was that sometimes I'm given a mission to determine whether or not the system does some very specific thing under some very specific condition. Most often, this is to determine whether or not someone is in compliance with some legal contract that I may or may not ever get to see.

I have heard compelling arguments that this activity (the one that I tend to refer to as "Validation" and how I interpret Antony's use of the term "Confirmation") isn't actually "testing". While I don't necessarily disagree with that argument, it's still an activity that gets assigned to testers -- occasionally for a completely reasonable and appropriate reason.

Maybe I'm hearing a similar discussion because I'm missing something and/or I'm biased toward hearing a discussion I'm used to having. Just figured I'd jot down my thoughts in case they were useful.

--
Scott Barber
CTO, PerfTestPlus
Executive Director, AST
Co-Author, Performance Testing Guidance for Web Applications
» login or register to post comments

Ok, but at a higher purpose than the level I'm talking about...

Submitted by Antony Marcano on Tue, 08/04/2008 - 20:30.
I do appreciate this discussion as it will help me refine how I explain this to others in the future... if only to make sure that the person isn't distracted from my point by a philosophical debate around the purpose of testing.

I'm trying to differentiate the intent behind certain tests at a basic level and do so in an accessible form to a wider audience...

A test resulting from a test-first approach has many purposes. One of them is to confirm when the system satisfies a concrete case (or example of usage) that we've anticipated.

At a similarly basic level, a test performed during exploratory testing is trying to help us discover new things because we are often performing tests we hadn't previously anticipated.

Step back from that and I accept that a higher purpose of a test is to add to the body of evidence that supports a theory (or matures an hypothesis into a theory)... But this aspect of the discussion is way beyond the purpose of the original post.

So, if it helps to not distract from the intent of my previous comment, I can elaborate on the statement you called into question as follows:

"I'm not trying to discuss how a test supports a theory with evidence but instead to illustrate that at a more basic level there are tests that help us confirm something - that the system can cope with a given set of concrete cases".

So, yes - I agree that in both cases a higher intent is on supporting a theory but at a more basic level one can contrast tests between each other as having greater emphasis on confirmation in one context and more on discovery in another.

Bear in mind that I'm discussing these contrasting characteristics in the context of a team employing Acceptance Test Driven Development...

I've been using this Confirmation/Discovery contrast to good effect with teams learning how to use exploratory testing to discover new tests that are then used to drive the implementation of new capabilities in future iterations.



Antony Marcano
» login or register to post comments

Yes, you are trying to confirm a theory.

Submitted by jamesbach on Fri, 04/04/2008 - 10:34.
"I'm not trying to confirm a theory but instead a concrete case"

Think one step farther, Anthony: WHY are you trying to see the result of your concrete case? What possible relevance could your concrete case have to anything? The answer is you are collecting evidence about something. That something is your theory of the status of your software. There's no other answer. You aren't composing poetry. It's not a religious ritual. It's not an aesthetic issue. It's not performance art.

All I have to do is ask "what do the test results mean?" and you will give me an answer that discloses your theory.

You believe, when you see your concrete cases do their little dance, that this indicates something about the health of the software even for cases you have not seen. It's called inductive reasoning: from the specific, you leap to the general.

Even if you have no interest in cases beyond the ones you have tried, you are still collecting evidence regarding a theory, because even for those specific concrete cases, there are a host of assumptions you make. Perhaps the cases are miscoded so that they always pass no matter what the software does. Perhaps the cases will always pass on your machine but not on anyone else's. Perhaps the tests would fail between midnight and 4am.

Ultimately, when you run tests, it's only a theory that you have actually run them! Sound silly to you? Tell that to the Apple CFront Compiler team, who thought they were running the math coprocessor code generation tests a whole year before realizing they were, in fact, commented out. (This happened in 1989, at Apple. We had a big laugh about it.)

Running the tests does not confirm anything, in any hard sense. It provides evidence that is subject to interpretation. A tester must understand that he is in the evidence and inference business, not the proof/confirmation business.

I agree that, in many cases, the evidence you collect from your concrete cases is strong. But in all cases it is just evidence, and it is not logically possible (David Hume proved this in the eighteenth century) for any evidence to exclude all possible theories except one.

So, there's no such thing as confirmation, except in the loose and non-logical sense of "providing strong evidence."

-- james
» login or register to post comments

Two different, but complementary perspectives

Submitted by Antony Marcano on Sun, 30/03/2008 - 01:48.
Hi James,

I think you are focusing on a single purpose of testing and I think that the purpose you describe is what I've labelled as 'discovery'. This is especially in the case of a iterative and incremental, test-driven development methodology (i.e. XP).

The point of the post is that within testing there are different, complementary purposes: confirmation and discovery.

I say "confirmation" where the emphasis is on the test confirming that given input x I get output y. I'm not trying to confirm a theory but instead a concrete case - such as with an automated test. In the case of automated tests I want them to inexpensively confirm that the concrete cases continue to be satisfied.

The corroboratory tests that you refer to are more relevant to the motive I've labelled 'discovery'. In the context of a highly iterative test-driven methodology, discovery is the intent behind inspective exploratory testing... discovery of missing behaviours or misbehaviours - i.e. discovery of tests we (as a project team) didn't anticipate. These tests-we-didn't-anticipate are fed back into the test-driven aspect of the process.

Does that make it clearer?

Antony

Antony Marcano
» login or register to post comments

The Essential Asymmetry

Submitted by jamesbach on Sat, 29/03/2008 - 10:07.
As Karl Popper is famous for pointing out, there's an asymmetry here. It is possible to refute a theory, but not confirm it. Confirmation requires an infinite number of tests, disconfirmation requires only one.

That means "confirmatory" tests do not confirm that the product does what you expected it to. They, at MOST, show that the product CAN do what you expect it to.

A better word than confirm is "corroborate": tests that allow the product to show what it can do provide evidence that corroborates the theory that the product works. But that theory can never be confirmed.

That's why I say that the aim of testing is to try the product, sufficiently, to show that it can work, AND show how it might not work.

We can show CAN, but what we really want to know is WILL it work? Confirmatory-style tests generally provide weak evidence for whether the product WILL work. This is because they are usually too static and stale (exploratory is fresh by design), and usually too simple (owing to the desire to have them easily repeated and automated).
» login or register to post comments

Comment viewing options

Select your preferred way to display the comments and click 'Save settings' to activate your changes.