How do I know when I'm done... (again)
Submitted by Antony Marcano on Wed, 06/02/2008 - 18:08.
exploratory testing | extreme programming (XP)
In the context of an XP-style Story/Acceptance Test Driven Development process, that uses exploratory testing to generation intra-iteration feedback... The question “how do we know we are done” is often asked and I'm not sure I always give or get the best answer. One of the problems is that there is so much context... so, here's one way you might know (based on my better experiences)...
That sweet check-in...
The developer checks in the latest changes to the code to get the first of the acceptance tests (ATs) for a story passing. She updates the test as passing (In Fitnesse I have a RunningTestedFeatures page to which I add a symbolic link to the passing test page) and thus it gets included in the 'Acceptance Test' phase of the build pipeline (this assumes a multi-phase build pipeline).
It's good to talk
The developer and tester have a conversation to decide whether it's worth doing any exploratory testing around that yet... Often that first test on a story for a brand new feature is so basic that there's often little point. By the second or third passing ATs, there is usually much more to be gained from exploratory testing...
Exploratory testing ensues...
In conjunction with the team the tester will have determined charters that focus the scope of these testing sessions (see session based testing (PDF)) and is captured as task cards on the team's board. Many would have been captured during planning and will form part of the collection of tasks associated with that story on your project wall. Bear in mind that other charters may arise as an outcome of exploratory testing.
These charters are there to focus attention on the areas of greatest concern, whilst avoiding areas where there's little value doing exploratory testing... There's little point testing around capabilities characterised by yet-to-pass acceptance tests... Why? – well the capability characterised by those tests will fail because the feature hasn't been implemented yet (surprising how often I have to clarify this).
Tests that are worth repeating...
During these sessions, the outcome will be that the tester(s):
-Don't discover any more tests worth repeating (maybe because they are adequately characterised by the existing ATs)
-Do discover some tests worth repeating against a future build (in this context, worth creating an automated AT for)
This story or another story?
Of the tests that are worth repeating, the tester(s) will have discovered scenarios that:
(a)Illustrate a specific stimulus within the scope of the behaviours characterised by the existing ATs but causing an undesirable outcome (e.g. a specific value that causes a failure despite appearing to be equivalent to the values used in the existing ATs)
(b)Illustrate a new behaviour enabled by the current implementation and related to the story but extending the capabilities beyond the spirit of the existing ATs (the tester should have a good idea about this if they were involved in the original discussion about the ATs – which of course they should be).
These tests all describe undesirable behaviours rather than your typical pre-determined AT which often characterised an 'absent behaviour'.
In short - they've found some bugs :-)
If it isn't clear which category the ATs (representing these bugs) fall into, I find a conversation with the product owner is useful.
Items in (b) are summarised by an evolution of the story (in the form of a new story), elaborated with the tests and added to the backlog for customer prioritisation and inclusion in a future iteration.
Prioritisation, prioritisation, prioritisation...
Items in (a) will have new automated ATs created and must be passing before the story is considered 'DONE'. If there are more of these than the team can implement in the iteration as well as their other committed work then these ATs and the other stories in the iteration will need to be prioritised...
Sometimes this results in some of these new tests being bundled under a new story as in category (b) above. Sometimes this results in yet-to-pass pre-determined ATs for that story being de-scoped and treated as in category (b). Sometimes a different story, also scheduled for that iteration, is deferred to a later iteration. It's always a judgement call...
I always ask a question like - "Let's say the current capabilities represented all the necessary scope to go live, would you go live given those tests that fail with undesirable behaviour (as opposed to merely absent behaviour)?"
This cycle continues throughout the iteration, adding new charters for new testing sessions as they become apparent, until all Acceptance Tests for the Story are passing and there are no new bug-inspired ATs in category (a) being found OR those being found (or those remaining) category (a) ATs are those that the Product Owner decides don't matter...
Let's not forget those charters... if you still have testing session cards left on the project wall, you'll want to get those finished too. Unless the team and product owner decide that not doing them is a risk worth taking.
Then , and only then, is the story 'DONE'.
Intra-iteration is the key phrase...
All this really needs to happen during an iteration... If you run out of time before you are 'DONE' then something hasn't worked. If you are doing your first exploratory testing for a story, an iteration or more after it was considered 'DONE' then it wasn’t really 'DONE'...
If, in this sense, you never really get things 'DONE', it's often caused by over-optimistic planning or insufficient ruthlessness in your prioritisation... I find that the main cause is over-optimistic planning - because people always want things to be done as soon as possible. What the cause will be for you is something for you to discuss in your end-of-iteration retrospective... You do have end-of-iteration retrospectives, right? Of course you do, you'd be crazy not to.
That sweet check-in...
The developer checks in the latest changes to the code to get the first of the acceptance tests (ATs) for a story passing. She updates the test as passing (In Fitnesse I have a RunningTestedFeatures page to which I add a symbolic link to the passing test page) and thus it gets included in the 'Acceptance Test' phase of the build pipeline (this assumes a multi-phase build pipeline).
It's good to talk
The developer and tester have a conversation to decide whether it's worth doing any exploratory testing around that yet... Often that first test on a story for a brand new feature is so basic that there's often little point. By the second or third passing ATs, there is usually much more to be gained from exploratory testing...
Exploratory testing ensues...
In conjunction with the team the tester will have determined charters that focus the scope of these testing sessions (see session based testing (PDF)) and is captured as task cards on the team's board. Many would have been captured during planning and will form part of the collection of tasks associated with that story on your project wall. Bear in mind that other charters may arise as an outcome of exploratory testing.
These charters are there to focus attention on the areas of greatest concern, whilst avoiding areas where there's little value doing exploratory testing... There's little point testing around capabilities characterised by yet-to-pass acceptance tests... Why? – well the capability characterised by those tests will fail because the feature hasn't been implemented yet (surprising how often I have to clarify this).
Tests that are worth repeating...
During these sessions, the outcome will be that the tester(s):
-Don't discover any more tests worth repeating (maybe because they are adequately characterised by the existing ATs)
-Do discover some tests worth repeating against a future build (in this context, worth creating an automated AT for)
This story or another story?
Of the tests that are worth repeating, the tester(s) will have discovered scenarios that:
(a)Illustrate a specific stimulus within the scope of the behaviours characterised by the existing ATs but causing an undesirable outcome (e.g. a specific value that causes a failure despite appearing to be equivalent to the values used in the existing ATs)
(b)Illustrate a new behaviour enabled by the current implementation and related to the story but extending the capabilities beyond the spirit of the existing ATs (the tester should have a good idea about this if they were involved in the original discussion about the ATs – which of course they should be).
These tests all describe undesirable behaviours rather than your typical pre-determined AT which often characterised an 'absent behaviour'.
In short - they've found some bugs :-)
If it isn't clear which category the ATs (representing these bugs) fall into, I find a conversation with the product owner is useful.
Items in (b) are summarised by an evolution of the story (in the form of a new story), elaborated with the tests and added to the backlog for customer prioritisation and inclusion in a future iteration.
Prioritisation, prioritisation, prioritisation...
Items in (a) will have new automated ATs created and must be passing before the story is considered 'DONE'. If there are more of these than the team can implement in the iteration as well as their other committed work then these ATs and the other stories in the iteration will need to be prioritised...
Sometimes this results in some of these new tests being bundled under a new story as in category (b) above. Sometimes this results in yet-to-pass pre-determined ATs for that story being de-scoped and treated as in category (b). Sometimes a different story, also scheduled for that iteration, is deferred to a later iteration. It's always a judgement call...
I always ask a question like - "Let's say the current capabilities represented all the necessary scope to go live, would you go live given those tests that fail with undesirable behaviour (as opposed to merely absent behaviour)?"
This cycle continues throughout the iteration, adding new charters for new testing sessions as they become apparent, until all Acceptance Tests for the Story are passing and there are no new bug-inspired ATs in category (a) being found OR those being found (or those remaining) category (a) ATs are those that the Product Owner decides don't matter...
Let's not forget those charters... if you still have testing session cards left on the project wall, you'll want to get those finished too. Unless the team and product owner decide that not doing them is a risk worth taking.
Then , and only then, is the story 'DONE'.
Intra-iteration is the key phrase...
All this really needs to happen during an iteration... If you run out of time before you are 'DONE' then something hasn't worked. If you are doing your first exploratory testing for a story, an iteration or more after it was considered 'DONE' then it wasn’t really 'DONE'...
If, in this sense, you never really get things 'DONE', it's often caused by over-optimistic planning or insufficient ruthlessness in your prioritisation... I find that the main cause is over-optimistic planning - because people always want things to be done as soon as possible. What the cause will be for you is something for you to discuss in your end-of-iteration retrospective... You do have end-of-iteration retrospectives, right? Of course you do, you'd be crazy not to.