AA-FTT - Agile Alliance Functional Test Tools Workshop
Submitted by Antony Marcano on Sun, 14/10/2007 - 04:56.
agile
I've spent the last two days (Thursday and Friday) in the Agile Alliance Functional Testing Tools Workshop. The title of the workshop is somewhat misleading. What we were really talking about was tools for "Executable Automated Acceptance Test Driven Development". The process by which you explore and elaborate requirements with Tests that describe the customer's intent. FIT is an example of a tool designed for this process.
Attendeees were:
(Alphabetical order of surname)
Jennitta Andrea
André Brissette
Bob Cotton
Ward Cunningham
John Dunham
Adam Geras
Elisabeth Hendrickson
Naresh Jain
Pekka Laukkanen
Kevin Lawrence
Antony Marcano
Brian Marick
Frank Maurer
Grigori Melnik
Gerard Meszaros
Stephen Michaud
Juha Rantanen
Christian Schwarz
Jim Shore
Ben Simo
Stein Kåre Skytteren
Pierre Veragen
Firstly, thank you to Jennitta (& team) for inviting me to join this amazing group of people on this workshop.
Also thank you to Elisabeth for being a superb facilitator!
During this interesting and enlightening workshop, we saw many different ideas from each of the attendees, many of whom brought along tools that they have built, or are in the process of building.
There were numerous impressive tools. Some of those not addressing the problems associated with FIT and taking a completely different approach included "Robot" (Pekka Laukkanen, Juha Rantanen) and "Cubic Test" (Christian Schwarz, Stein Kåre Skytteren).
During the workshop, I had several AHA!!! moments, most of them happening in the last 30 minutes of the workshop.
During one of the Demos - Ben Simo demonstrated his model based testing framework. It automatically generates thousands of test cases based on a model that is described in a collection of tables.
AHA!! I had an idea... It occurred to me that it should be possible to incrementally extract a 'model' of the domain from the acceptance tests we write... we might then use that model to generate automated tests that could throw many more scenarios at the application. The purpose of this would be to quickly identify 'gaps in our thinking'. Ideally, I'd like to then select any of those tests that fail and 'render' them as acceptance tests - driving further work.
Brian Marick showed his graphical test style. He and Ward realised that they were attempting to visualise tests, except Brian's was a visualisation of what should happen and Ward's was a visualisation of what did happen. (Both tools were still creating tests first).
Ward Cunningham provided a demo on the integrated eclipse portal self-testing capabilities (also known as Swim), mentioned earlier.
It showed how you can run entire tests, visualise the results seeing the portions of the views relevant to each part of the test. You can view a 'timeline' of the actions in a test and see the various roles involved in an end to end use case. There are other capabilities such as running the tests up to a selected step and then jumping into the application at that point. Tests were also used as part of the help documentation.
The significance of this didn't really hit me until later.
During the wrap up Bob Cotton observed that Ward's self-testing software may make tools irrelevant... AHA!!!!
This is when it dawned on me... The future testing tools may not be "tools" at all... instead, we are talking Design Patterns for Self-Testing software.
Thus, I proposed a future step of the group is to investigate this possibility (perhaps as we build the next phase of tools) and gather experiences with a view to writing a book "Patterns of Self Testing Software".
So, when you see that book... you know where you heard it first ;-)
I'll try to tell you more about the workshop as time goes on.
Attendeees were:
(Alphabetical order of surname)
Jennitta Andrea
André Brissette
Bob Cotton
Ward Cunningham
John Dunham
Adam Geras
Elisabeth Hendrickson
Naresh Jain
Pekka Laukkanen
Kevin Lawrence
Antony Marcano
Brian Marick
Frank Maurer
Grigori Melnik
Gerard Meszaros
Stephen Michaud
Juha Rantanen
Christian Schwarz
Jim Shore
Ben Simo
Stein Kåre Skytteren
Pierre Veragen
Firstly, thank you to Jennitta (& team) for inviting me to join this amazing group of people on this workshop.
Also thank you to Elisabeth for being a superb facilitator!
During this interesting and enlightening workshop, we saw many different ideas from each of the attendees, many of whom brought along tools that they have built, or are in the process of building.
There were numerous impressive tools. Some of those not addressing the problems associated with FIT and taking a completely different approach included "Robot" (Pekka Laukkanen, Juha Rantanen) and "Cubic Test" (Christian Schwarz, Stein Kåre Skytteren).
During the workshop, I had several AHA!!! moments, most of them happening in the last 30 minutes of the workshop.
During one of the Demos - Ben Simo demonstrated his model based testing framework. It automatically generates thousands of test cases based on a model that is described in a collection of tables.
AHA!! I had an idea... It occurred to me that it should be possible to incrementally extract a 'model' of the domain from the acceptance tests we write... we might then use that model to generate automated tests that could throw many more scenarios at the application. The purpose of this would be to quickly identify 'gaps in our thinking'. Ideally, I'd like to then select any of those tests that fail and 'render' them as acceptance tests - driving further work.
Brian Marick showed his graphical test style. He and Ward realised that they were attempting to visualise tests, except Brian's was a visualisation of what should happen and Ward's was a visualisation of what did happen. (Both tools were still creating tests first).
Ward Cunningham provided a demo on the integrated eclipse portal self-testing capabilities (also known as Swim), mentioned earlier.
It showed how you can run entire tests, visualise the results seeing the portions of the views relevant to each part of the test. You can view a 'timeline' of the actions in a test and see the various roles involved in an end to end use case. There are other capabilities such as running the tests up to a selected step and then jumping into the application at that point. Tests were also used as part of the help documentation.
The significance of this didn't really hit me until later.
During the wrap up Bob Cotton observed that Ward's self-testing software may make tools irrelevant... AHA!!!!
This is when it dawned on me... The future testing tools may not be "tools" at all... instead, we are talking Design Patterns for Self-Testing software.
Thus, I proposed a future step of the group is to investigate this possibility (perhaps as we build the next phase of tools) and gather experiences with a view to writing a book "Patterns of Self Testing Software".
So, when you see that book... you know where you heard it first ;-)
I'll try to tell you more about the workshop as time goes on.