Test Driven Documentation
Submitted by Ainars Galvans on Mon, 29/01/2007 - 08:57.
exploratory testing | general software testing
You can’t imagine how lucky I am. I’ve got a technical writer in my project who is technically very strong. He is technically stronger than an average tester. So what a big deal? I’ll tell you what! He doesn’t simply fix spelling errors in text provided, he actually play with software to describe it himself. I could guide him through the software jungle (while it is not typically possible to follow developers as they are like monkeys in jungle – they use trees to travel fast). On the other hand he helps me to find defects – yes he is finding defects and I know what kind of defects he will find and exclude some tests form my testing. As he need to completely describe the path through the jungle he is paying attention to details that I miss as my goal is to find as much possible paths as possible instead of analyze the main path to the details.
First of all a context.
The software I’m talking about is not an end-user application created according to detailed business requirements. This is a product, a tool or even framework with target audience – technical people or even developers. For example imagine an UML diagram editor tool. This will typically be used by developers or architects. We may suppose certain UML models as requirements, but actually requirement is to support any model creation, editing, exporting, etc.
Now – how do you create documentation for such tool? Documenting each window’s each feature, right? That’s typical, but not the best. I would like the documentation to describe for each UML feature how to implement it using the tool, wouldn’t you? “To define 1:N relation you have to make a link between nodes and in each end set up the property called relationship” instead of “set property relationship to define relationship [with another object]”?
The classic practice and why I don’t like it
Working with other tech. writers our methodology was following developer –input->technical writer –output documentation -> tester – verify if accord to what he test –documentation bugs-technical writer-> fixed documentation. This way tester could only suggest to fix what is described incorrectly. He can’t influence the style of document and neither could technical writer. As a result we have documentation actually written by developer. And remember – normal developer hates writing it, so he will write as little as possible.
In the contest I’ve described above it is sometimes even worse. The only input that tester receive is this documentation and he write test cases based on that. As a result he only test that button add_item_x adds item, but don’t test how it affects the system as it is not documented how it is supposed to affect and why we need to add this item at all.
One more thing. We had such a documentation for our legacy product. I know quite frequently our solution implementation team have asked me to send them our test cases as they find them better than documentation because they describe how really to use product in certain cases (including hints, samples, etc.) not how it is intended to be used in general case.
The different practice
My practice is following. Me as a tester do exploratory testing, talk to developer, do whatever to understand how to use the product. Once I start to understand it I call the technical writer to my place and present him the product. I show how to use it. While showing I tend to discover problems or a feature that I missed while testing alone – this is well known phenomena I believe. He is perfectly able to understand me as our technical levels are very close. Well, we are probably pair-testing product to some degree.
Next he goes to his place and try to repeat what I’ve shown him and document it as he progress. While doing so he pays attention to details that I do not so he also find more problems. And the most typically he find that there are something that should not be there. A property, an item, a button that does nothing or some bad things. Because he ask for each item what does it mean as he need to document it. As a result I know that I don’t need to test too much for each available UI element in each place, I could pay more attention to different test scenarios instead of covering UI items.
Test Driven Documentation – summary
I’ve stolen the name from Test-Driven-Development, haven’t I? Well... as I’ve said before English is my 3rd language so I may be wrong but I (as ex-developer and mathematic) have always resisted the idea that limited list of unit tests are driving the code development. I’m not against test-before code, not at all. I only think such a tests are controlling and/or directing the development, not driving it.
I feel much better when say that tests are driving documentation. When a certain test passes it is a good driver to document the case that works. When there are defects, workarounds, limitations and we must document those things along if want to provide useful information instead of nice-looking – well marketing materials.
I really do believe that manual tests, especially when done with a degree of exploratory testing approach, should be the driver for user documentation. Not requirements, not system design and not developer’s vision of how he intended things to work – only the tester’s vision – how things really works.
First of all a context.
The software I’m talking about is not an end-user application created according to detailed business requirements. This is a product, a tool or even framework with target audience – technical people or even developers. For example imagine an UML diagram editor tool. This will typically be used by developers or architects. We may suppose certain UML models as requirements, but actually requirement is to support any model creation, editing, exporting, etc.
Now – how do you create documentation for such tool? Documenting each window’s each feature, right? That’s typical, but not the best. I would like the documentation to describe for each UML feature how to implement it using the tool, wouldn’t you? “To define 1:N relation you have to make a link between nodes and in each end set up the property called relationship” instead of “set property relationship to define relationship [with another object]”?
The classic practice and why I don’t like it
Working with other tech. writers our methodology was following developer –input->technical writer –output documentation -> tester – verify if accord to what he test –documentation bugs-technical writer-> fixed documentation. This way tester could only suggest to fix what is described incorrectly. He can’t influence the style of document and neither could technical writer. As a result we have documentation actually written by developer. And remember – normal developer hates writing it, so he will write as little as possible.
In the contest I’ve described above it is sometimes even worse. The only input that tester receive is this documentation and he write test cases based on that. As a result he only test that button add_item_x adds item, but don’t test how it affects the system as it is not documented how it is supposed to affect and why we need to add this item at all.
One more thing. We had such a documentation for our legacy product. I know quite frequently our solution implementation team have asked me to send them our test cases as they find them better than documentation because they describe how really to use product in certain cases (including hints, samples, etc.) not how it is intended to be used in general case.
The different practice
My practice is following. Me as a tester do exploratory testing, talk to developer, do whatever to understand how to use the product. Once I start to understand it I call the technical writer to my place and present him the product. I show how to use it. While showing I tend to discover problems or a feature that I missed while testing alone – this is well known phenomena I believe. He is perfectly able to understand me as our technical levels are very close. Well, we are probably pair-testing product to some degree.
Next he goes to his place and try to repeat what I’ve shown him and document it as he progress. While doing so he pays attention to details that I do not so he also find more problems. And the most typically he find that there are something that should not be there. A property, an item, a button that does nothing or some bad things. Because he ask for each item what does it mean as he need to document it. As a result I know that I don’t need to test too much for each available UI element in each place, I could pay more attention to different test scenarios instead of covering UI items.
Test Driven Documentation – summary
I’ve stolen the name from Test-Driven-Development, haven’t I? Well... as I’ve said before English is my 3rd language so I may be wrong but I (as ex-developer and mathematic) have always resisted the idea that limited list of unit tests are driving the code development. I’m not against test-before code, not at all. I only think such a tests are controlling and/or directing the development, not driving it.
I feel much better when say that tests are driving documentation. When a certain test passes it is a good driver to document the case that works. When there are defects, workarounds, limitations and we must document those things along if want to provide useful information instead of nice-looking – well marketing materials.
I really do believe that manual tests, especially when done with a degree of exploratory testing approach, should be the driver for user documentation. Not requirements, not system design and not developer’s vision of how he intended things to work – only the tester’s vision – how things really works.