Skip navigation.

Dimensions of a "good" test report

general software testing
[textile] While working with James Bach a couple of weeks ago, I was asked to report my status while I was in the middle of my testing. After stuttering out some abstract bullet points (in no particular order), James stopped me and gave me some advice for how to deliver a test report. James told me to consider the following dimensions:
  • Mission: This is what I'm trying to accomplish...
  • Coverage: I'm looking at this...
  • Risk: I'm looking for this...
  • Techniques: This is how I'm looking for it...
  • Environment: This is where, when, for how long, with who, and with what...
The results of those inputs are:
  • Status:
    • Where am I?
    • What have I found so far?
    • How much more do I have to do?
  • Obstacles:
    • Do I have any issues I need help with?
    • Is there anything I can't work around?
This advice was for both verbal and written reports. The report should cover the above dimensions and be at the appropriate level of specificity for the person asking. We practiced different levels of specificity (which is something I need to practice more), but I think talking about that might become it's own blog post in the future. [/textile]

Thanks Mike for shairing such a valuable tips from James Bach

Prakash

Re: Can you tell me how you determine that?

Mike,
at the moment I am still working on the idea of the appropriate heuristics - I am tending towards leaning a lot on the work Alan Richardson has done of using the [textile]"NLP Meta language in testing":http://www.compendiumdev.co.uk/nlp/default.php as a starting point.
I try to use the audience part as way of finding the correct refelective of the language they use as the prefered communication style. To be fair I cope better with the visual group as i tend to that myself. I also have a simple Formal, Neutral, Informal filter in my brain to try and run over the audience as I prep the report (even if it is adhoc and verbal) to try and make the information exchange as effective as possible.
The one thing, rather like the "level" of detail I still need to work on this - Alan Richardson will vouch for the need for my improvement.
Like Scott, I tend not to use "mission" as part of my heuristic, I tend to "charter" as that does not class in the context of the organisations I work with the "mission statements" that are endemic and seem to lead to a level of switching off in people I know.

Neill McCarthy
"Agile Testers of the World UNIT !"

Intent vs. Mission

I've always liked "Intent" over "Mission". It is a subtle difference that is probably not particluarly relevant here, but I'll make my case anyway - simply because it is on my mind.

- a Mission is assigned by superiors.
- an Intent is self-determined.

- Missions succeed or fail.
- Intentions guide.

- a Mission cannot be changed without the approval of superiors.
- an Intent can adapt to a situation.

- I'd expect to find "Mission" in a CMM Level 5, IEEE829 compliant, blah, blah, blah document written by Elfreide
- I associate "Intent" with agile.

It may or may not be worth noting, that the distinction I'm making comes directly from the U.S. Army and may, therefore, be completely out of line with the rest of the world.

--
Scott Barber
Chief Technology Officer
PerfTestPlus, Inc.
Software Performance Specialist,
Consultant, Author, Educator
sbarber@perftestplus.com
www.perftestplus.com

Great post

Thank you Mike and James. I'll try to remember to use these points in future to describe my testing, etc.

Julian Harty

audience matters

Good point, Neill. A test report is a product, so you must also consider your client.

Can you tell me how you determine that?

Interesting... What heuristics do you use to determine what the audience what's to hear? Where would I incorporate that information in the report? Would it be in the status or somewhere else?

I'm genuinely interested in getting better at this and any insights you can offer into how you do it would be helpful.

A good test report - in some contexts

The list of dimensions works well for me in a number of contexts however it misses one dimenson, for me that I often find useful. This is the audience expectation - what does this audience want to hear. The dragnet view "just the facts, mam" is not always the report required and the levels of specificity does not always cover this.
I am not advocating "falsifying" the results or information transfer, just incase anyone tries to read this into the comment, but finding the appropriate language to communicate to that audience effectivily. I think this "audience" dimension is useful to consider for both written and verbal forms.
Neill McCarthy
"Agile Testers of the World UNIT !"

Comment viewing options

Select your preferred way to display the comments and click 'Save settings' to activate your changes.