How Much/Does stakeholders affect testing definition?

[textile]

Recently discussed in my test team the variety of definitions of testing. We finally agreed that testing definitions is impossible without mentioning stakeholders. Why? Because there are so different things people do on different projects calling it "testing" not because they have red different books or have different skills, attitude, etc. but because of the different stakeholder needs and expectations. The reason is behind the project context: goals and mission of testing vary: making sure user acceptance tests will go smoothly; minimizing defects reported in production; documenting the evidence of an attempt to build quality into software, certify the process etc. Not only we agreed that testing definitions is impossible without mentioning stakeholders. First of all we found that almost none of the definitions that could be found on web include it. More over, testing definition as we see it defined by context-driven leaders approach stakeholders too superficially (casually mentioning them).

Foreword: trying to develop context-driven approach I've announced recently "joining" context-driven school . With this and further posts I'm going to attack the published ideas by this school leaders. I have and overweening intention to help the leaders to improve their messages. Even if what I write will show that I misinterpret something they write about - it will mean others could misinterpret it as well. This post is a result of the following activity. Recently started weekly workshops on exploratory testing within my team. The first hour session analyzing presentation by Cam Kaner ended up in several days of discussions over testing definition - slide 11.

Stakeholder with or without authority. Cam Kaner once defined

Testing is an empirical investigation conducted to provide stakeholders with information about the quality of the software under test

I would like to discuss "...conducted to provide stakeholders with information about the quality..." . It is not as straightforward as it may seem at the first look at it (define all stakeholders and ask each one what are the info they want). For example classic stakeholder - Users. They simply never got this information even if they want. Yes, they get the list of known problems and workaround to them, but this is not information about quality, it is information about workarounds. Problems that has no workaround are not typically revealed to them. Other characteristics of quality are more frequent reveled by third parties such as reviews. While talking about project stakeholders in pm-related resources they are talking about the responsibility and authority of the stakeholders. I feel like this is a key context in testing. So my extension would be something (I'm not that good at definitions) like this "...conducted to provide those who have the responsibility and authority with regards to quality with information about quality as it will be perceived by stakeholders...". My project manager have different value compared to users, but he want to know information about user's value.

Stakeholders may change as time goes It could appear that stakeholders change during the project or even when it is over. Perhaps consultants never ever observe that. Working for the same company already almost 10 years I've seen even project mission and software goals changing in an instant (not to mention stakeholders and those with responsibility and authority). Once sales were able to sell that product for what it was never intended to and customer is willing to pay - we have no choice but to comply. So you can base your testing on the knowledge of the current stokeholds - you will survive the day. How about tomorrow? How about the day after? It is especially true in case of performance testing. What if tomorrow you will learn that another stakeholder want the same tests but to monitor different attributes of the system or hardware? It could appear that tomorrow a new stakeholder will blamed you for not detecting certain types of defects (not significant for other stakeholders). That's where not-context-driven but a generally accepted practice makes sense. I don't say you should base on them, but you have to include them. You have to monitor what the stakeholder need and also a little bit more of what the next stakeholders could probably want and will expect you to do as generally accepted. You have to report at least a few bugs of the certain type to get them rejected, so that you have evidence that they are typically rejected. You have to report all the bugs that may seriously bug some stakeholder in future - even they all get postponed (because current stakeholders don't care)- you will be able to provide the list to new stakeholders for re-prioritization.

Different stakeholders in different project stages/phases/moments Caner extend his definition (see above) with comment

Stakeholders have different informational needs at different times, in different situations.

I totally agree with that. Suppose your deadline is 3 moths from now and you start testing. Program is unstable and has a lot of bugs at this moment. Who is your primary stakeholder? Yourself - the test team. I believe everyone know what doe "blocked" status means on a test report. It means testing could not proceed. Does this provide information about quality to stakeholders? No it doesn't, it only say that we are unable to provide it and so we need the issue to be solved soon. As I move forward in project I think my primary clients - those who I want to satisfy - are development. I want to provide them with defects as fast as possible, so that they could organize, prioritize and manage their work on fixing the problems. Only when we are close to our deadline the project manager become interested in test reports. Only then grammar errors become part of "information about quality" :) .

So I have to fix my extend vision with "...those stakeholders who have the responsibility and authority with regards to quality in the given time interval (phase).A good test strategy is however to take into account those who will have the responsibility and authority later during the project and mitigating risk for unknown stakeholders to have the responsibility and authority. Mitigating the risk by gathering generic information - this effort should be proportional to the risk. Of course it is now too long and complicated to be a good definition...

[/textile]