The only defect measure I would publish
Submitted by Ainars Galvans on Mon, 14/04/2008 - 14:55.
general software testing | metrics
Recently I’ve found my (sever years old) presentations with a lot of defect metrics and analysis. Only one slide I am proud of still – the percentage of “cancelled” bugs (drilled-down to percentage of reject reason: duplicated, not repeatable, etc.). What I like about this measure that it is direct – I’m really interested to keep the number of rejected bugs low – yes I’m interested in testers not reporting them in the first place.
PM experience from a large project
Few years ago I was managing approximately 15 testers testing approximately 10 years old product. PM was more than interested in metrics, so I was improving process to get better defect metrics: not only bug trends (open and opened per week), but also bug return rates, regression VS new development bugs, etc., etc. Almost year almost half of my time wasted in this are, so I wanted to share the experience at a testing conference, but was not accepted. Now as I review my old presentations I am happy I was not accepted. Because I see only one metric that I would like to recommend you using and only if you understand how do you do it. Everything else was did at least some type of a harm to the project when published. And they bring us almost no information, at lest none that we didn’t know (or at lest assumed) before.
The measure(s) to publish
This is the only measure that I am really interested in maximizing. Well not exactly maximizing but at least keeping high enough. It means that this is the only one that I would agree and want to publish. Or actually want to minimize. I want testers to minimize bugs that are reported but had no impact on the software: deferred, rejected and even postponed/delayed defects. Because not only reporting, but also reviewing and deciding about the defect takes time, sometimes a lot of time.
So the only number I keep look at is % of defects fixed, rejected, postponed . For example: assume tester reported 100 bugs, 50 of them are fixed, 30 more rejected (duplicated, not repeatable, simply wrong) and 20 more postponed (i.e. management decided not to care about this bug yet). Then it appears that 50 % of the bugs he have reported were worth to report, 30% more was waste of someone’s time and 20% more – is somewhere between those two categories.
If the number of rejected is high, I want to lower it. So I want to know deeper analysis - how much of them are duplicated defects, how much “not repeatable”, etc.
P.S. How about "know bugs"
There is one special category – defects that get’s reported, accepted as defects and documented either as known bug or even a feature. This is not really a waste of time – you helped your project team to document a defect and helped customers to red about the defect and avoid it. So I count them as fixed, but you may do otherwise depending on your project specific.
PM experience from a large project
Few years ago I was managing approximately 15 testers testing approximately 10 years old product. PM was more than interested in metrics, so I was improving process to get better defect metrics: not only bug trends (open and opened per week), but also bug return rates, regression VS new development bugs, etc., etc. Almost year almost half of my time wasted in this are, so I wanted to share the experience at a testing conference, but was not accepted. Now as I review my old presentations I am happy I was not accepted. Because I see only one metric that I would like to recommend you using and only if you understand how do you do it. Everything else was did at least some type of a harm to the project when published. And they bring us almost no information, at lest none that we didn’t know (or at lest assumed) before.
The measure(s) to publish
This is the only measure that I am really interested in maximizing. Well not exactly maximizing but at least keeping high enough. It means that this is the only one that I would agree and want to publish. Or actually want to minimize. I want testers to minimize bugs that are reported but had no impact on the software: deferred, rejected and even postponed/delayed defects. Because not only reporting, but also reviewing and deciding about the defect takes time, sometimes a lot of time.
So the only number I keep look at is % of defects fixed, rejected, postponed . For example: assume tester reported 100 bugs, 50 of them are fixed, 30 more rejected (duplicated, not repeatable, simply wrong) and 20 more postponed (i.e. management decided not to care about this bug yet). Then it appears that 50 % of the bugs he have reported were worth to report, 30% more was waste of someone’s time and 20% more – is somewhere between those two categories.
If the number of rejected is high, I want to lower it. So I want to know deeper analysis - how much of them are duplicated defects, how much “not repeatable”, etc.
P.S. How about "know bugs"
There is one special category – defects that get’s reported, accepted as defects and documented either as known bug or even a feature. This is not really a waste of time – you helped your project team to document a defect and helped customers to red about the defect and avoid it. So I count them as fixed, but you may do otherwise depending on your project specific.