How test team itself could improve test process: experience
Submitted by Ainars Galvans on Wed, 07/12/2005 - 09:23.
project management | test management
Test Process Improvement is most frequently addressed topic by different publications. Most authors either refer to CMM(i) and/or take the managerial approach: talks about the better communications, measurements or sometimes test methodology or improving tester’s skills. This is natural approach as test process improvement is supposed to be test manager task.
I hope to take a little bit different approach in this paper. I will show path for a whole test team improvement through change of the attitude and even culture supported by minimal improvements into test system and process.
The goal of my 3 previous posts here was only to prepare ground for this one, so it is short enough and describe my experiences in managing classic test project (with formal regression test cycles).
Issue: Company goals should drive test improvement, not tester goals
I’m working for company Exigen Group that state in the home page to be “… our Joint Value Assessment (JVA) , builds a roadmap of tactical and strategic improvements. Prioritisation is based upon risks and returns for each improvement that creates the optimal solution by considering all financial, organizational and technical considerations…“
The bottom line for this is that companies don't want to automate their processes just because they want it to be managed by software. There was the time, but not any more. They want to see the ROI out of managing their business process by software application. I also see the trend of looking for ROI from testing within project nowadays. It is not any more "we test (or we don't) just because we need (or don't) to test".
I will try to approach testing as a business process inside an IT company. This process as a service for engineering team has own business goals Diary: becoming a PM : measuring quality and enabling it’s improvement. I should consider limitations: technical - set by test system (such as Test Director), organization - such as policies, guidelines and standards to be followed as well as financial – perform improvement within given test team and with minimal training. That’s why I’m not going to invent brand new testing methodologies/techniques, change the measurement program or any other activities requiring financial, organization or technical support.
Resolution: tester should take time to better evaluate the test deliverables
Testing as a business process have direct goal of creating following artifacts: defect (or anomaly according to IEEE) reports, test documentation including reusable (test cases, test procedures, test scripts) and partly reusable (test plans, test specification, etc) and the last but not least important – creating run-logs (or Execution Records according to IEEE).
As I mentioned above the actual business goal of testing is to measure and improve quality. Defect reports suite input for improvement (done by development), defect severity set according to IEEE and optionally other properties – according to Defect severity and priority suites for quality measurement but this measurement is not complete. According to Pass/Fail for run-logs. Is it useful info? this measurement could be complemented by evaluating each run-log success in terms of testers disappointment in quality (not in terms of validation or if any defect is found).
More over improvements/perfection of test documentation are commonly addressed in different topics. Kaner in his 16 page large publication What is a good test case address only the issue of good and bad test case stating: “Test cases can be “good” in a variety of ways. No test case will be good in all of them.“ I believe that this is even more so talking about partly reusable documentation. Even in a single project I have both – very detailed test plans for new features and test plans with empty content for regression.
So my approach for test process improvement lies behind evaluating defect and run-logs according to testing business needs (opposed to business needs, such as defect business value evaluation). Although it is so simple the results achieved in a project where I implemented this and monitored improvements for several years outlined continuous improvement. The improvement took so much because it required both testers and whole developers to change attitude and even culture. I could publish the measures I collected if there is an extended interest, but I myself approach skeptic to the local collected measurements published: there are lies, big lies and statistics…
Implementation path: a simple one
Firs of all I chose IEEE standard as one of the most popular and at the same time most detailed describing requirements against items that I’m addressing. My suggestions don't go outside of this standard.
Secondly the only requirements for implementation is existing system for defect tracking with severity to be valudated and test system that allow recoding execution status with pass/fail flag.
There hardest work as I mentioned above lies behind change of culture/attitude. It includes both - testers and developers.
I hope to take a little bit different approach in this paper. I will show path for a whole test team improvement through change of the attitude and even culture supported by minimal improvements into test system and process.
The goal of my 3 previous posts here was only to prepare ground for this one, so it is short enough and describe my experiences in managing classic test project (with formal regression test cycles).
Issue: Company goals should drive test improvement, not tester goals
I’m working for company Exigen Group that state in the home page to be “… our Joint Value Assessment (JVA) , builds a roadmap of tactical and strategic improvements. Prioritisation is based upon risks and returns for each improvement that creates the optimal solution by considering all financial, organizational and technical considerations…“
The bottom line for this is that companies don't want to automate their processes just because they want it to be managed by software. There was the time, but not any more. They want to see the ROI out of managing their business process by software application. I also see the trend of looking for ROI from testing within project nowadays. It is not any more "we test (or we don't) just because we need (or don't) to test".
I will try to approach testing as a business process inside an IT company. This process as a service for engineering team has own business goals Diary: becoming a PM : measuring quality and enabling it’s improvement. I should consider limitations: technical - set by test system (such as Test Director), organization - such as policies, guidelines and standards to be followed as well as financial – perform improvement within given test team and with minimal training. That’s why I’m not going to invent brand new testing methodologies/techniques, change the measurement program or any other activities requiring financial, organization or technical support.
Resolution: tester should take time to better evaluate the test deliverables
Testing as a business process have direct goal of creating following artifacts: defect (or anomaly according to IEEE) reports, test documentation including reusable (test cases, test procedures, test scripts) and partly reusable (test plans, test specification, etc) and the last but not least important – creating run-logs (or Execution Records according to IEEE).
As I mentioned above the actual business goal of testing is to measure and improve quality. Defect reports suite input for improvement (done by development), defect severity set according to IEEE and optionally other properties – according to Defect severity and priority suites for quality measurement but this measurement is not complete. According to Pass/Fail for run-logs. Is it useful info? this measurement could be complemented by evaluating each run-log success in terms of testers disappointment in quality (not in terms of validation or if any defect is found).
More over improvements/perfection of test documentation are commonly addressed in different topics. Kaner in his 16 page large publication What is a good test case address only the issue of good and bad test case stating: “Test cases can be “good” in a variety of ways. No test case will be good in all of them.“ I believe that this is even more so talking about partly reusable documentation. Even in a single project I have both – very detailed test plans for new features and test plans with empty content for regression.
So my approach for test process improvement lies behind evaluating defect and run-logs according to testing business needs (opposed to business needs, such as defect business value evaluation). Although it is so simple the results achieved in a project where I implemented this and monitored improvements for several years outlined continuous improvement. The improvement took so much because it required both testers and whole developers to change attitude and even culture. I could publish the measures I collected if there is an extended interest, but I myself approach skeptic to the local collected measurements published: there are lies, big lies and statistics…
Implementation path: a simple one
Firs of all I chose IEEE standard as one of the most popular and at the same time most detailed describing requirements against items that I’m addressing. My suggestions don't go outside of this standard.
Secondly the only requirements for implementation is existing system for defect tracking with severity to be valudated and test system that allow recoding execution status with pass/fail flag.
There hardest work as I mentioned above lies behind change of culture/attitude. It includes both - testers and developers.