QA/Testing (Again)

Sounds like we agree Michael on most things there...

Although the subject isn't interesting to you, it is a question that seems to come up frequently at conferences and in teams. I'm not suggesting we go out there and go on a campaign to rid the world of the incorrect usage of QA... but I just wanted to share my standard response when the discussion does come up... It's not just at conferences... Almost invariably when I say "so you're a tester" the response is "what's the difference".

Regarding empowerment to 'assure quality'. I don't think managers are often empowered to do that... not in my experience at least... They are only able to influence the pressures. Ultimately, our customers (in one way or another) are the ones with the real power... maybe they are managers... maybe not. Managers of software teams and the members of those teams can only influence with the information they provide.

In my experience, too many software teams and their managers provide the answers they think their customers want to hear... "sure we can do all that in the time you've given us". They even convince themselves that it is possible.

We need to use techniques to make the consequences of such underestimations more obvious to ourselves and transparent to our customers...

One of the things that I like that has come out of the agile movement is this type of transparency... and growing awareness of metaphors to helps us communicate the consequences of implementing that feature faster today and the resulting cost to tomorrow.

Technical debt is one such metaphor that I find immensely useful. I can use this to illustrate my point. When that customer comes back and says - we need that feature delivered in half the time of your estimate... I say "sure we can hack that in really quick, but that will incur a technical debt that we'll have to use the next 1-2 iterations to pay it back so won't be able to implement any more business-facing features until after those iterations"... inevitably they come back and say - oh, can you defer that work and hack in this next feature" and my answer is, sure but without paying back the debt it's going to take twice as long to do and then we will probably have 2-3 times as much debt to pay back... many more of these requests will mean we simply can't make any more progress and the system will be getting closer and closer to being unmaintainable... but it's your money and your choice"... The customer is then armed with the information they need to make that choice... live for today at the expense of tomorrow - or strike a balance so we can live for today without it being at the expense of tomorrow.

Some say that we should act more like a regulated industry... and flat refuse to compromise... Trouble is then they will find someone less scrupulous to spend their money with... or give them the answers they want to hear. Perhaps an alternative could be to express a fear of liability and get customers to sign legal waivers every time they ask us to compromise how we build software...

More realistically, I think we, as software teams, need to demonstrate professional integrity and clearly explain the consequences of corner-cutting to our clients and make this transparent.

For example, one client I introduced to the concept of technical debt has had great success with it. They now write an index card for each compromise they make and add it to a special technical-debt wall for their project. When giving estimates for new features, they refer to specific cards on the wall and say "because of these 3 items of code-debt this change and others like it will take twice as long each time we hack it in - and the hack will add a new card because we'll have to remove that once we've refactored". Their client, the sales director, is now aware of this issue and it affects how he prioritises work... sometimes code-debt gets priority sometimes the urgent implementation gets the priority. Simply seeing it grow as the number of cards on the wall grows is enough to capture his interest and ask the team what suggestions they have for doing something about it.

The development team are much happier now as a result because they are proud and want to do a good job and deliver quality... that was the only justification they had... that they preferred to deliver quality... Now they have a way of explaining it in terms that their client can relate to.

That's just one of many examples... but I still say that, based on my experience, managers are seldom empowered... it's the customer (who may or may not be a manager). It's our job to make sure our customers understand the impact on them in the short, medium and long term for every compromise they directly or indirectly request of us.

Antony Marcano

"Can our product work?

Will our product work?

On the surface, these may appear to be the same questions -- but they are quite different. (Thanks to Michael Bolton for expressing this difference this way.)"

And thanks to James Bach for thinking it up so that I could express it that way. That is, credit for this distinction goes to James, not to me.

The whole QA/QC debate isn't interesting to me, though. This question is already settled, in that it's apparently never going to be settled. People are going to persist in calling themselves "Software Quality Assurance Engineers" or "Software Quality Control"--and arguing about which is which--forever, apparently. So when someone gives me either title, I just say, "So you mean you're a tester? You test software?", and they say, "Yeah." "Okay," I reply, and then we move on.

The fallacious notion that testers can assure quality is marginally more interesting to me, but these days, only for long enough to skewer it. If you don't have the authority to change the budget, schedule, scope of the product, contractual obligations, staffing, marketing materials, how the work shall be done, and so forth, then you can't really assure quality. And if you do have that authority, I think you're a manager. However, you can still test in either case.

---Michael B.

P.S. the can/will our product work questions are relevant to the "Testing" or QC 'question' I propose... they are the next level of detain down from this high-level distinction... My two 'questions' are intended to differentiate between QA and Testing... your questions help to drill down into testing to start deepening one's understanding of testing alone.

But, none of this changes the point that people have misunderstood that QA is synonymous with testing... QA is a different thing altogether... that's really what my post was about.

Nonetheless, I enjoyed reading your comment Ben. Thanks for sharing your insights.

Antony Marcano

QA & QC isn't exclusively applied in manufacturing. There are many other domains, older than software development, in which QA & QC have been applied.

When people talk of manufacturing, I ask them to consider car manufacturing... software development is more akin to developing a concept car than manufacturing the production version... it still needs quality control nonetheless.

On an agile project we aim to have software that is always of releasable quality (even if it is not yet of sufficient scope to justify a release from a business point of view)... so, I do not ask - "will it work". I want to know "does it work (to the best of our current understanding of anticipated usage scenarios)".

With ATDD, we aren't just asking can it work - we are using the process to understand what 'it works' means. How will the business use it? We incrementally and iteratively collect and generate examples of anticipated usage and infer the system from those examples. The fact that we automate those tests tells us when we've fulfilled certain examples of usage and by their nature, continue to validate the evolving software against those previously-fulfilled-examples. So, actually, ATDD supports the business in helping them to understand what they need the software to do and the development team in understanding how the business want to use the software... and then that the software still does all the things we previously got it to do...

Your impression of ATDD is a common one... but there is so much more to it than that... Look out for my article on "Understanding ATDD" in the October issue of Better Software Magazine where I go into the detail...

Antony Marcano

A nice distinction, although ...

Both QC and QA concepts are often taken too far in software development. Software development is mostly design work -- not manufacturing work. Manufacturing QA and QC work tends to focus on producing the same thing every time a thing is produced. Consistency and repeatability are key. This isn't a problem with software. Every time we copy code, we get the same thing. Software flaws are generally design flaws, not manufacturing flaws. Why then do we as an industry keep looking to manufacturing QC and QA concepts for processes to apply to design work?

Due to the nature of software, I believe we need testing (QC) that does something more important that tell us if we are consistent. In software we build new things. I believe we need testing that answers two distinct questions:

Can our product work?

Will our product work?

On the surface, these may appear to be the same questions -- but they are quite different. (Thanks to Michael Bolton for expressing this difference this way.)

Developer testing tends to lean towards demonstrating that our products can work. TDD, ATDD, and agile-ish automated acceptance tests are great at supporting programming by showing that our products can work.

At the other end of the testing spectrum is a critical evaluation of what we build. This kind of testing seeks to show that our products will work for our stakeholders (e.g., our business, customers, users, maybe even the public at large, anyone else that needs our software to work). This is an investigation into possible threats to the value of our products.

Maybe this can vs. will work model is too simple. As with any model, it is wrong, but I find it useful.

And thinking of modeling...

Testing of anything but the most trivial system is sampling at best. Sampling based on models -- whether mental or explicitly defined in documentation or automated tests. Testing neither assures or controls quality. At best, testing helps us forecast quality. As testers we need to seek the best samples to provide the most accurate forecast possible given the resources available to us.

Ben Simo
President, AST

FROSST Conference
Questioning Software