Skip navigation.

Why do we write Test Cases?

exploratory testing | test management
Writing Test Cases as almost the only widely accepted and QC-specific idea/technique are object of my wonder since I started in testing field. Now after almost 10 years I have at last some understanding about it. Today I could say that I am advocate of exploratory testing, but I was actually even before I learned the term and idea. And still writing TC makes sense in a lot of cases, it is just wrong to believe that it is the silver bullet in any context.

Last year I compared Test Cases with shield and testing itself with a sword . I still believe Test Case creating could have two purposes/goals:
1) Test Cases are supposed part of the deliverable to the customer. TC goal credibility in this case. Typically UAT (acceptance) level.
2) Test Cases are for team internal use only. Typically System level testing. Testing efficiency should the goal in this case. The idea is to write test cases based on design while code is incomplete, so that we could test product quickly once the code is ready.
With moving to more agile development the second case begin to fail. I’ve seen this happening in my company and posts about this happening in other companies as well. It appears this ends up in one of the following ways:
a) TC are used internally, but the goal is credibility, not efficiency. It also means that TC are dramatically reworked during test execution
b) Exploratory type testing take place, only specific regression TC are written during exploratory testing or afterwards
c) Exploratory type testing take place, no TC written
I will not investigate further type a) as it is simple evidence of weak test manager – he was unable to convince management that this is ineffective usage or this resources. There are also sometime test cases created only to have something to report test progress against. Like we have 80% of test cases written and 70% of them pass. I have already attacked this approach and will in future as much as possible. This is most typical mistake to measure quality in number of defects open and test progress in numbers of test cases, I suggest everyone to read James Bach to learn why.
Cases b) and c) are both OK and depend on either we will need reusable test cases. To be true I believe that regression tests cases written and automated test scripts written have a lot in common. I would even say there are three levels:
I Pure exploratory testing
II Executing the test case written
III Executing automated test script
The design time increases from up (I suppose 0 for exploratory tests) to down, while test execution time decreases. However the scope of defects to be found also decreases, because automated tests for example will only validate what you scripted to validate, that means you should forecast what the defects may appear. While during manual testing you may see the indirect evidence of some defect. More over, the more detailed test case is, the more times one tester have executed it already (and as a result runs it faster now), the less probably he will find those indirect validation problems.

So much theory, now a little bit of practice. What I do in new project is following:
First of all I find any UI automation for the first release of product to be useless. This may be different for one-release projects, but I don’t have experience with those. Of course unit tests like JUnit executing specific API functions makes sense and are ideally created by developers, but sometimes testers may help with that.
Next I don't write ANY tests cases any more during testing cycle. I only update Test Plan that at the end of release has a very detailed "features tested" list with some hints and notes about features not working and bug IDs. Just after the release I do create test cases document details on how to invoke each feature, what input is expected by the feature etc. It is little bit like documentation, but have different goal/approach: goal is to make regression test execution as fast as possible – e.g. I attach data to be imported as much as possible to reduce data preparation time and I don’t care to describe why I use exactly such data – have no time; I explain in details how to do perform the most trivial use case, tester (unless newbie) may add details such as error handling using brains.

I try to use Testing Dashboard as replacement for formal test report with test cases executed/passed/failed/no executed. Sometimes I just communicate progress informally as my “gut-feeling”, and this is actually what PM wants to know, not the numbers of test cases.

Thank you for comments

I'm happy that even more aspects of test documentation have been revealed here in comments. Thinking in advance, reviewing, reusing ... there are so much reasons other than simply must (because required). Note that I've just blogged about test case review to address some comments posted here last year.

More on test cases....

I can't say that I hate writing test cases. I'm a huge list person. Lists are my starting point, my guide. They keep me in line - I can stray and go off as long as I want, but I can always come back to my list to keep my goal in site and to make sure I've covered the most important tasks.

That's what I use test cases for. I use them as a base, and a starting point. Most specifically for new projects. Writing test cases before I even get an exe help me organize how I'm going to manage the project and how I'm going to build the test plan. Test cases are perfect every four months when new co-op students start, or when I have a contractor. They can familiarize themselves with the software, and begin learning to test at the same time. I use exploratory testing at the same time to cover both worlds. Once v 1.0 starts heading out the door, we start automation. So, when work starts on the next version, less time is spent on manually testing test cases. More and more time is spent on et, but still we have deliverables that show marketing, or whoever, that the basic functionality (according to specs - giggle...specs?) has been tested as per request. To me, that does not mean the product is good to go.

There have been times when I've slipped into spending too much time writing test cases. I always need to keep myself aware of this and move out of it as soon as possible. It's an easy trap for me. I do think they're necessary for testers and PM's alike. ET is powerful and attempts to cover everything else test cases do not...which is pretty much everything...

-Ves

-Ves

When test cases are good?

I hate writing test cases. I love exploratory testing. But i'm writing test cases and there is the reasons why:
*This allow me to learn or remember test design techniques in more systematically way (the result of this learning I use in exploratory testing)
*Test cases are good training material for new persons (even for developers), because often documentation is poor
*Newcomers often are used for testing in our company. They don't know nothing about testing and about system. They could perform testing only by test cases (less training required)
*Person, which have less experience in testing, could tell about quality of code only by executed test cases, because they still not able to use gut-feeling

Test cases long term investments

Software testers find bugs, good software testers measure quality and excellent software testers can prove quality.
I agree that spending too much time on test cases is a waste. So make sure you don't go overboard. Just the same, they are a mighty tool for reporting and proving quality. If the best you can say about what you are testing is a "gut-feeling" your product designs need work. Gut-feelings can be accurate, but you should be able to say "I can prove" not "I feel."
I always dislike entering test cases, but they pay-off in a lot of subtle ways, even when they don't prove anything about quality. I can get a sanity check that I am testing the right things, I stay organized and focused during test passes, I can clearly articulate my progress, I can share my test case load with other people or pitch in where others need help, I can pass the component on after the current release.
Certianly no silver bullet, but I think it's well worth the time to make a list of cases from your exploratory testing phase.

Test Scripts and Audit Requirements

Hi Ainars,

A recent assignment had the 'test team' write something like 580 Test scripts and the reason was because its was an audit requirement (because it was a payroll system the external auditors where watching). The funny thing was that the auditor (who was a senior auditor) had very little understanding of testing. He was satisfied with what we had documented (i.e test plan, charter, scripts etc)but took almost no notice of the content. His satisfaction level consisted of..."...they've created 580 test scripts so they must be doing it right!".
Interesting enough, the majority of the major severity defects were found during the Exploratory test phase - very few when 'testing' with scripts.
As you say - the test scripts were created to achieve creditibility - nothing else!

All Tests Cannot Be Documented

JakeBrake says, "In my experience, they are contractual requirements. All tests must be documented and reviewed."

All tests cannot be documented, unless we come up with a technology that is capable of reading a tester's mind and identifying every aspect of her thought process.

Things which are not documented can be reviewed. We have all had retrospective discussions about a burst of testing, have we not?

When someone says "must", I like to ask some questions. What authority is stipulating "must"? Whose purposes are we trying to serve? What is the value of the thing that we must do? What is the cost of the thing that we must do?

---Michael B.

Comply contractual requirements as the only goal

Hi JakeBrake, nice to see you reading my blog. Good point you took – this is really not context of my experience. More over I believe you are talking about something else than I’ve covered that by case 1 “ Test Cases are supposed part of the deliverable to the customer” – case that in my QA Plan template the QA goals would look like the following: completeness is the only quality goal.
There are really more cases. For example customer wants to see that we work according to documented procedure that complies with some standards, like IEEE. They think that following defined process is something that will guarantee product quality. If we have CMM lvl 5 (and yes we have for few departments, not mine) we will make quality product, do you think so? If testers doing that have ISTQB or other certificate that guaranty they know how to do test case review?

Why Write Them?

In my experience, they are contractual requirements. All tests must be documented and reviewed.

Review test ideas before test them...

You are right I’ve skipped this type of goal. In my context I’ve found it more effective and cheaper to find errors in test cases while testing. Unless you are working in waterfall and your goal is not cheap but quick testing once the code freeze is there. And you have all the time you need to write, review, improve and polish your test cases. More over there are still an option to plan and review test ideas even in exploratory testing, I’ve just blogged about it . However I do believe that one tester’s mind is enough to do a good enough testing of a feature given that developer of this feature is going to cooperate a lot. I don’t really like the idea that two or more peer testers are spending their time to learn a feature and generate test ideas of a single feature, given testing resources are so limited in our industry. The only exception is when a mentor is reviewing novices’ test ideas to help to extend them.

Exploratory tests can't be reviewed in advance

The main benefit of test cases written in advance is that they can be reviewed, by test colleagues and developers. Errors in code and test cases can be prevented, and new test ideas will be generated.
Personally I like the combination of documented test cases (or test ideas) and exploratory tests, simultaneously performed.

I don't see much difference - API or UI

Exploratory testing APIs is exactly what I'm doing in my current project. In Eclipse (or other IDEs like this) it is so simple to explore what are the methods a class provide and to try to use them for some purpose, but only once the code is complete and you could debug and extend your tests. TDD is opposite to ET.
I don't need to write test cases to make everyone aware. I write test scripts, run them and ... sometimes detect defects. I don't want to bother developer with all the "flows and data combinations" while he is coding. I let him/her finish... who knows maybe he/she is lucky enough not to run into trouble with my tests.

I agree with your point of vi

I agree with your point of view that writing test cases doesn't add much value in the initial phase of projects. But if you are working on the API testing than it certainly add value. Test cases written during the development phase of APIs can identify defects, make every one aware of the flows and data combinations that can be used. It also make sure that all the APIs are covered in the testing.
In a GUI application, functionality is visible and can be explored easily. Exploratory testing for API is difficult and if you do not have document specifying test cases/results for every exposed API, it is very difficult to build confidence in the system.

Comment viewing options

Select your preferred way to display the comments and click 'Save settings' to activate your changes.