SMART thinking.... agile testability analysis

[textile]When you have a basis for a test, such as a requirement, a story or a step in a use-case... how do you know it is 'testable'. You could adopt an approach that involves 20, 30 or more check-points, you could do it based on fuzzy feelings - or you can adopt a simple checklist that covers most of your needs with minimal effort and a degree of consistency.

If your input isn't measurable, then you need to make your test measurable at the least... perhaps the test will become the written definition (test-first)... but you will still need a measurable input, even if it is the spoken-word of your customer.

So, here is a simple, 5 point check-list to make sure that the input is testable...

h1. SMART: agile testability analysis for requirements, stories, use-cases and more

These are my version of the definitions:

* **Specific** - Is distinct i.e. multiple items (requirements/steps/etc.) are not compounded into one and are not duplicated
* **Measurable** - Can be quantified in some way (even if it is in terms of TRUE/FALSE against one or more qualitative measures), does not contradict another item under review. Succinctly, it is 'reasonably' unambiguous
* **Achievable** - Can be achieved within the prevailing constraints (time, resources, etc.)
* **Realistic** - Could be achieved if one or more of the prevailing constraints were lifted
* **Traceable/Timebound** - This depends on your priorities, but you can apply both... see below

I call each of these a 'Perspective'. Each perspective is applied against each distinct item within an input. If the input is a requirements list, then the item is a single requirement. If the input is a use-case then the item is a step.

Each of these perspectives plays a key role in establising testability... For each one that isn't satisfied, you have more work to do or the value of what you can do with the input is reduced.

h2. Broad appeal

SMART can make the whole project more effective (doing the right things) and efficient (doing things by the right time).

Once you have mastered taking a SMART perspective, you will start seeing many other things that it applies to. I first learned about it on a project management course many years ago as a checklist for writing good project objectives. Since then I have adapted it for testability.

Because of the broad applicability of it, it is something that you really only need to learn once and five definitions are pretty easy to remember. This also helps to bring consistency if your whole project is thinking 'SMART'.

Sometimes, all you might have to work with are some project objectives... get them SMART and your life will become dramatically easier!


h2. SMART in context

The beauty of SMART is that it can be applied very quickly. You don't even need to have a special form or database... pencil and hard-copy often do the trick. It needn't take long (unless you are in a project that is trying to specify a big system to the last nut and bolt).

If you are faced with a large document and you have the luxury of a team, split it up... but not in the obvious way (giving pieces of the document to different people). Give each team member one or two 'perspectives'. So, for example the person taking the 'Specific' perspective ignores all the others and concentrates only on determining if the items within the whole document/input are specific.

Some perspectives require more effort than others and some need to be done in conjunction with others. For example, you can't say if it is achievable until you have an idea of what measurement you would derive your assertion (expected result) from.

And make sure you prioritise on the ones you feel are most important.

If you are on your own and short on time then sample at least a 5%-10% chunk of the document and skim through the rest.

Now I will elaborate on each of the perspectives a little more.


h3. Specific

**Specific** - Is distinct i.e. multiple items (requirements/steps/etc.) are not compounded into one and are not duplicated

How many times have you seen a statement that is meant to be a requirement, or a step in a use-case that has many items all compounded into one? This can sap valuable time extracting the distinct items to test from. Also, it can create an illusion of context when infact, the distinct items within the compounded statement might be very separate. This alone won't mean that the item is not testable, but it does affect the effort required to work with it.

h3. Measurable

**Measurable** - Can be quantified in some way (even if it is in terms of TRUE/FALSE against one or more qualitative measures), does not contradict another item under review. Succinctly, it is 'reasonably' unambiguous


I need something to base my assertion on. Even if it is an assumption, it needs to be measurable in some quantifiable and objective way. This is especially important if your customer is a client that will only pay for the system if the agreed acceptance tests all pass. Subjective assertions like "the sponsor is satisfied" (as I have seen in real life) exposes the development organisation to a lot of risks... especially since with some people, satisfaction may be more to do with the mood they are in, rather than whether you have measurably delivered on your promises.

Quantification can include qualitative methods that allow you to translate it into some quantifiable measure. This might be TRUE/FALSE (to derive PASS/FAIL) or even in terms of the analysis of structured responses, such as Strongly Agree/Agree/Disagree/Strongly Disagree.


h3. Achievable

**Achievable** - Can be achieved within the prevailing constraints (time, resources, etc.)

You need an idea of how you will go about your test by now. You need to be sure that you have the necessary tools and resources (human or otherwise) to perform the relevant test. This is where "Realistic" becomes useful!


h3. Realistic

**Realistic** - Could be achieved if one or more of the prevailing constraints were lifted

If you have established that something is not achievable, but it is realistic then you have something to negotiate with. Perhaps you have put a question mark against the achievable perspective for each item because something has to go but you can't decide what... Presented with the facts i.e. not achievable with current resources but realistic subject to the resources being made available, a project manager will either determine some way of addressing the risk (including ignoring it) or you will get your resources.

Be ready to provide evidence of your reasoning... but if your reasoning is sound you will have one less thing to worry about. Either you get your resources or, the project manager agrees for the item to be descoped from your work-stack... now you can focus on the agreed priorities and do a great job of what you do get to test.


h3. Traceable/Timebound

**Traceable/Timebound** - This depends on your priorities, but you can apply both...

**Traceable:** If you are dealing with documented inputs (regardless of the medium), and if you also need to demonstrate test-coverage of those inputs, then you will probably find that Traceable inputs make your life easier. The more maintenance you have to do to keep your tests reverse traceable to the input that it was derived from, the less time you are spending developing more tests. Particularly a problem with document-based requirements with auto-numbering used to identify each requirement. Add one requirement in the middle and all the subsequent numbers change.

**Timebound** This is very useful in terms of prioritisation. Time needn't be actual dates but can be tied into other date-bound milestones, such as 'iteration'. With this information, you are better able to determine the appropriate breadth (feature coverage) or depth (rigor) that is appropriate at a given stage of your project... allowing you to get the most effective use of your resources.


h2. So when is it testable?

Here are some guidelines that I use to help decide if an input is testable... these guidelines are not a substitute for your experience and knowledge, but they can help highlight areas at greater risk of being misunderstood or excluded from testing... Although they are guidelines, they are written in a pseudo-code/structured English type format, just because I find it an effective way of communicating them...

if **specific**=FALSE then ReviewEffortEstimate()
// it will probably take longer to extract the information you need

if **measurable**=FALSE then testable=FALSE
// get more information

if **achievable**=False then testable=FALSE

if **realistic**=false then JustForgetIt()

if **traceable/timebound**=FALSE then ReviewEffortEstimate()
// there will probably be an overhead in making
// the inputs traceable or maintaining traceability
// back to the input as its numbers change

**:ReviewEffortEstimate()**
{
if effort is greater than (timeAvailable/resources)*managementOverhead then achievable=FALSE
}

**JustForgetIt()**
{
return "no one in the world can do that, perhaps we can find out what your underlying objective is and determine a more realistic solution";
}

//THE-END