Is there such a thing as a bug?

Note: Title Changed – see comments below I have had this niggling thought bouncing around in the back of my mind for a while now and I am no longer sure that it is necessary to always record bugs as bugs. In fact, I think that in a lot of situations, it may actually be innefficient. Perhaps instead of a bug, a reusable test can be recorded, taking a test-driven approach to managing software problems…

Some die-hard testers and other test-infected individuals will want to burn me at the stake based on the title but I hope I can clarify my thinking before it gets to that.

Whatever your initial reaction – I hope you can put your preconceptions to one side briefly while I explain why I think that there is no such thing as a bug…

Before I continue, understand that this is something of a brain-dump – there are many details to iron out and I am not saying that traditional approaches to bug-tracking are wrong, just proposing an optional alternative. A more formal paper may fall out of these thoughts but for now it is all a little rough around the edges…

You have some means of recording tests. It might be a spreadsheet or a test-tracking/management tool. In this repository you store information about each test and associate PASS or FAIL against each test. A test fails and you then log a separate record as a bug, often in a separate application.

Perhaps if you have an undocumented test, such as with exploratory testing or you observe some anomalous behaviour whilst performing a different test, you still log a bug.

Either way, the bug is thought of as a very separate entity to the test and is recorded as such. It is recorded, often, in negative terms of what shouldn’t have happened rather than leading with what should happen.

In my mind, a bug is really just a failure-instance of a test. I.e. a period of time in which the test is in a ‘Fail’ state. Whether that test was pre-planned or designed and executed on the fly – it is still a test. I am suggesting that at this point, the repeatability of the test carries some value.

But what if you log a bug as a result of an undocumented test, say, during an exploratory session? What if you observed a bug while running a test that actually passed? What if it was a support incident?

I still say that it is still just a failure-instance of a test. I will explain these scenarios a little more and how my way of thinking fits into these scenarios and why I believe we may be, in many situations, be handling bugs in completely the wrong way.

Certain integrated test-tracking tool-suites allow you to quickly create a new bug record with a reference to the test, automatically duplicating information from the test. Some stand-alone bug tracking tools just provide a means of logging a bug as a unique entity.

Consistently, however, they all conceptualise bugs as a very separate entity to the test. This isn’t just tools-vendors forcing a process on us. This is how we have expected them to work.

Let’s imagine a world where there is no such thing as a bug-tracking tool and where the word ‘bug’ doesn’t even exists. I want to put that word aside for now. Instead, I am going to use some other terms instead:

• Failure: Deviation of the software from its expected delivery or service.
• Fault: A manifestation of an error in software. A fault, if encountered may cause a failure

So here are some common scenarios but handled subtley differently than we are used to…

Scenario 1: Documented test fails

Let’s assume your manual and automated tests are tracked in a test-management tool. Let’s assume that the tool provides you with a way of logging results of manual tests as well as automated ones and presents them as a consolidated list of passes and failures.

The Test Manager assigns a batch of tests to you, setting their status to ‘Ready to Execute’. This status change is logged automatically as part of the history of the test.

At the beginning of the test-run you log configuration information about the system under test so you won’t have to repeate that information as you Pass or Fail each test.

While executing your tests, one fails. Perhaps it is a manual test so you hit the ‘Fail’ button next to the test-record in your test-tracking suite. This changes the status from, say, ‘In Progress’ to ‘Failed’. You then have to record details in the audit trail for this status change and you are presented with some status-specific fields for this audit-trail entry:

• Failure summary – a short description
• Failure detail – stack traces and any other detail

• Attachments – speaks for itself

  This audit-trail information is captured and the test is assigned to the programmer (yes the whole test record with its associated audit trail). I have skipped assigning to the project manager or team leader for prioritisation in order to keep my explanation simple.

  If this was an automated test, the actuals could be passed back from the automated script and the history entry created automatically. You might want the tester to double check these failures for false-negatives before sending to the developer, so perhaps automated tests might have an intermediary state ‘Automated Fail’ before being promoted to ‘Failed’.

  Once assigned to the programmer, it is their responsibility to make the test pass by fixing the code. When they believe that the test passes they can change the test to ‘Fixed’ and, in the history for that change they record the detail of where the fault was in the code and what they did to fix it. This test is then batched up with all the other tests to be included in the next test-run.

  A simplified illustration of the resulting record could be (don’t be fussy about the test-template used – it is intentionally simplified):

  Test ID:	T0001
  Test Objective	Verify maximum field lengths don’t exceed 255 characters
  Inputs:	A string containing 255 chars. Attempt to enter 256th character
  Expected Result:	A message is displayed informing the user that the field only accepts 255characters
  Current Status:	Passed
  Last executed on version:	3.1.2565
  Assigned to:	Antony
  History:
  Status	Date	Notes	Who
  Passed	26-01-06	Annotation: That’s more like it!	Joe
  Fixed	25-01-05	Changed val = maxLen to val == maxLen in FormFactory.dataValidator	Silvie
  Failed	24-01-05	Failure Summary: It is possible to enter text of any length into all text fields	Joe
  Ready to test	24-01-05	Version: 3.2.2928; tserver01	Fiona
  Approved	23-01-05	Include in next test-run	Fiona
  New	23-01-05	——	Antony

  The history can, of course, get quite long. Perhaps the test-tracking tool will only show a few of the most recent entries in the history log and allow the user to expand the list as necessary. It may also be possible to drill down into more detail for each history item. Who knows, I am not trying to design the implementation of this concept right now, just trying to illustrate my thinking through examples.

  The key here is that you aren’t passing around a bug (did I just use the ‘bug’ word) you are passing around a test. The test just happens to be a bug at a given point in time, while it is in a failed state.

  Scenario 2: Undocumented Test Fails (e.g. during an exploratory session)

  You are in an exploratory testing session and find some anomalous behaviour in the system. Perhaps you make the observation while executing a documented test but it doesn’t constitute a failure of that test. Maybe the behaviour was observed in the wild by a user. Instead of calling up your ‘bug tracking tool’ (because they don’t exist in this imaginary world) you log a new test in the system. This test-record would look just like the test in Scenario 1 (above) but it would have an initial status of ‘Failed’. It would then be treated like any other test and go to the developer to make it pass.

  It is perfectly acceptable to document verification tests as part of exploratory testing (as in James Bach’s General Functionality and Stability Test Procedure ). This would be a verification test for the version that is released to testing with the necessary fix for that previously failed test. It may not remain as as an ongoing verification test because once you have confirmed it to be closed, the value in repeating it may be diminished. Despite this, you would still be left with an explicit, repeatable test in the tester’s library for use (or not) at some appropriate time in the future (Discussing when to repeat tests is beyond the scope of this article).

  Scenario 3: The Test is Wrong and it actually passed

  This is handled much as in Scenario 1, except it turns out that the failure is incorrect because the test was wrong. The test is rejected back to the tester who edits the test accordingly and passes it. Perhaps the test is entirely invalid – so the test is closed. It is possible that the feature has been depreciated – and so, depreciate the test.

  In Conclusion

  OK, I admit that I made the statement that there is ‘no such thing as a bug’ to get your attention. There is such a thing as a bug but it isn’t this divorced entity that is ‘loosly coupled’ (for want of a better phrase) to the original test, in my opinion.

  One concern with this be an expectation to keep the tests up to date. I don’t think this is necessarily an issue. The test only needs to be maintained if you are going to use it again after a given time. You would have lost nothing at this point because you would have had to record a bug anyway. The bug information would have been reused at least once to retest it. What you would have gained is a test that is reusable – if it is deemed appropriate.

  In this approach, I want you to see the term ‘Bug’ as meaning ‘a test that fails against the current version of the system’. In BS7925-1 or the online living glossary based on the same original draft , ‘Bug’ is just another term for ‘Fault’ which is actually the part that is wrong in the code or configuration. Despite this, most people seem to use the term in the context of the above definition of a ‘Failure’ – some invalid or problematic behaviour of the system.

  Where a documented test fails, I believe that the typical bug report is often documentation for documentation-sake. Instead of all that effort to log the bug as a separate entity, we could just register a failure against the test. If the test was undocumented, just register a new test with a failure. There is a lot more re-use opportunity with a test than an isolated bug record. Why retest your bugs from the bug database and then execute tests from a test-database with two different user interfaces when you can easily do them all from the same place and present them in much the same way.

  There are a great many opportunities to integrate this into the development life-cycle, particularly if you are using Extreme Programming (or similar agile process) which I won’t explore here.

  Regardless, I still agree that a bug is anything that bugs someone who matters but I would like to add this approach to my arsenal of approaches that I can use in a given context. In this approach, the bug-record is the (failed) test and the (failed) test is the bug-record. They aren’t separate entities. So, when I say that there is no such thing as a bug, I don’t intend for you to take it literally.

  For now we may have to make do. My current client needed a test-tracking tool and I suggested using a modified version of the bug tracking tool in the interim. There was talk about pushing test failures out of the test-tracking instance to the bug-tracking database but I suspect that they may consider my more integrated approach. Early indicators are that the bug-tracking tool (a very flexible one) can be used reasonably effectively in this way, thanks to two of the guys in the team spending some ‘spare’ time setting it up.

  There, this is only an interim solution until they can get in a more powerful test-management tool. But, by seeing the bug-tracking tool used in this way and setting up the workflow, it has further highlighted the potential redundancy in modeling bugs and failed-test-results as two very separate concepts.

  Time will tell

  Happy bug-hunting!