Repeatable tests, not repeatable bugs.
Submitted by Ainars Galvans on Wed, 03/01/2007 - 11:22.
context-driven testing | functional testing | test techniques
There are bugs that sit still waiting for you to catch them. There are flies that could only be killed with proper skills/techniques/tools.
This is the first issue in series of mini-experiences – learning opportunities I’ve recently described . It is typically mentioned that tests should be designed repeatable, so that if any defect is found it could be easily repeated. Nevertheless I’ve seen recently numerous blogs mentioning not repeatable bugs and casualties caused. However it seems the only message is to fix them, not how to make them repeatable.
Lecture course by Cam Kaner & James Bach talking about Oracle problem (slide 16) based on notes from Doug Huffman describe besides intended inputs system under test is also influenced by a lot of different aspects such as Program/System state, configuration, and other activities taking place in your computer and network (especially if you test client-server).
There are no not-repeatable problems. It is simply our inability sometimes to repeat them. To understand what it the actual reason for a problem. And then we give up and ask developers to do debugging – add some code that will investigate the system state – in case the problems is related to it.
The experience – the problem
Let me explain context of my story first. I was testing content repository software. It is JSR-170 compliant, but it is not very important here, what is important that it is an advanced, tree-like data storage with multi-user access. Imagine file system with arbitrary number of folder and document properties. So I was testing security management features of this system and found a bug – sometimes invoking “Edit his Node Security” do not work. More over once it failed to open the security editing window once it will never ever open it until I restart the application. Although I was observing this issue quite regularly and once happened it is 100% repeatable within the same session, it was still not repeatable because I was unable to write down any scenario (which starts with a new session) that would in 100% lead to the error. While it happened quite regularly I was able to repeat tests again and again. I had a lot of hypotheses (for example it is related to editing both parent and child node security) but all of them were dismissed.
The only clue I had: it happened more frequently for nodes direct-child to the root one... I will tell you one more thing that is important. However you are lucky to know that it IS important – I never had an idea that it is. We are two testers using the logical content repository. It means I see the data he have created and VV. But everyone of us is editing only a data entered by himself.
How I solved this
When I’ve analyzed the context described above I eventually come up with additional tests (a few of them). One of them repeated the problem 100% of the time. I suggest you to try to design your tests and submit them here are comments. I will later (depending on responses) tell you what was my tests and what was the scenario to repeat the problem. I will also tell everyone who have commented if their tests would succeed (if not too much of them:).
Feel free to ask clarification questions – English is my third language so I assume you may misunderstand anything...
This is the first issue in series of mini-experiences – learning opportunities I’ve recently described . It is typically mentioned that tests should be designed repeatable, so that if any defect is found it could be easily repeated. Nevertheless I’ve seen recently numerous blogs mentioning not repeatable bugs and casualties caused. However it seems the only message is to fix them, not how to make them repeatable.
Lecture course by Cam Kaner & James Bach talking about Oracle problem (slide 16) based on notes from Doug Huffman describe besides intended inputs system under test is also influenced by a lot of different aspects such as Program/System state, configuration, and other activities taking place in your computer and network (especially if you test client-server).
There are no not-repeatable problems. It is simply our inability sometimes to repeat them. To understand what it the actual reason for a problem. And then we give up and ask developers to do debugging – add some code that will investigate the system state – in case the problems is related to it.
The experience – the problem
Let me explain context of my story first. I was testing content repository software. It is JSR-170 compliant, but it is not very important here, what is important that it is an advanced, tree-like data storage with multi-user access. Imagine file system with arbitrary number of folder and document properties. So I was testing security management features of this system and found a bug – sometimes invoking “Edit his Node Security” do not work. More over once it failed to open the security editing window once it will never ever open it until I restart the application. Although I was observing this issue quite regularly and once happened it is 100% repeatable within the same session, it was still not repeatable because I was unable to write down any scenario (which starts with a new session) that would in 100% lead to the error. While it happened quite regularly I was able to repeat tests again and again. I had a lot of hypotheses (for example it is related to editing both parent and child node security) but all of them were dismissed.
The only clue I had: it happened more frequently for nodes direct-child to the root one... I will tell you one more thing that is important. However you are lucky to know that it IS important – I never had an idea that it is. We are two testers using the logical content repository. It means I see the data he have created and VV. But everyone of us is editing only a data entered by himself.
How I solved this
When I’ve analyzed the context described above I eventually come up with additional tests (a few of them). One of them repeated the problem 100% of the time. I suggest you to try to design your tests and submit them here are comments. I will later (depending on responses) tell you what was my tests and what was the scenario to repeat the problem. I will also tell everyone who have commented if their tests would succeed (if not too much of them:).
Feel free to ask clarification questions – English is my third language so I assume you may misunderstand anything...