Exploratory VS Scripted =?= Investigate VS Validate
Submitted by Ainars Galvans on Mon, 13/11/2006 - 10:56.
exploratory testing | performance testing
The idea that was floating about in my head is shaped at last! Scott Barber described Investigation VS Validation phenomena in performance testing. He draws some references to James Bach Exploratory VS Scripted . However Scott insists “the relationship between investigation and validation in performance testing is fundamentally different from the relationship between investigation and validation in functional testing.” James was non-committal talking about his writing applicability to performance testing. So I assume responsibility myself.
Vocabulary
The first explanation of word explore I see in webster is: “to investigate, study, or analyze“. Do you see – the word investigate are there! As I search more for definitions I found that both are related to systematic process with a goal of gaining some information. Investigation is more about (scripted) inquiring. Exploring is more a search for something. If I become totally abstract I would say exploring try to find the correct questions to get the desired answers while investigation have some idea of questions and are interested in answers.
Reasons for talking about it
My reasons for talking about performance Investigation VS Validation is as described below. Why to you think management commit to performance testing? I believe it is typically the fear of performance issues in production. They want testing to Validate=make sure no problems will be there. Write down all tests to be performed without software and even hardware in mind (only with users in mind). Then test-and-fix, this should work, shouldn’t it? They don’t seem to realize the power of (and the need for) Investigation.
Exactly the same happens to the functional testing. Testers are there in project to Validate= find deviance from requirements to be fixed. Scripted testing is best suited for that. All scripts could be written from requirements. As far as I understand this is what James Bach are writing about – to expand this narrow vision/mission/goal of functional testing. This is what I’m battling against in my company.
My Experience: tools testing
I’ve come to realization that in my current project I’m using exploratory testing a lot to investigate the quality rather than validate it. Defects found and more defect forecast allows (managers) to make decisions with regards to resource balancing between fixing bugs and adding more features (or extending the schedule).
But it is due to agile context (not Agile, but agile as I realized talking with Antony Marcano, while applying to WOPR). I’m testing Tools. Let me try to define what I mean by Tools. First of all I’ve red a lot that there are no real difference in testing products (multiple yet unknown customers) and projects (one defined customer) other than need to test on different environments for the first one. Tools are Products developed to be used internally during Project implementation. The customers are known – company internal users, but the use cases and environments may vary depending on their customers. As a result each: functional scope, quality and time are negotiable and alternating. It is acceptable to drop features and allow implement them as custom code, it is OK to left defects if they are documented and workarounds exist, the schedule should however be synchronized with project schedules.
Foreword: more on oversimplified functional testing
I’ve blogged about the issues with performance testing simplification. Let me touch functional testing a little.
Testing in textbooks is typically oversimplified (except a few practical/empirical rather than theoretical ). For example login screen testing – any book using this as example talking about we need to verify that we have been logged in with the name we entered user name and that you’ve got all the privileges supposed? How about testing if copy-paste works to input fields, short-cuts (ctrl+v)? Could we copy and paste to another application the password typed it ?! I’ve typically seen that we should not be able to do that.
Another typical example – ATM (automated teller machine). Anyone suppose testing cases when expired card is entered? How about case when you want to get 100$ in cache and it is supposed that you get 4*20$ plus 4*5$ banknotes but machine is out of 20$ banknotes? Maybe you don’t want to get 20*5$ and additional confirmation is required?
Textbooks on testing concentrate on how to test a feature. I’ve yet to read any generic testing book concentrating on fact that there is much more features shipped along with what is specified. Well few include chapters talking about functionality controlled by OS or something like that. But It does not seem to be reflected a lot in other chapters. Or I’ve red wrong books.
Vocabulary
The first explanation of word explore I see in webster is: “to investigate, study, or analyze“. Do you see – the word investigate are there! As I search more for definitions I found that both are related to systematic process with a goal of gaining some information. Investigation is more about (scripted) inquiring. Exploring is more a search for something. If I become totally abstract I would say exploring try to find the correct questions to get the desired answers while investigation have some idea of questions and are interested in answers.
Reasons for talking about it
My reasons for talking about performance Investigation VS Validation is as described below. Why to you think management commit to performance testing? I believe it is typically the fear of performance issues in production. They want testing to Validate=make sure no problems will be there. Write down all tests to be performed without software and even hardware in mind (only with users in mind). Then test-and-fix, this should work, shouldn’t it? They don’t seem to realize the power of (and the need for) Investigation.
Exactly the same happens to the functional testing. Testers are there in project to Validate= find deviance from requirements to be fixed. Scripted testing is best suited for that. All scripts could be written from requirements. As far as I understand this is what James Bach are writing about – to expand this narrow vision/mission/goal of functional testing. This is what I’m battling against in my company.
My Experience: tools testing
I’ve come to realization that in my current project I’m using exploratory testing a lot to investigate the quality rather than validate it. Defects found and more defect forecast allows (managers) to make decisions with regards to resource balancing between fixing bugs and adding more features (or extending the schedule).
But it is due to agile context (not Agile, but agile as I realized talking with Antony Marcano, while applying to WOPR). I’m testing Tools. Let me try to define what I mean by Tools. First of all I’ve red a lot that there are no real difference in testing products (multiple yet unknown customers) and projects (one defined customer) other than need to test on different environments for the first one. Tools are Products developed to be used internally during Project implementation. The customers are known – company internal users, but the use cases and environments may vary depending on their customers. As a result each: functional scope, quality and time are negotiable and alternating. It is acceptable to drop features and allow implement them as custom code, it is OK to left defects if they are documented and workarounds exist, the schedule should however be synchronized with project schedules.
Foreword: more on oversimplified functional testing
I’ve blogged about the issues with performance testing simplification. Let me touch functional testing a little.
Testing in textbooks is typically oversimplified (except a few practical/empirical rather than theoretical ). For example login screen testing – any book using this as example talking about we need to verify that we have been logged in with the name we entered user name and that you’ve got all the privileges supposed? How about testing if copy-paste works to input fields, short-cuts (ctrl+v)? Could we copy and paste to another application the password typed it ?! I’ve typically seen that we should not be able to do that.
Another typical example – ATM (automated teller machine). Anyone suppose testing cases when expired card is entered? How about case when you want to get 100$ in cache and it is supposed that you get 4*20$ plus 4*5$ banknotes but machine is out of 20$ banknotes? Maybe you don’t want to get 20*5$ and additional confirmation is required?
Textbooks on testing concentrate on how to test a feature. I’ve yet to read any generic testing book concentrating on fact that there is much more features shipped along with what is specified. Well few include chapters talking about functionality controlled by OS or something like that. But It does not seem to be reflected a lot in other chapters. Or I’ve red wrong books.