Sweep Testing
Submitted by Karen N. Johnson on Tue, 20/11/2007 - 18:56.
I have my own style of exploratory testing (ET) I’ve been using for years. I thought I’d use this blog entry to outline how I test.
I find it interesting that no one has asked me how I test when people email me and ask me all sorts of other questions. Perhaps people assume I execute exploratory testing as James has defined and honed ET for the past decade.
Perhaps my nature of testing and exploring an application isn’t different from ET – from a mentality point of view. I make this statement based on the concepts of testing an application by use of exploring versus following prescriptive test scripts. But my test ideas are often not recorded in charters.
Historically when I’ve had an application to test, I start building my ideas in Excel. I don’t use any of the Excel functions per se, I just like the grid view for organizing my thoughts and in all reality, creating tables in Word is too much of a pain. I like that Excel forces me to be brief; I could create longer entries in cells but I generally don’t.
I create a separate worksheet for each area I’m going to test. And this, I think is important. I see each worksheet as a blank canvas. I might list ideas, build a chart, I might cut and paste key requirements that I want to test (not too many because I don’t like duplicating efforts – so if I have many requirements I want to work around, I might reference the pages of a requirements doc etc.) Each worksheet can have its own format because each worksheet and area to test has its own story.
When I sit back and think, I often can’t restrict my test ideas to forming only in one area at a time. Nor would I try. My ideas roam and this is why I like the workbook format of Excel. I move from worksheet to worksheet to capture test ideas. I record ideas before, during, and after testing. I also keep a notebook when I test. I record different things in my workbook and different things in my notebook – I’ll blog about note-taking another time.
To get ideas, I generally do three things: I pass through any documentation I have, I talk to people on the team asking questions and I stare out the window sometimes muttering to myself looking crazy I’m sure. I don’t care. I get ideas from all three avenues.
The next thing I do that I believe is different from ET is what I would call “sweep testing.” When I test, I test in one area at a time and explore. But I never expect that I’ll hit all my ideas on one session. In fact, I expect my first and second sessions in each area will be the most beneficial – the best bug finds. I sweep through an area. I look for bugs. I expect to return to each area for another sweep. The second sweep is helpful because by then I have a feel for the build and a feel for different areas of the application.
A benefit of sweep testing is I find that sometimes I’m distracted by my first test ideas, so I test those ideas first to get them out of my head and see what I can find. I add thoughts to my worksheets. I don’t capture ideas that turned out fruitless. I don’t write test doc to cover my assets, I write what I want to recall. I write what I learn. I write notes. My worksheets seem to be pretty readable by other testers but there are few full sentences and my test ideas are far from test scripts.
By executing testing in rounds or sweeps it helps me to test wide (meaning many areas) and then move back to each area to go deep (meaning more detailed test ideas). I focus on one area at a time, unlike when I’m collecting ideas – and this sense of focus and digging in, exploring and learning is the value of ET and I believe is essential to thoughtful testing.
I’ve used the Excel workbook for years, it’s been effective. It works for me. Sometimes, I write charters but I have to say, I like the workbook because it allows me to have one file to capture many ideas. I roll the version of the workbook when I get a new build or make a significant update to the file. It depends if I’m working alone as to how meticulous I am about file versioning - if I’m alone, then I don’t worry about file versions as much. And I have often worked alone preferring small software companies where the value of what I bring to the product team isn’t measured by metrics or detailed test documents.
Is this ET or not, I have never worried. I have listened to and known James for years, I have deep respect for what he has brought to our field. The value of ET in my mind is the thinking not the documentation. After all why replace prescriptive test scripts only to fret about a different format of documentation? I've given myself permission to be logical and practical and not fret over form. I’ve been sharing more and more of what I do and how I do it so that I can help other testers. So these are thoughts around how I test. I have sincere hopes this helps testers learn. More to come -
I find it interesting that no one has asked me how I test when people email me and ask me all sorts of other questions. Perhaps people assume I execute exploratory testing as James has defined and honed ET for the past decade.
Perhaps my nature of testing and exploring an application isn’t different from ET – from a mentality point of view. I make this statement based on the concepts of testing an application by use of exploring versus following prescriptive test scripts. But my test ideas are often not recorded in charters.
Historically when I’ve had an application to test, I start building my ideas in Excel. I don’t use any of the Excel functions per se, I just like the grid view for organizing my thoughts and in all reality, creating tables in Word is too much of a pain. I like that Excel forces me to be brief; I could create longer entries in cells but I generally don’t.
I create a separate worksheet for each area I’m going to test. And this, I think is important. I see each worksheet as a blank canvas. I might list ideas, build a chart, I might cut and paste key requirements that I want to test (not too many because I don’t like duplicating efforts – so if I have many requirements I want to work around, I might reference the pages of a requirements doc etc.) Each worksheet can have its own format because each worksheet and area to test has its own story.
When I sit back and think, I often can’t restrict my test ideas to forming only in one area at a time. Nor would I try. My ideas roam and this is why I like the workbook format of Excel. I move from worksheet to worksheet to capture test ideas. I record ideas before, during, and after testing. I also keep a notebook when I test. I record different things in my workbook and different things in my notebook – I’ll blog about note-taking another time.
To get ideas, I generally do three things: I pass through any documentation I have, I talk to people on the team asking questions and I stare out the window sometimes muttering to myself looking crazy I’m sure. I don’t care. I get ideas from all three avenues.
The next thing I do that I believe is different from ET is what I would call “sweep testing.” When I test, I test in one area at a time and explore. But I never expect that I’ll hit all my ideas on one session. In fact, I expect my first and second sessions in each area will be the most beneficial – the best bug finds. I sweep through an area. I look for bugs. I expect to return to each area for another sweep. The second sweep is helpful because by then I have a feel for the build and a feel for different areas of the application.
A benefit of sweep testing is I find that sometimes I’m distracted by my first test ideas, so I test those ideas first to get them out of my head and see what I can find. I add thoughts to my worksheets. I don’t capture ideas that turned out fruitless. I don’t write test doc to cover my assets, I write what I want to recall. I write what I learn. I write notes. My worksheets seem to be pretty readable by other testers but there are few full sentences and my test ideas are far from test scripts.
By executing testing in rounds or sweeps it helps me to test wide (meaning many areas) and then move back to each area to go deep (meaning more detailed test ideas). I focus on one area at a time, unlike when I’m collecting ideas – and this sense of focus and digging in, exploring and learning is the value of ET and I believe is essential to thoughtful testing.
I’ve used the Excel workbook for years, it’s been effective. It works for me. Sometimes, I write charters but I have to say, I like the workbook because it allows me to have one file to capture many ideas. I roll the version of the workbook when I get a new build or make a significant update to the file. It depends if I’m working alone as to how meticulous I am about file versioning - if I’m alone, then I don’t worry about file versions as much. And I have often worked alone preferring small software companies where the value of what I bring to the product team isn’t measured by metrics or detailed test documents.
Is this ET or not, I have never worried. I have listened to and known James for years, I have deep respect for what he has brought to our field. The value of ET in my mind is the thinking not the documentation. After all why replace prescriptive test scripts only to fret about a different format of documentation? I've given myself permission to be logical and practical and not fret over form. I’ve been sharing more and more of what I do and how I do it so that I can help other testers. So these are thoughts around how I test. I have sincere hopes this helps testers learn. More to come -