Testing Consistency
Submitted by Karen N. Johnson on Sun, 08/04/2007 - 16:39.
I’ve tested search functionality in a number of websites. One of my checkpoints is a consistency test. I locate all the locations in a website I can search from and then I run a search using the same search criteria and compare the results. If I search with the same criteria, do I get the same result s? Not always.
I find bugs when I check for consistency.
Check out this example.
If you access the front page of a well-known book selling website and query on “software testing,” you get a certain number of books in the result set. But if you create a particular type of list and then run a search on “software testing,” a smaller number of books are returned from the search.
I’m ok with getting a different count in search results or even a different search result set from the same search criteria as long as the UI gives me the reason. For example, tell me the search results will discard out of print books or the results will only show books available for immediate shipping. I need the reason for any difference in results or I expect consistent results.
The most troubling issue with seeing inconsistent search results is I lose some faith in the website. I start to think: Is search broken? Can I trust searching from one screen but not another? Isn’t the SQL behind the search the same? Why is the SQL different? And why didn’t the UI tell me why I should expect a difference in the result set?
It’s kind of funny that from my perspective - the result set might be wrong and the SQL might have an error but that getting consistent results is reassuring in some sense. It adds to my belief that when a search defect is resolved, it will be resolved everywhere in the application.
So what’s the WIIFM in this entry? My suggestion is to look for opportunities to check for consistency in the website you’re testing. Consistency builds faith.
I find bugs when I check for consistency.
Check out this example.
If you access the front page of a well-known book selling website and query on “software testing,” you get a certain number of books in the result set. But if you create a particular type of list and then run a search on “software testing,” a smaller number of books are returned from the search.
I’m ok with getting a different count in search results or even a different search result set from the same search criteria as long as the UI gives me the reason. For example, tell me the search results will discard out of print books or the results will only show books available for immediate shipping. I need the reason for any difference in results or I expect consistent results.
The most troubling issue with seeing inconsistent search results is I lose some faith in the website. I start to think: Is search broken? Can I trust searching from one screen but not another? Isn’t the SQL behind the search the same? Why is the SQL different? And why didn’t the UI tell me why I should expect a difference in the result set?
It’s kind of funny that from my perspective - the result set might be wrong and the SQL might have an error but that getting consistent results is reassuring in some sense. It adds to my belief that when a search defect is resolved, it will be resolved everywhere in the application.
So what’s the WIIFM in this entry? My suggestion is to look for opportunities to check for consistency in the website you’re testing. Consistency builds faith.