How to teach grey-box testing?
Submitted by Ainars Galvans on Wed, 18/06/2008 - 09:37.
general software testing
When I’m teaching new testers I use (specific chapters from) Black Box Software Testing course by Cem Kaner and James Bach. Those are great materials. However there are some topics not covered to an extent that I would like to (probably because of the specific of software we are creating in our company). For instance multi-edit heuristic and environment-specific handling tests. Maybe those topics are more in grey-box than in black-box, but for sure they are not the only topics of grey-box...
Clarification and Example of (example exam question)
Let’s suppose the classic AMT testing. I have a card associated with an account with a balance X. I want to test money withdrawal operation. What are the tests missed by typical black-box test?
Let’s first think about multi-user cases. What are all the ways to change the amount of my account? Some person may transfer me some money. Maybe my wife knows how to access my account from internet and transfer some money away from it at the same moment when I’m withdrawing all the money I have.
Next - environment issues. Could my account (associated with the valid card) be locked or otherwise compromised? If yes I should test how it affects ATM.
What if I withdraw money in a country having different currency?
How about withdraw from a location having different time zone? Who care? Well… how about printing the report of withdrawals with timestamps at the end of the month? How about sorting all account transfers by time, including transfers done from this region?
How about running out of money in ATM case?
How about testing different ATMs? I know there are at least two modifications in Latvia – the simple one and with protective cover that opens only when you insert a valid card.
My Exam Questions
Below I’m publishing the first draft of an exam questions for the topics described above. The questions are hard and complicated (comes from my experience) and it is assumed that you have to use a computer (search the net, etc) for investigation before you have an answer.
If anyone could suggest more good cases please add a comment here or e-mail me.
Q1. Suppose you are testing a defect tracking system which is supposed to be used for open-source projects, having teams spread across the world. This creates challenge to correctly store, display and filter by date-time fields, such as created on, closed on, etc.
You have to design tests that address specifically those challenges.
Q2. Suppose you have to test server application that supports load balanced and fault tolerant execution of scheduled activities. For instance if we have 3 hosts included in load balancing and there are 10 “jobs” scheduled at 12:00, then 3-4 jobs should be executed on each host and if any host hangs up before jobs are completed they are redistributed to other hosts.
What additional tests could you design based on following knowledge: the scheduled "jobs" are stored in a single table in SQL database. Each server servers participating in load balancing filter jobs by scheduled time; mark (set some flag to true for) a job entry in a table before execution and delete it upon succesfull execution.
Q3. Content Repository System is something like file system, but specially designed for multi-user access and with more functionality, such as security, versioning and adding more searchable properties. You have to test following features of Content Repository System: import from file system and export to file system. Suppose classic functional tests are executed and you have to address the other challenges, such as multi-user simultaneous usage, etc. A hint: your existing file system properties such as size, date created, etc. could also be exposed in the Content Repository System.
Q4. A workflow system is a system where you could define task sequence, declare task assignment to people involved and many more. Suppose a workflow system that supports task queues, i.e. there is a queue of unassigned tasks and any person could request a task out of a queue. Each person could prioritize tasks, i.e. one could ask for the most recent task, another for the most critical task. Security may restrict certain person certain task type(s) and could allow managers to “distribute” (assign from queue, reassign, send back to queue) tasks among personnel manually.
Suppose we have users all around the world connecting to a single web server to work with our system. Design tests to test task assignment and distribution functionality, given that the classic black-box function testing passed.
Clarification and Example of (example exam question)
Let’s suppose the classic AMT testing. I have a card associated with an account with a balance X. I want to test money withdrawal operation. What are the tests missed by typical black-box test?
Let’s first think about multi-user cases. What are all the ways to change the amount of my account? Some person may transfer me some money. Maybe my wife knows how to access my account from internet and transfer some money away from it at the same moment when I’m withdrawing all the money I have.
Next - environment issues. Could my account (associated with the valid card) be locked or otherwise compromised? If yes I should test how it affects ATM.
What if I withdraw money in a country having different currency?
How about withdraw from a location having different time zone? Who care? Well… how about printing the report of withdrawals with timestamps at the end of the month? How about sorting all account transfers by time, including transfers done from this region?
How about running out of money in ATM case?
How about testing different ATMs? I know there are at least two modifications in Latvia – the simple one and with protective cover that opens only when you insert a valid card.
My Exam Questions
Below I’m publishing the first draft of an exam questions for the topics described above. The questions are hard and complicated (comes from my experience) and it is assumed that you have to use a computer (search the net, etc) for investigation before you have an answer.
If anyone could suggest more good cases please add a comment here or e-mail me.
Q1. Suppose you are testing a defect tracking system which is supposed to be used for open-source projects, having teams spread across the world. This creates challenge to correctly store, display and filter by date-time fields, such as created on, closed on, etc.
You have to design tests that address specifically those challenges.
Q2. Suppose you have to test server application that supports load balanced and fault tolerant execution of scheduled activities. For instance if we have 3 hosts included in load balancing and there are 10 “jobs” scheduled at 12:00, then 3-4 jobs should be executed on each host and if any host hangs up before jobs are completed they are redistributed to other hosts.
What additional tests could you design based on following knowledge: the scheduled "jobs" are stored in a single table in SQL database. Each server servers participating in load balancing filter jobs by scheduled time; mark (set some flag to true for) a job entry in a table before execution and delete it upon succesfull execution.
Q3. Content Repository System is something like file system, but specially designed for multi-user access and with more functionality, such as security, versioning and adding more searchable properties. You have to test following features of Content Repository System: import from file system and export to file system. Suppose classic functional tests are executed and you have to address the other challenges, such as multi-user simultaneous usage, etc. A hint: your existing file system properties such as size, date created, etc. could also be exposed in the Content Repository System.
Q4. A workflow system is a system where you could define task sequence, declare task assignment to people involved and many more. Suppose a workflow system that supports task queues, i.e. there is a queue of unassigned tasks and any person could request a task out of a queue. Each person could prioritize tasks, i.e. one could ask for the most recent task, another for the most critical task. Security may restrict certain person certain task type(s) and could allow managers to “distribute” (assign from queue, reassign, send back to queue) tasks among personnel manually.
Suppose we have users all around the world connecting to a single web server to work with our system. Design tests to test task assignment and distribution functionality, given that the classic black-box function testing passed.