Grey-box techniques for excluding tests

Don’t test the reused code tested previously – you will have more time to test new code. This heuristic for a black-box tester is basically all the idea of this blog – one I was creating for a week … I started this as blog about techniques, but wanted to be clear about grey box. A research on the term ended quite a long story below. If you don’t care about it – scroll down to the last chapter describing the techniques.

What is Grey-Box testing and what it is not?

I believe this discussion on QAForums perfectly reflect the situation in industry. I would dismiss all of the most prevailing answers: a) combination of black-box and white box b) testing the API level using “black-box approach”; c) partially interacting with system in a way other than UI (e.g. writing data directly to database, updating program variables from a debugger, etc.). This is how I see grey box testing. I will need to step back beforehand and redefined black and white boxes. We know that there are too much tests to do so there are some techniques developed to select a small, but good enough sub-set. There was an attempt (academic I believe) to classify the techniques: those that use only specification to decide what tests to select and those that also (yes, ALSO) use code. The issue is – it appears that in real life we use other techniques which are based on other types of information – see the list below. So

If you know how Operational System (OS) configuration impact our application we could run tests with the same inputs for different configurations

If we know how multiple user activities and processes may affect each other we may run tests with the same inputs in different conditions

The past experience (typical bugs or confidence of their absence) with this specific programming language/technology, by the developer, etc. may suggest to ignore testing specific input values

Quality goals/expectations, the (business and technical) risk identification, etc. may also suggest that for specific features not all the identified tests (input values) are to be executed, or at least with less priority

Who Cares?

I’m participating tester interviews for years. I never use black/white box terminology in interview (unless other interviewers or interviewee mention it). Because it doesn't matter whether a cat is black or white so long as it catches mice . Any experienced tester uses ideas going beyond black-box testing techniques, even if they never classify them. For example, last year I analyzed black-box technique equivalence partitioning. Let me quote myself

Given that I know that fields for data entry they use as standard ones (tested already for invalid data entry) and data is saved in Oracle DB and retrieved using SQL (where statement), so there are no sense to test the correct ordering of date, only the where statement generation.

When designing black-box tests I used my knowledge of system internals.

I care… when train testers

Yes I don’t care if experienced tester could classify the techniques, only if she/he is able to find the bugs fast enough. Just as I don’t care if they know English (which is not the first language for us) grammar rules – only if they could write right. However the only approach to teach them to write right that I know of is to teach the rules. That’s how teachers do it. Same do I when training testers: I teach them classified techniques, although I agree with Daniel Suciu QAForums Moderator

here is no pure white or black box testing... only different nuances of grey”

So when I’m training testers in our company I’m using among other resources the first lecture of BBST course by Cem Kaner and James Bach (which I use) where slide 17 recommends beside intended inputs to consider also Program State, System State, Configuration and System Resources, influence by cooperating processes, clients and servers. You could find more about web configuration issues in an article by Hung Q. Nguyen or I believe in his book mentioned above. For this reason I classified in my previous blog some technique and some questions about how to test additionally for different states, configurations and cooperation. What I’ve left to say – that there are ways to select less to test not only more to tests. You could select less to test knowing what code is reused and tested already.

Grey box techniques for excluding tests

It is essential to know how to test less (not only how to test more), because you will have more time to tests more powerful tests, right? So how to reduce number of tests based on code and/or architecture knowledge? I was only able to articulate one heuristic: consider if this code have been tested already? For example if your application rely on database „sort by” to sort items you would do less tests for sorting compared to case when programmer have coded his own sorting algorithm inside UI. Or if you know that the calendar control used in an application is a 3rd party control used in 3 previous projects all in production form months. I believe I could skip testing Jun 31st and July 32nd, couldn’t I? If I know that in-house developed table control is used in multiple places in an application and I’ve carefully tested it in one screen – should I tests it carefully again on another screen?