The conundrum of testability modifications
Submitted by Charlie Audritsh on Sun, 04/02/2007 - 22:51.
testability
In my current role I'm lucky enough to get a lot of support from development. They really care about the performance of the application and want to help me test it. So when I run into certain kinds of application related stumbling blocks -- a generated hash key that expires in 3 days as a security measure, after which that part of the app won't work with the old key by design, or a page with really a lot of IDs generated each time, that the server won't accept any that don't match upon submit, another security measure -- just to give two examples, developers often offer to take them out for me so I can test. "We can take that out so you can test, and we'll just put it back in before we ship." Woah. Red flag here. Wait. Stop.
Admittedly this is the kind of thing that tends to break automated tests, performance or otherwise, so it seems I can't test with it as it is, yes. But if this is taken out, then the functional test team is not testing the code that will actually be shipped/deployed/put into prod later. Then we're talking about making a special build just for me, which opens the door for version control issues, and then I'm not testing what will ship, though it seems I may not really have that choice anyway.
So what to do here? The way it works is very incompatible with testing it in an automated way, but changing that for testability means we're testing a version that customers will never see, or more to the point we won't be testing what they will see. The only option I see at this point is that if it's really important to test, is to put this risk on the table and raise awareness of it. If the risk is accepted, make the testability code change, do the special build and test, with everyone being aware that what we're testing is, in this one respect, not the code the customers will get.
I don't know though. This is a gray area that I, as a tester, am not comfortable with. I need to think this through some more. Does anyone here at testing reflections have any thoughts on this, or know of anything that's been written that explores these questions further?
Admittedly this is the kind of thing that tends to break automated tests, performance or otherwise, so it seems I can't test with it as it is, yes. But if this is taken out, then the functional test team is not testing the code that will actually be shipped/deployed/put into prod later. Then we're talking about making a special build just for me, which opens the door for version control issues, and then I'm not testing what will ship, though it seems I may not really have that choice anyway.
So what to do here? The way it works is very incompatible with testing it in an automated way, but changing that for testability means we're testing a version that customers will never see, or more to the point we won't be testing what they will see. The only option I see at this point is that if it's really important to test, is to put this risk on the table and raise awareness of it. If the risk is accepted, make the testability code change, do the special build and test, with everyone being aware that what we're testing is, in this one respect, not the code the customers will get.
I don't know though. This is a gray area that I, as a tester, am not comfortable with. I need to think this through some more. Does anyone here at testing reflections have any thoughts on this, or know of anything that's been written that explores these questions further?