Survivor’s lessons in test management

No I don’t mean TV show about people living weeks on a beach. I somehow mean show on a discovery channel. I want to tell you what I learned from a project. It was supposed for 3-4 experienced testers, but I did it all by myself. I had to sacrifice all my beliefs of test management to do so. That’s why I admit I no more understand how to manage testing (well).

Story about self managing tester

Once upon a time I was managing team of almost 20 testers. I was sure I do the right things, a lot of the right things to improve the process. I changed testcase pass/fail criteria. I questioned our test case writing practices (which caused quite a conversation on that blog). I tried to make sense of measurements and a lot of other things. Until the project ended and I get to a new project supposed team of 3 skilled testers in a startup. There was the team. But for different reasons they all get reallocated. I lost them one-by-one but quite quickly. So when I was offered juniors instead I rejected: this would mean stop testing and start training/managing. This would in turn mean at least two months of serious slowdown of testing and I know that testing debt in the biggest risk to quality. So I come up with following suggestion: let me do testing all by myself. Let me save time on test documentation. I will write nothing, but bugs. I will keep everything in my head. I will communicate test progress and quality verbally/informally. They agreed! Today, I’m surprised about it. It should be the reputation that granted me a deal like this. So I worked hard to fulfill my promise. Of course I was overloaded with information. I wouldn’t be able to work long in such a way. A bit less than half a year living with overheated head and successful software release was enough for me to realize that I’m still good tester despite my project manager career shift.

So, what I’ve learned

First of all let me analyze what I was surprised about during this project:

Occasional regression bugs . I did regression tests only once, a week before release and only found a few bugs. However I found a lot of regression bugs occasionally – by testing other things, by preparing data for other tests, by doing careful testing around the bugs that came for retesting

Occasional coverage extension Sometimes I found also non-regression bugs. Because occasionally I did something wrong or different while preparing for next test. Or preparation requires something quite specific to be done.

Test prioritization on the fly (Ad Hoc) I trusted my instincts and developer’s suggestions on how deep should I test each feature. As I started testing new feature I did not stop until I know that I’m done testing it. Well sometimes I did more test the next morning as ideas pop up on a way home or work. But never ever I had any “if time permits” type of tests. I respected my time, so if I think of any test as: if time permits” I just don’t think about it at all.

Bug retesting time varies a lotBuy the way I also trusted my instinct when doing bug retesting. For some bugs I spent hours testing around, for other I only tested if the bug is fixed. For some… well let’s be honest – for some I blindly trusted developer and closed bug without any testing.

Not repeatable, so what? I was unable to repeat the tests that I did or describe them in details. However the cognitive experience was sitting down there in my head. Because as I tested new features, the ideas pup-up with ideas pop-up in my head about which old features I have to test integration with.

Actually there is only one lesson: no formal methodology could keep up with an expert mind. As I try to analyze how I find bugs and how I manage my time, it often seems occasional, or at least hard to explain even to myself. Almost impossible to analyze well .

To Be Continued: Luck - what does it have to do with testing

Not only with this project but other projects as well I’m surprised at how lucky I am. Really! Project Managers has asked me why didn’t we find that bug earlier. In a lot of cases analyzing the bug it appeared it’s a regression bug that is one or at most two builds old. Sometimes it appears we have already tested what stakeholder asks us to do additionally. It turns out quite often we have reported the bug that customers discover. Not always, but quite often, too often. I want to discuss what randomness and statistics has to do with testing in my next blog… If chance and risk are synonyms, then risk-based testing should not be too far from chance-based testing, should it?

P.S. The story in this blog is a few years old one. The software I’m talking about is still alive and used in several projects. I just needed a lot of time and more projects with real test teams to manage to make the right conclusions.