Predicting the Path of the Storm
Submitted by David Gilbert on Tue, 27/06/2006 - 18:04.
design & development | exploratory testing | general software testing | project management | test management
Recently, three unrelated events have given birth to a new idea in my head, and I wanted to share it with you. So, to properly set up the background, the three unrelated events:
At the recent CAST, James Bach presented a keynote, “Against Certification”. During that presentation, one of the things he did was to review some of the “Body of Knowledge” documentation upon which such certifications were based. As he reviewed this documentation, one of the things I was struck by was an underlying pattern of motive driving much of the documentation. That motive was the ability to predict, manage, and control the SDLC. Much of this “Body of Knowledge” for test certification was obviously written by managers, not testers.
In a recent blog, Michael Bolton talked about estimation and prediction for rapid testing efforts in projects. I thought it was a very good blog, and agreed with everything in it, but in the end it left me with a sense of a lack of resolution, and that bothers me.
I live in the Tampa Bay, Florida area. We recently had our first storm of the hurricane season. After the last few years, we who live here watch keenly the daily, and sometimes hourly, updates to the tracking models and projected paths of these storms.
So what do all of these events have in common? They all deal with trying to predict the progress and ultimate resolution of a process that is full of risk and ambiguity. But that’s not really the interesting part. What I find fascinating is that only one of them has a universally accepted solution…the hurricane trackers.
Now, when I say universally accepted, make no mistake…not everyone likes the model that is used. Many people will tell you they think weathermen may as well throw darts at the map as do what they do. What I mean is that it is used consistently across the industry, with very, Very, VERY little variation from one specific implementation to the other. It is very flawed, but it is good enough that any reasonable professional in that industry understands it is a tool, and when used properly, it is a tool that provides good value, and so they use it appropriately. That fascinates me…why can’t we reach that kind of consensus about estimating the development and testing process?
After thinking about this for awhile, I believe there are some concrete elements in the hurricane tracking models that are not present in any estimation model I have ever seen in software; and I believe it is the existence of these elements that make that model acceptable. Lets look at some of them.
First, hurricane tracking models always start from the current position of the storm. While they track the history of the storm for historical purposes, where the storm went yesterday has little to no bearing on where it will go later today, other than having put it into it’s current place. And more to the point, where we expected, or planned, for it to go yesterday, last week, last month, has absolutely NO bearing on where it is now, or where we expect, or plan, for it to go tomorrow. It does not even enter into the conversation. So the plan gets reset based on reality at a given time. No one holds onto the past.
From where it is today, the second element of the model is a path of where we expect it to go, and that path ultimately leads to an expected location and time for landfall, the ultimate result of tracking the storm. That path is based on many factors…how fast the storm is making progress, which direction it is going, where the prevailing winds are, what the temperature of the water is , how much time there is between now and landfall, etc. All of those factors are highly variable, and largely out of control of the hurricane trackers. This is the single element of the model that most resembles the current models we use for estimating development and testing efforts…a straight line, or at best a gantt chart, plotted against a calendar, with the end date carved in stone. Interestingly enough, this element of the model is the single element that most forecasters trust the least, and they all say so loudly.
Which brings us to the third element of the model, and I think this is the biggie. The Cone of Uncertainty. Starting at the point where the storm is now, following roughly along the predicted path of the storm, is an ever expanding cone. This cone represents where, given all the information currently available, the storm MAY go if something in the model changes. I have NEVER seen this in any model of software planning. Yet this is the part of the model that forecasters talk about the most, and encourage everyone to pay the most attention to. This is the part of that model that takes into account Risk and Ambiguity.
I think these elements are powerful in terms of creating a model that deals largely in truth, acknowledges uncertainty, visually depicts it’s status and communicates clearly, and updates rapidly and understandably. And I think it is a model we could emulate to predict, in a reasonable and acceptable manner, the software testing process, even when using rapid / exploratory testing.
It is the test teams job to provide information to management to better enable them to make decisions about releasing software. It is managements job to keep the software development and creation process on track, and release software by balancing a vast array of considerations and risks. While neither of us can know everything, or plan everything, in advance, the test team can also not simply throw up our hands and say “We cannot provide any answer to the questions your asking.” when the questions they are asking are perfectly valid. At least not if we want to be perceived as a team player adding value to the process. But a model such as this, that plots not only a course, but a reasonable expectation of possible variation around that course, and updates itself regularly, would let management see not only where the project is, and where it is headed, but it would also better help them understand the costs and risks with getting it back on track versus accepting the new point of impact. I believe that would be a far better answer than “We will test all we can until the day you shove it out the door.”
I am so intrigued by this, that I am going to begin working on such a model. I will be happy to publish both the ongoing work, and the final results, online. If you would like to collaborate on this with me, drop me a line at dgilbert@sirius-sqa.com.
In the meantime, batton down the hatches, and good luck weathering the storms of your own testing efforts.
At the recent CAST, James Bach presented a keynote, “Against Certification”. During that presentation, one of the things he did was to review some of the “Body of Knowledge” documentation upon which such certifications were based. As he reviewed this documentation, one of the things I was struck by was an underlying pattern of motive driving much of the documentation. That motive was the ability to predict, manage, and control the SDLC. Much of this “Body of Knowledge” for test certification was obviously written by managers, not testers.
In a recent blog, Michael Bolton talked about estimation and prediction for rapid testing efforts in projects. I thought it was a very good blog, and agreed with everything in it, but in the end it left me with a sense of a lack of resolution, and that bothers me.
I live in the Tampa Bay, Florida area. We recently had our first storm of the hurricane season. After the last few years, we who live here watch keenly the daily, and sometimes hourly, updates to the tracking models and projected paths of these storms.
So what do all of these events have in common? They all deal with trying to predict the progress and ultimate resolution of a process that is full of risk and ambiguity. But that’s not really the interesting part. What I find fascinating is that only one of them has a universally accepted solution…the hurricane trackers.
Now, when I say universally accepted, make no mistake…not everyone likes the model that is used. Many people will tell you they think weathermen may as well throw darts at the map as do what they do. What I mean is that it is used consistently across the industry, with very, Very, VERY little variation from one specific implementation to the other. It is very flawed, but it is good enough that any reasonable professional in that industry understands it is a tool, and when used properly, it is a tool that provides good value, and so they use it appropriately. That fascinates me…why can’t we reach that kind of consensus about estimating the development and testing process?
After thinking about this for awhile, I believe there are some concrete elements in the hurricane tracking models that are not present in any estimation model I have ever seen in software; and I believe it is the existence of these elements that make that model acceptable. Lets look at some of them.
First, hurricane tracking models always start from the current position of the storm. While they track the history of the storm for historical purposes, where the storm went yesterday has little to no bearing on where it will go later today, other than having put it into it’s current place. And more to the point, where we expected, or planned, for it to go yesterday, last week, last month, has absolutely NO bearing on where it is now, or where we expect, or plan, for it to go tomorrow. It does not even enter into the conversation. So the plan gets reset based on reality at a given time. No one holds onto the past.
From where it is today, the second element of the model is a path of where we expect it to go, and that path ultimately leads to an expected location and time for landfall, the ultimate result of tracking the storm. That path is based on many factors…how fast the storm is making progress, which direction it is going, where the prevailing winds are, what the temperature of the water is , how much time there is between now and landfall, etc. All of those factors are highly variable, and largely out of control of the hurricane trackers. This is the single element of the model that most resembles the current models we use for estimating development and testing efforts…a straight line, or at best a gantt chart, plotted against a calendar, with the end date carved in stone. Interestingly enough, this element of the model is the single element that most forecasters trust the least, and they all say so loudly.
Which brings us to the third element of the model, and I think this is the biggie. The Cone of Uncertainty. Starting at the point where the storm is now, following roughly along the predicted path of the storm, is an ever expanding cone. This cone represents where, given all the information currently available, the storm MAY go if something in the model changes. I have NEVER seen this in any model of software planning. Yet this is the part of the model that forecasters talk about the most, and encourage everyone to pay the most attention to. This is the part of that model that takes into account Risk and Ambiguity.
I think these elements are powerful in terms of creating a model that deals largely in truth, acknowledges uncertainty, visually depicts it’s status and communicates clearly, and updates rapidly and understandably. And I think it is a model we could emulate to predict, in a reasonable and acceptable manner, the software testing process, even when using rapid / exploratory testing.
It is the test teams job to provide information to management to better enable them to make decisions about releasing software. It is managements job to keep the software development and creation process on track, and release software by balancing a vast array of considerations and risks. While neither of us can know everything, or plan everything, in advance, the test team can also not simply throw up our hands and say “We cannot provide any answer to the questions your asking.” when the questions they are asking are perfectly valid. At least not if we want to be perceived as a team player adding value to the process. But a model such as this, that plots not only a course, but a reasonable expectation of possible variation around that course, and updates itself regularly, would let management see not only where the project is, and where it is headed, but it would also better help them understand the costs and risks with getting it back on track versus accepting the new point of impact. I believe that would be a far better answer than “We will test all we can until the day you shove it out the door.”
I am so intrigued by this, that I am going to begin working on such a model. I will be happy to publish both the ongoing work, and the final results, online. If you would like to collaborate on this with me, drop me a line at dgilbert@sirius-sqa.com.
In the meantime, batton down the hatches, and good luck weathering the storms of your own testing efforts.