Performance (identified) requirements: Trap
Submitted by Ainars Galvans on Tue, 18/07/2006 - 14:34.
agile | performance testing
Well, this is something I’ve touched several times, mostly telling the experiences. I’m going to be direct (or even offensive) this time and tell what I feel like: The view “documenting (identifying) performance requirements and predicting load is the right way doing performance tests” is a trap for all - testers, customers and management. I do welcome everyone who doesn’t think so to add comments. I also welcome everyone who doesn’t agree with that view or doesn’t think it is the most common one, to comment.
foreword
Some time ago I wrote Performance myths provoked by performance tools advertisement/docs where I tried to understand reasons for article by Robert that was critiqued by almost anyone.
James Bach recently wrote about personal attacks . I’m afraid that every experienced tester see Robert’s post “is directed against me as a person“, because it declines the performance tester role.
Just as James I would like to ask my friends and enemies (and Roberts’) alike to help me identify and correct any of my own bad behaviour.
More over the last STP Scott Barber talking about paradigm shift. What I was trying to say there in the old blog was, that the old paradigm (quite logically) caused the Roberts’ opinion.
So the (identified) performance requirements
It seems commonly accepted wisdom, that performance tests need goals. Goals should be measurable (among other characteristics listed in SMART). You should know how it all end up: convincing management and the customer that they need to define mathematically what are the performance needs, i.e. identified requirements. It is really hard to convert their expectations (unidentified requirements) to the requirements, but we are ready to suggest the methodology for it. We will let everyone think about it, understand how complicated it is and how hard it is to understand what they actually want.
Have you ever tried to document what are the performance requirements against your new car you want to purchase? Like how fast it should be able to accelerate from 0 to 100 km/h. Or what load(eight) it should be able to maintain (carry) and still provide no more than 5 seconds or response time when you want to accelerate from 100 to 120 in order to out-distance anyone. Is it a way you are thinking? How about running up-the hill: what is the maximum angle in your region, etc. ? Could you identify all those requirements and believe that you have really documented exactly what you want?
Performance expectations
Customer expectations are all about the production-time, long-term. The most significant, they expect that given the business changes it will be able to tune the system performance (for example by adding more hardware). And the business changes quite a lot.
Do you think that customers prefer comprehensive documentation of the system performance characteristics, the most reliable tools and process used to validate it? Do you believe that the performance requirements negotiated by contract are going to protect customer against issues in production? Do you think the performance test plans approved by all parties and correctly followed is what they want?
Well, that was what I believed in for years but found it failing all the time. Now look at agile manifesto and you will find that I’m actually quoting items “on the right“ while I have learned to value items on the left more in performance projects as well.
my vision: agility
I don’t yet have a complete understanding of how things should happen. I have some vision however. What I do in my projects:
1) Communicate with developers about performance (your experienced issues, patterns, etc). Developers as individuals are able to avoid performance issues. Sometimes you could avoid them by simply engaging a discussion.
2) Plan small steps, plan next steps based on information gathered so far: both from testing and communication. (Developer could tell you that feature X will not scale by design, so there is no sense to test it).
3) Set the goal to improve the system performance characteristics wherever it makes sense (in terms of cost VS gain) instead of achieve the pre-identified requirements
Well I have not yet concluded how to include customer. In item 3) we could make the most obvious decisions without them, but in a lot of cases we need to communicate with customers to understand if this improvement is worth it. It will however mean that customer will be required to forecast the gain... it should really be a collaboration, not the request to provide a signed answer.
foreword
Some time ago I wrote Performance myths provoked by performance tools advertisement/docs where I tried to understand reasons for article by Robert that was critiqued by almost anyone.
James Bach recently wrote about personal attacks . I’m afraid that every experienced tester see Robert’s post “is directed against me as a person“, because it declines the performance tester role.
Just as James I would like to ask my friends and enemies (and Roberts’) alike to help me identify and correct any of my own bad behaviour.
More over the last STP Scott Barber talking about paradigm shift. What I was trying to say there in the old blog was, that the old paradigm (quite logically) caused the Roberts’ opinion.
So the (identified) performance requirements
It seems commonly accepted wisdom, that performance tests need goals. Goals should be measurable (among other characteristics listed in SMART). You should know how it all end up: convincing management and the customer that they need to define mathematically what are the performance needs, i.e. identified requirements. It is really hard to convert their expectations (unidentified requirements) to the requirements, but we are ready to suggest the methodology for it. We will let everyone think about it, understand how complicated it is and how hard it is to understand what they actually want.
Have you ever tried to document what are the performance requirements against your new car you want to purchase? Like how fast it should be able to accelerate from 0 to 100 km/h. Or what load(eight) it should be able to maintain (carry) and still provide no more than 5 seconds or response time when you want to accelerate from 100 to 120 in order to out-distance anyone. Is it a way you are thinking? How about running up-the hill: what is the maximum angle in your region, etc. ? Could you identify all those requirements and believe that you have really documented exactly what you want?
Performance expectations
Customer expectations are all about the production-time, long-term. The most significant, they expect that given the business changes it will be able to tune the system performance (for example by adding more hardware). And the business changes quite a lot.
Do you think that customers prefer comprehensive documentation of the system performance characteristics, the most reliable tools and process used to validate it? Do you believe that the performance requirements negotiated by contract are going to protect customer against issues in production? Do you think the performance test plans approved by all parties and correctly followed is what they want?
Well, that was what I believed in for years but found it failing all the time. Now look at agile manifesto and you will find that I’m actually quoting items “on the right“ while I have learned to value items on the left more in performance projects as well.
my vision: agility
I don’t yet have a complete understanding of how things should happen. I have some vision however. What I do in my projects:
1) Communicate with developers about performance (your experienced issues, patterns, etc). Developers as individuals are able to avoid performance issues. Sometimes you could avoid them by simply engaging a discussion.
2) Plan small steps, plan next steps based on information gathered so far: both from testing and communication. (Developer could tell you that feature X will not scale by design, so there is no sense to test it).
3) Set the goal to improve the system performance characteristics wherever it makes sense (in terms of cost VS gain) instead of achieve the pre-identified requirements
Well I have not yet concluded how to include customer. In item 3) we could make the most obvious decisions without them, but in a lot of cases we need to communicate with customers to understand if this improvement is worth it. It will however mean that customer will be required to forecast the gain... it should really be a collaboration, not the request to provide a signed answer.