Again about performance requirements
Submitted by Alexander Podelko on Fri, 26/01/2007 - 04:22.
non-functional testing | performance testing
Looks like performance requirements is a popular topic with wide range of opinions. In one more discussion about the subject Scott Barber wrote about performance requirements specs:
Actually, I disagree. I want qualitative business requirements. The longer I do this, the less I like when I get numbers to test against. The numbers are always wrong and lead to decisions that don't match the business needs or user expectations of the application.
Just tell me who the app is for, how happy you want/need your users to be, who your competitors are, how much risk you are willing to take and access to 10 representative users (on their own machines, in their own workspace, with their own internet/network connections) and I'll figure out the numbers in a day or two... and validate user satisfaction periodically throughout the project when folks start wanting to negotiate with the numbers.
Hmm… I'll figure out the numbers in a day or two... So does it means that Scott will figure out requirements better than some other person who is responsible for that (a Business Analyst, for example)? Well, that won’t surprise me: I guess that Scott is one of the best people in the world in doing that. Sure he will do that much better than any random person assigned to this duty.
Unfortunately there are much more systems than experts of such class, so probably we need to come out with some ways how other people can come up with some meaningful requirements when they don’t have Scott around. So far Scott’s old position about the subject looks more useful for me.
My current view of the response time requirements (other performance requirements like throughput, while sometimes may be a challenge to find, are more straightforward: if you can not keep up to workload you have, you clearly have a problem) is 1) we need to have them specified as a point of reference 2) we need to specify them before the system is created (not before final performance testing to validate them).
What exactly should be specified – goal vs. requirements (or both), average vs. x% percentile vs. APDEX, etc. – depends on the system and environment. That should be something you can measure and measure where you are against that metric(s). I agree that over-formalizing may hurt here. We need to find meaningful goals / requirements, not invent something just to satisfy a formalized process.
If we have a response time metric as a point of reference, we can use it throughout the whole development and testing process (for example, apportion that time to underlying components) and track our progress from performance engineering point of view. Tracing this metric in production give us valuable feedback that can be used for further releases of the system.
I looked through APDEX presentations I got at the CMG conference and had a conversation with Peter Sevcik. I was amazed how much more is behind the scene comparing with what is presented on the APDEX site! APDEX is a single metric of customer satisfaction. Target response time “T”, threshold between satisfied and tolerating users, is a parameter of APDEX. Another introduced number is F: threshold between tolerating and frustrated users is calculated (F = 4T). Probably there is a relationship between T and a response time goal and between F and a response time requirement (but they shouldn’t necessarily match, should they?). Peter suggests ten different approaches to finding T, including default value (4 sec), empirical data, outside reference, controlled performance experiment, etc. And the idea is to use several (say, three) of these approaches for the same system. If all come to approximately the same numbers, they gives us T. APDEX is developed for existing systems as a metric of customer satisfaction, but the approach probably can be used to find / specify response time requirement too.
Actually, I disagree. I want qualitative business requirements. The longer I do this, the less I like when I get numbers to test against. The numbers are always wrong and lead to decisions that don't match the business needs or user expectations of the application.
Just tell me who the app is for, how happy you want/need your users to be, who your competitors are, how much risk you are willing to take and access to 10 representative users (on their own machines, in their own workspace, with their own internet/network connections) and I'll figure out the numbers in a day or two... and validate user satisfaction periodically throughout the project when folks start wanting to negotiate with the numbers.
Hmm… I'll figure out the numbers in a day or two... So does it means that Scott will figure out requirements better than some other person who is responsible for that (a Business Analyst, for example)? Well, that won’t surprise me: I guess that Scott is one of the best people in the world in doing that. Sure he will do that much better than any random person assigned to this duty.
Unfortunately there are much more systems than experts of such class, so probably we need to come out with some ways how other people can come up with some meaningful requirements when they don’t have Scott around. So far Scott’s old position about the subject looks more useful for me.
My current view of the response time requirements (other performance requirements like throughput, while sometimes may be a challenge to find, are more straightforward: if you can not keep up to workload you have, you clearly have a problem) is 1) we need to have them specified as a point of reference 2) we need to specify them before the system is created (not before final performance testing to validate them).
What exactly should be specified – goal vs. requirements (or both), average vs. x% percentile vs. APDEX, etc. – depends on the system and environment. That should be something you can measure and measure where you are against that metric(s). I agree that over-formalizing may hurt here. We need to find meaningful goals / requirements, not invent something just to satisfy a formalized process.
If we have a response time metric as a point of reference, we can use it throughout the whole development and testing process (for example, apportion that time to underlying components) and track our progress from performance engineering point of view. Tracing this metric in production give us valuable feedback that can be used for further releases of the system.
I looked through APDEX presentations I got at the CMG conference and had a conversation with Peter Sevcik. I was amazed how much more is behind the scene comparing with what is presented on the APDEX site! APDEX is a single metric of customer satisfaction. Target response time “T”, threshold between satisfied and tolerating users, is a parameter of APDEX. Another introduced number is F: threshold between tolerating and frustrated users is calculated (F = 4T). Probably there is a relationship between T and a response time goal and between F and a response time requirement (but they shouldn’t necessarily match, should they?). Peter suggests ten different approaches to finding T, including default value (4 sec), empirical data, outside reference, controlled performance experiment, etc. And the idea is to use several (say, three) of these approaches for the same system. If all come to approximately the same numbers, they gives us T. APDEX is developed for existing systems as a metric of customer satisfaction, but the approach probably can be used to find / specify response time requirement too.