Performance Requirements
Submitted by Alexander Podelko on Sun, 26/02/2006 - 21:40.
performance testing
After moving to my new job I stopped to receive all magazines (still puzzle for me: I spent a lot of time updating my subscriptions four months ago, still no results), so just recently read December Software Test and Performance magazine on-line. As usual, one of the most interesting parts is Scott Barber’s column about performance requirements (p.11-12).
I completely agree with Scott’s idea that some performance requirements are usability requirements. Even more, I am not sure that there are completely technical performance requirements. For example, Scott classified scalability/capacity planning and stability requirements as pure technical. Is it so? Not sure. One more variable in capacity planning is speed (response time): depending on what response time is required, you need more or less resources to support the same throughput. Even more so is stability requirements: while it is often formulated in pure technical terms, I believe it is almost pure usability requirements: how much instability/downtime users could tolerate. And exact technical requirement is just a translation of somebody’s vague idea about that.
When we speak about speed, in many cases, in addition of frustration criterion, there is also optimal productivity criterion. When users should work with the same system again and again (for example, users of internal transactional systems) it is important how many documents (claims, orders, or whatever) that user can process in a unit of time.
Really usability research of response times may be tracked back to 1968, to the "Response Time in Man-Computer Conversational Transactions" paper by Robert B. Miller: while Web systems have some specific, I don’t see any reason to separate them from all other computer systems.
I still miss the “Eureka!” moment in the column: for me, looks like almost everything was already in Scott’s earlier paper Beyond performance testing part 3: How fast is fast enough? (one of my favorite writing about the subject). I even referred to Scott’s usability study mentioned in that paper with conclusion that even users who are accustomed to a subsecond response time on a client/server system are happy with a 3-second response time from a Web-based application.
There are efforts to make an objective metric of user satisfaction. One is Application Performance Index (Apdex). Apdex is a numerical measure of user satisfaction with the performance of enterprise applications. The Apdex metric is a number between 0 and 1, where 0 means that no users were satisfied, and 1 means all users were satisfied.
I completely agree with Scott’s idea that some performance requirements are usability requirements. Even more, I am not sure that there are completely technical performance requirements. For example, Scott classified scalability/capacity planning and stability requirements as pure technical. Is it so? Not sure. One more variable in capacity planning is speed (response time): depending on what response time is required, you need more or less resources to support the same throughput. Even more so is stability requirements: while it is often formulated in pure technical terms, I believe it is almost pure usability requirements: how much instability/downtime users could tolerate. And exact technical requirement is just a translation of somebody’s vague idea about that.
When we speak about speed, in many cases, in addition of frustration criterion, there is also optimal productivity criterion. When users should work with the same system again and again (for example, users of internal transactional systems) it is important how many documents (claims, orders, or whatever) that user can process in a unit of time.
Really usability research of response times may be tracked back to 1968, to the "Response Time in Man-Computer Conversational Transactions" paper by Robert B. Miller: while Web systems have some specific, I don’t see any reason to separate them from all other computer systems.
I still miss the “Eureka!” moment in the column: for me, looks like almost everything was already in Scott’s earlier paper Beyond performance testing part 3: How fast is fast enough? (one of my favorite writing about the subject). I even referred to Scott’s usability study mentioned in that paper with conclusion that even users who are accustomed to a subsecond response time on a client/server system are happy with a 3-second response time from a Web-based application.
There are efforts to make an objective metric of user satisfaction. One is Application Performance Index (Apdex). Apdex is a numerical measure of user satisfaction with the performance of enterprise applications. The Apdex metric is a number between 0 and 1, where 0 means that no users were satisfied, and 1 means all users were satisfied.