Performance Requirements / Load Testing
Submitted by Alexander Podelko on Mon, 25/10/2004 - 15:34.
Mercury LoadRunner | performance testing | performance testing patterns
We can easily control load intensity in LoadRunner with think time. If you have scripts without think time and 1,000 users, you have 1,000 users running requests concurrently.
Let's guess that a query takes 1 sec in average. If you put 9 sec think time between queries, you will have about 100 users running requests concurrently (with 1,000 users logged in). If you put 99 sec think time between queries, you will have about 10 users running requests concurrently (with 1,000 users logged in). That change picture completely.
Without specifying the intensity of load, requirements like "With 400 users, no query will take longer than 15 seconds" are not too meaningful. And here we usually have problems. People defining performance requirement usually are ready to agree for 5-10 sec, perhaps 30 sec, but object, for example, 15-30 min think time and decreasing the number of users below the number of their named user. That usually creates much, much heavier load (and usually is equal to buying much much more powerful hardware and creating much much more sophisticated deployment). You would never, never have ALL your user running queries one after another without analyzing results.
Another extreme can be illustrated, for example, by a Cognos benchmark :
"USERS
To accurately judge the number of users that can be supported in a real world environment based on performance in a test environment, you must distinguish between named, active, and concurrent users.
Named users make up the total population of individuals who can be identified by and potentially use the system. They represent the total user community, and can be active or concurrent at any time. In a real-life BI environment, this is the total number of individuals authorized to use the system. It is the number of most interest when planning a BI implementation, because it tells you how many users you can expect to support in a given environment with the response times reported in a test environment.
Active users are logged on to the system at a given time. They include users who are simply viewing the results returned from a query. Although this subset of active users are logged on to the system, they are not sending requests. Based on Cognos’ experience with thousands of customers, a good assumption is that 10 percent of named users are active at any given time. Therefore, 100 active users represent an environment with 1,000 named users.
Concurrent users are not only logged on to the system, but represent the subset of active users who are sending a request or waiting for a response. They are the only users actually stressing the system at any given time. Based on Cognos’ experience with thousands of customers, a good assumption is that 10 percent of active users are concurrent users. Therefore, 100 concurrent users represent an environment with 1,000 active users and 10,000 named users."
While I completely agree with approach in general and user's classification, the ratio looks a little too high. And I completely disagree with the idea that if you test the system with 1,000 concurrent users you can state that you support 100,000 users. Although let's leave it on marketing conscience - I hope none of experienced engineers would state that.
Unfortunately we have one terminology / phychological trap here. When business/IT people defining performance requirements talk about concurrent users, they mean "active" users according the above specification. But as soon as you mention the classification above, they start insisting that they really mean "concurrent" users increasing workload we speak about in 10 (or whatever) times and probably increasing the cost of the system to support that workload in 100 (or whatever - but I guess that function is closer to exponential) times.
Let's guess that a query takes 1 sec in average. If you put 9 sec think time between queries, you will have about 100 users running requests concurrently (with 1,000 users logged in). If you put 99 sec think time between queries, you will have about 10 users running requests concurrently (with 1,000 users logged in). That change picture completely.
Without specifying the intensity of load, requirements like "With 400 users, no query will take longer than 15 seconds" are not too meaningful. And here we usually have problems. People defining performance requirement usually are ready to agree for 5-10 sec, perhaps 30 sec, but object, for example, 15-30 min think time and decreasing the number of users below the number of their named user. That usually creates much, much heavier load (and usually is equal to buying much much more powerful hardware and creating much much more sophisticated deployment). You would never, never have ALL your user running queries one after another without analyzing results.
Another extreme can be illustrated, for example, by a Cognos benchmark :
"USERS
To accurately judge the number of users that can be supported in a real world environment based on performance in a test environment, you must distinguish between named, active, and concurrent users.
Named users make up the total population of individuals who can be identified by and potentially use the system. They represent the total user community, and can be active or concurrent at any time. In a real-life BI environment, this is the total number of individuals authorized to use the system. It is the number of most interest when planning a BI implementation, because it tells you how many users you can expect to support in a given environment with the response times reported in a test environment.
Active users are logged on to the system at a given time. They include users who are simply viewing the results returned from a query. Although this subset of active users are logged on to the system, they are not sending requests. Based on Cognos’ experience with thousands of customers, a good assumption is that 10 percent of named users are active at any given time. Therefore, 100 active users represent an environment with 1,000 named users.
Concurrent users are not only logged on to the system, but represent the subset of active users who are sending a request or waiting for a response. They are the only users actually stressing the system at any given time. Based on Cognos’ experience with thousands of customers, a good assumption is that 10 percent of active users are concurrent users. Therefore, 100 concurrent users represent an environment with 1,000 active users and 10,000 named users."
While I completely agree with approach in general and user's classification, the ratio looks a little too high. And I completely disagree with the idea that if you test the system with 1,000 concurrent users you can state that you support 100,000 users. Although let's leave it on marketing conscience - I hope none of experienced engineers would state that.
Unfortunately we have one terminology / phychological trap here. When business/IT people defining performance requirements talk about concurrent users, they mean "active" users according the above specification. But as soon as you mention the classification above, they start insisting that they really mean "concurrent" users increasing workload we speak about in 10 (or whatever) times and probably increasing the cost of the system to support that workload in 100 (or whatever - but I guess that function is closer to exponential) times.