Performance Requirements / Load Testing
Submitted by Alexander Podelko on Wed, 17/11/2004 - 04:20.
Mercury LoadRunner | performance testing
What are "reasonable think times"? It can be 3 sec or 3 hours depending on context (for example, a data operator typing a few numbers into a simple form and then save it in the first case and a data analytic getting data from a database and then analyzing them offline in the second case).
Perhaps it would be better to look at it from throughput point of veiw. You can measure performance of your system in the number of transactions per period of time, whatever transactions mean for you - like reports per min, queries per hour or pages per sec. I mean here business transactions important to you, LoadRunner (or another tool) has different notion of transaction.
The number of user (i.e. connections) is important too, but if you haven't any particular problems with a large number of transactions in your software and you don't max out the system (in this case you can get any kind of weird effects), it should be quite similar to simulate 100 users running 60 queries per hour (one query per minute - if we guess that query are very fast, it means 1 minute think time between queries) OR 500 users running 12 queries per hour (one query per five minutes - if we guess that query are very fast, it means 5 minute think time between queries).
Again, above is true if you tested it with 500 users, sure that it works fine, and you have some resources available on the system (for all of them - cpu, memory, i/o, network) - otherwise results can differ significantly.
So my suggestions is to start with the total throughput the system should handle, then find the number of concurrent users, get the number of transaction per user for a test periond of time, and then try to set think times to ensure the proper number of transaction per user.
If you have license limitations (or don't have time to run a real-life scenario), you can try to use less users submitting more transaction (so the total throughput would be the same). So instead of 500 users submitting 10 queries per hour you will simulate 250 users submitting 20 queries per hour (see my comments above when it can be appropriate). In this case we can speak about ratio of LoadRunner and real user (1:2 for that example), I don't see much sense in it in other cases.
I believe that 5-15 sec think time is good only for professional operators that enter data the whole day. Otherwise they are unrealistic and you get much heavier load. I believe that think time should be not how quickly user can work, but avarage of all breaks (including cofee breaks, meetings, Internet browsing, etc.) across all users. Difficult to estimate, so probably it is better to go from any facts (if you can find any) - the number of reports per day, the number of filled form per day, etc.
My point of view is that you make vuser more (quite oftem) or less (rather rare) stressful than the real user, there is no inherent ratio or relationship. Simulation never would be equal to the real life, but your task is to get close (as close depends on your risks / resources / skills / intuition).
Perhaps it would be better to look at it from throughput point of veiw. You can measure performance of your system in the number of transactions per period of time, whatever transactions mean for you - like reports per min, queries per hour or pages per sec. I mean here business transactions important to you, LoadRunner (or another tool) has different notion of transaction.
The number of user (i.e. connections) is important too, but if you haven't any particular problems with a large number of transactions in your software and you don't max out the system (in this case you can get any kind of weird effects), it should be quite similar to simulate 100 users running 60 queries per hour (one query per minute - if we guess that query are very fast, it means 1 minute think time between queries) OR 500 users running 12 queries per hour (one query per five minutes - if we guess that query are very fast, it means 5 minute think time between queries).
Again, above is true if you tested it with 500 users, sure that it works fine, and you have some resources available on the system (for all of them - cpu, memory, i/o, network) - otherwise results can differ significantly.
So my suggestions is to start with the total throughput the system should handle, then find the number of concurrent users, get the number of transaction per user for a test periond of time, and then try to set think times to ensure the proper number of transaction per user.
If you have license limitations (or don't have time to run a real-life scenario), you can try to use less users submitting more transaction (so the total throughput would be the same). So instead of 500 users submitting 10 queries per hour you will simulate 250 users submitting 20 queries per hour (see my comments above when it can be appropriate). In this case we can speak about ratio of LoadRunner and real user (1:2 for that example), I don't see much sense in it in other cases.
I believe that 5-15 sec think time is good only for professional operators that enter data the whole day. Otherwise they are unrealistic and you get much heavier load. I believe that think time should be not how quickly user can work, but avarage of all breaks (including cofee breaks, meetings, Internet browsing, etc.) across all users. Difficult to estimate, so probably it is better to go from any facts (if you can find any) - the number of reports per day, the number of filled form per day, etc.
My point of view is that you make vuser more (quite oftem) or less (rather rare) stressful than the real user, there is no inherent ratio or relationship. Simulation never would be equal to the real life, but your task is to get close (as close depends on your risks / resources / skills / intuition).