Qualitative and Quantitative
Submitted by Alexander Podelko on Fri, 16/11/2007 - 21:36.
non-functional testing | performance testing | performance testing patterns | performance testing tools | reliability testing
A few thoughts inspired by a question about qualitative and quantitative information reporting performance test results in Yahoo's LoadRunner group.
My understanding of qualitative and quantitative in the context of reporting performance test results is:
Quantitative is what we directly measure or straightforwardly calculate from direct measurements (like response times or resource utilization)
Qualitative is whatever conclusions we do on the base of quantitative results and whatever else information we possess (requirements, architecture, etc.). For example, "the system will support up to 100 user" or "performance won't be adequate over WAN with bandwidth below 256Kbit/sec". While some such statements look pretty quantitative due to use of numbers, they are not exact science and depend on multiple assumptions.
The goal of performance testing is qualitative information, ultimately if the system has adequate performance and reliability. Whatever numbers are reported, they are meaningless without context and proper interpretation based on the context.
Reporting only quantitative results means that you leave interpretation to the reader who in most cases doesn’t have enough expertise and, even if has, doesn’t have time to digest and interpret all results.
In most cases performance testing is an iterative process of getting qualitative information. The sequence of tests iteratively leads to getting necessary qualitative information. Numeric results are the base of providing qualitative results, but often it is not one-to-one relationship: conclusions made on the base of one test results determine what next tests are necessary.
So usually performance testing is iterative process more like peeling off onion one layer after another (where layers are bottlenecks). If we have a bottleneck (for example, the configured number of threads limits the number of users who may work with the system) the main qualitative result (unless we don't need more users than we already achieved) is to find the bottleneck and eliminate it.
That is why we don't see much math usage in performance testing. I mean such things like design of experiments theory, confidence interval, and statistical hypothesis test - which look pretty appropriate from a first look. You may apply it to numeric results you get, but it doesn't incorporate all assumptions you use. You may build a lot around numeric test results to find out that only thing you needed was to increase a configuration parameter.
So getting qualitative information from numeric test results usually is an almost-impossible-to-formalize procedure based on experience and knowledge of the context often including a sequence of tests where results of previous tests define next ones.
My understanding of qualitative and quantitative in the context of reporting performance test results is:
Quantitative is what we directly measure or straightforwardly calculate from direct measurements (like response times or resource utilization)
Qualitative is whatever conclusions we do on the base of quantitative results and whatever else information we possess (requirements, architecture, etc.). For example, "the system will support up to 100 user" or "performance won't be adequate over WAN with bandwidth below 256Kbit/sec". While some such statements look pretty quantitative due to use of numbers, they are not exact science and depend on multiple assumptions.
The goal of performance testing is qualitative information, ultimately if the system has adequate performance and reliability. Whatever numbers are reported, they are meaningless without context and proper interpretation based on the context.
Reporting only quantitative results means that you leave interpretation to the reader who in most cases doesn’t have enough expertise and, even if has, doesn’t have time to digest and interpret all results.
In most cases performance testing is an iterative process of getting qualitative information. The sequence of tests iteratively leads to getting necessary qualitative information. Numeric results are the base of providing qualitative results, but often it is not one-to-one relationship: conclusions made on the base of one test results determine what next tests are necessary.
So usually performance testing is iterative process more like peeling off onion one layer after another (where layers are bottlenecks). If we have a bottleneck (for example, the configured number of threads limits the number of users who may work with the system) the main qualitative result (unless we don't need more users than we already achieved) is to find the bottleneck and eliminate it.
That is why we don't see much math usage in performance testing. I mean such things like design of experiments theory, confidence interval, and statistical hypothesis test - which look pretty appropriate from a first look. You may apply it to numeric results you get, but it doesn't incorporate all assumptions you use. You may build a lot around numeric test results to find out that only thing you needed was to increase a configuration parameter.
So getting qualitative information from numeric test results usually is an almost-impossible-to-formalize procedure based on experience and knowledge of the context often including a sequence of tests where results of previous tests define next ones.