Performance Testing Trends
Submitted by Alexander Podelko on Mon, 13/03/2006 - 04:24.
non-functional testing | performance testing
Yes, I agree with Neill that articles like this are somewhat sobering. I completely disagree with author’s conclusions and it is quite clear that he generalized beyond his area of expertise, but I would like to know what and in what exactly area he really saw that give him the idea about declining importance of performance testing. If we speak about area of single-server systems and $5K servers (where we can guess the author works), not much of performance testing was ever done here. In the best case an enthusiastic developer used a profiler to figure out why it works so slowly. How could it decline?
Looks like the article is an interesting mix of correct observations and incorrect conclusions.
Hardware is much more powerful now and many things used to be “expensive” are not any more, a good list of examples are provided. Still many other things are as expensive as they were: something like a full table scan (or perhaps becomes even more expensive as organizations put more and more data into databases).
“Bottlenecks generally fall into the categories of: processor, memory, disk, or network speed.” Yes, when you have perfect software. By the way, in this case it is pretty easy to make projections: there is the well established field of computer systems simulation / capacity planning. Many systems I worked with had internal problems with threading, synchronization, locking, trashing, memory leaks, load balancing, etc. so they didn’t scaled well enough.
“In most organizations it's easier and cheaper to add more memory, another processor, or another server to enhance performance.” Correct, if 1) you know where to add: usually not the case until you do at least some kind of performance analysis 2) the system scales well that quite often isn’t the case.
“focusing on minor items which impact overall performance (utilization) may cause an inappropriate focus on design ideas that provide minimal performance in exchange for a substantial amount of development time.” Sure, you should focus on major items which impact overall performance.
I see an explosion of complexness of systems recently well beyond three tiers. Coming Service Oriented Architecture (SOA), by my opinion, creates a lot of new performance challenges making difficult even to figure out what and how you need to test. I think we have several trends that blur the borders of performance testing; it could be not always a special hectic activity two weeks before release when people first find that performance is not as great as they would like.
I believe that performance awareness slowly moving up to the stream, so some kind of performance testing happens earlier in the development cycle by developers and testers. You shouldn’t wait for load testing if you see that it takes 30 seconds for one user to open the web-page. Another slow trend is using of modeling / simulating tools to complement performance testing.
Looks like the article is an interesting mix of correct observations and incorrect conclusions.
Hardware is much more powerful now and many things used to be “expensive” are not any more, a good list of examples are provided. Still many other things are as expensive as they were: something like a full table scan (or perhaps becomes even more expensive as organizations put more and more data into databases).
“Bottlenecks generally fall into the categories of: processor, memory, disk, or network speed.” Yes, when you have perfect software. By the way, in this case it is pretty easy to make projections: there is the well established field of computer systems simulation / capacity planning. Many systems I worked with had internal problems with threading, synchronization, locking, trashing, memory leaks, load balancing, etc. so they didn’t scaled well enough.
“In most organizations it's easier and cheaper to add more memory, another processor, or another server to enhance performance.” Correct, if 1) you know where to add: usually not the case until you do at least some kind of performance analysis 2) the system scales well that quite often isn’t the case.
“focusing on minor items which impact overall performance (utilization) may cause an inappropriate focus on design ideas that provide minimal performance in exchange for a substantial amount of development time.” Sure, you should focus on major items which impact overall performance.
I see an explosion of complexness of systems recently well beyond three tiers. Coming Service Oriented Architecture (SOA), by my opinion, creates a lot of new performance challenges making difficult even to figure out what and how you need to test. I think we have several trends that blur the borders of performance testing; it could be not always a special hectic activity two weeks before release when people first find that performance is not as great as they would like.
I believe that performance awareness slowly moving up to the stream, so some kind of performance testing happens earlier in the development cycle by developers and testers. You shouldn’t wait for load testing if you see that it takes 30 seconds for one user to open the web-page. Another slow trend is using of modeling / simulating tools to complement performance testing.