Performance Testing Innovations
Submitted by Alexander Podelko on Thu, 24/09/2009 - 18:01.
performance testing | performance testing tools
Scott Barber's Performance Testing Innovations ... or Not is, as usual, pretty interesting and thought provoking.
It indeed looks like there are not many innovations in performance testing. Yes, I also believe that agile is the "native" way of doing performance testing and good performance testers always did it in an agile way (even when the word wasn't used in this context yet), it is exactly what I wrote in my CMG'08 paper Agile Performance Testing.
But I am not so sure in the idea to invest in the easier side of performance testing and that we were focused on the hard-to-impractical (bordering on impossible) side of performance testing.
There is a huge difference between organizations in how they address performance. For example, we can speak about performance maturity levels as described in A Performance Process Maturity Model by Michael Maddox. It may look a little different / be less structured from the performance tester / software development perspective, but it would be similar.
I guess that Scott meant that most organizations are on level 1 (perhaps 2) of performance maturity and they (organizations) should be investing in pretty basic stuff (like proper monitoring and performance testing). Any sophisticated stuff like modeling won't be much use for them until they implement basics.
It doesn't mean that organizations with high levels of performance maturity (at least for some parts of the organization – usually it is for their "main" systems) won't benefit from sophisticated innovations. And they indeed may bring real value in these cases. While applying this kind of stuff in organizations with level 1 of performance maturity would be indeed hard-to-impractical (bordering on impossible) and probably waste of time and money.
By the way, if we speak about load test vendors, I'd rather say that they are rather on the easier side, trying to implement something to make it easier for an inexperienced person to create and run load test scripts. Something like automatic correlation, click and script protocol, nice analysis graphs, etc. The stuff which usually works fine in simple cases and gives deceptive impression that performance testing is easy. While it is helpful in some cases, it definitely can't replace established performance engineering processes and good performance engineers (as sophisticated IDEs can't replace good architects and software developers in software development).
Education is a base for innovations (and a way to increase performance maturity level). I'd definitely add architects and developers to the list. It is interesting that all people I talked to from the organization with high performance engineering maturity level (and here I mean performance engineering maturity level, not performance maturity level as in Michael Maddox's article) stressed that you can't do it with performance testing team only. The next step, after you have established performance testing / engineering process, is to educate architects and software developers and involve them in performance-related activities.
It indeed looks like there are not many innovations in performance testing. Yes, I also believe that agile is the "native" way of doing performance testing and good performance testers always did it in an agile way (even when the word wasn't used in this context yet), it is exactly what I wrote in my CMG'08 paper Agile Performance Testing.
But I am not so sure in the idea to invest in the easier side of performance testing and that we were focused on the hard-to-impractical (bordering on impossible) side of performance testing.
There is a huge difference between organizations in how they address performance. For example, we can speak about performance maturity levels as described in A Performance Process Maturity Model by Michael Maddox. It may look a little different / be less structured from the performance tester / software development perspective, but it would be similar.
I guess that Scott meant that most organizations are on level 1 (perhaps 2) of performance maturity and they (organizations) should be investing in pretty basic stuff (like proper monitoring and performance testing). Any sophisticated stuff like modeling won't be much use for them until they implement basics.
It doesn't mean that organizations with high levels of performance maturity (at least for some parts of the organization – usually it is for their "main" systems) won't benefit from sophisticated innovations. And they indeed may bring real value in these cases. While applying this kind of stuff in organizations with level 1 of performance maturity would be indeed hard-to-impractical (bordering on impossible) and probably waste of time and money.
By the way, if we speak about load test vendors, I'd rather say that they are rather on the easier side, trying to implement something to make it easier for an inexperienced person to create and run load test scripts. Something like automatic correlation, click and script protocol, nice analysis graphs, etc. The stuff which usually works fine in simple cases and gives deceptive impression that performance testing is easy. While it is helpful in some cases, it definitely can't replace established performance engineering processes and good performance engineers (as sophisticated IDEs can't replace good architects and software developers in software development).
Education is a base for innovations (and a way to increase performance maturity level). I'd definitely add architects and developers to the list. It is interesting that all people I talked to from the organization with high performance engineering maturity level (and here I mean performance engineering maturity level, not performance maturity level as in Michael Maddox's article) stressed that you can't do it with performance testing team only. The next step, after you have established performance testing / engineering process, is to educate architects and software developers and involve them in performance-related activities.