Does it matter how you test performance?

After thinking for a few days about how to respond, I guess all I can say is "I agree", to all that. Thanks for commenting. :-)

I'd say that it doesn't matter much how you do performance testing while you are you doing it in a right way. There is always a difference between your tests and reality, so if difference between two approaches is lesser that difference between each of your tests and reality it doesn't matter. And it is always better to test in any way than just discuss how to do it in the best way.

An example of a wrong way: you query a database that supposed to be huge. Test it for performance with three records. Happens all the time, especially by developers.

Funny you should say that, because as I was writing this post I was thinking it was a very "investigative" position, and thinking, hmmm, I wonder if I can get Scott to read this and tell me what he thinks. So thanks!

>> 1) There are a lot of things about Performance Testing that is widely mis-understood or under-considered.

I couldn't agree more with that, especially the "under-considered" part.

>> 2) There are a bunch of things that *are* widely understood...

Surprisingly so, yes.

>> ...only true indicator of the goodness of performance is the user of the thing you are testing...

Absolutely. And these are all just different approaches for studying whether it will be fast enough for that user. Areas to improve performance can be found all these ways.

>> 6) I believe that collecting data is easy (at least conceptually), but assigning "goodness" or "badness" of the implications of that data often isn't.

Right, which is why I stress how important it is to understand what you are doing, or trying to do, and to try to anticipate how the system might react to that... It seems to me that after the test, whether you were right or were surprised, you'll be in a better position to figure out what did happen -- if you tried to figure that out beforehand as well...

Speaking of which guess what. Turns out the sine wave pattern is caused by the way we simualted user idle time...

Hmmm. ;-) I think I asked because I was not sure they were mutually exclusive...

Oh yes, I do plan on driving people mad with this. As many people as possible. ;-)

Why do you feel these are mutually exclusive? It is indeed and insignt and you have lost your mind, and found a different perspective now I would advocate driving people as mad with this new view point as Scott and I do with ours :)

Neill McCarthy
"Agile Testers of the World UNIT !"

Right on man.

Charlie,

Do me a favor, go back now and re-read your favorites of my articles/columns with this in mind. See if you get a different meaning out of them. Obviously, any technical aspects of how to apply a technique won't have changed, but see if the message sounds different to you.

I have a number of soap-boxes (duh) that sometimes get viewed of "Scott's One True Way" to Performance Test, but if I've accomplished one of my goals, I hope my writing conveys the following messages...

1) There are a lot of things about Performance Testing that is widely mis-understood or under-considered.
2) There are a bunch of things that *are* widely understood - and I'm not going inefficiently summarize them here. Please do your homework.
3) Here is one of those those things and why I think it's worthy of consideration.
4) Here is a tool in my "performance tester's toolkit", an example of why I may use it and some instructions on how to use it.
5) Overarching key point... No matter what technique, method, approach, process you use never, never, ever forget that the only true indicator of the goodness of performance is the user of the thing you are testing -- whether that user be human or machine.
6) I believe that collecting data is easy (at least conceptually), but assigning "goodness" or "badness" of the implications of that data often isn't.

If that doesn't come through, let me know... It may be that I'm overdue to write a column about exactly this!


--
Scott Barber
Chief Technologist, PerfTestPlus
Executive Director, Association for Software Testing
sbarber@perftestplus.com
http://www.perftestplus.com
http://www.associationforsoftwaretesting.org