Performance: art and/or technique(s)?
Submitted by Ainars Galvans on Wed, 12/03/2008 - 16:32.
performance testing
What follows is nothing new. I only try to take another look to what have been written so far. Scott Barber recently commented The whole situation is sad. Tool vendors dominate the training market (at least in performance testing) . Perhaps I don’t care for training as I train my people myself, but I care for stereotype created by dominating vendors. I will bring my own understanding of by-effect caused by load tools bringing the nice load tools. No I don’t say that performance testing is no more an art. But there are certain limits we are yet to break.
Story which I suffer
First of all thanks Alexander for his feedback on my last blog. I realized that I did something wrong in my blog. It was about developer’s attitude to what I’m doing not about what I’m doing. So let me tell you an abstract example. Developer say: “your test results are interesting, but we will ignore them unless you will 1) get our management to document exact performance requirements i.e. exact expected load 2) replicate exactly the same with your load tests 3) make sure your environment is as close as possible to production. Your results are meaningless until then.”
For example, if I say “my calculations shows that we will need 4 boxes instead of one to sustain the load” they answer “we don’t want your calculations we want tests to be run”. It takes a lot of time to run tests as functionality is instable at that moment – and all to only prove my initial statement…
The only technique promoted by tool vendors
I still count The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling published in 1991 as one of the best performance testing books existing. Even the title of a book all alone…. Book describes different techniques, but – listen carefully – it says you can’t relay on a single technique however. You need to use at least one more to validate your results.
It is not surprise that tool vendors promote the “best practice” to use simulation to gather results and Measurements (live production) to validate them.
Performance testing as an art
One more thanks Jason Gorman for his art of painting metaphor. When I started performance testing I did it following a certain pattern supported by performance tool we had purchased. Through years I begin to see the reasons why I’m doing what I’m doing, and begin to apply that insight to new situations. I became quite proficient in using the tool and still I faced a lot of “failures” as a performance tester: projects with unsatisfied customers and stakeholders, even cancelled performance testing projects. In internal projects, where I had more freedom I tried experimenting different techniques. And I found them to be useful – in certain cases. And I’m learning an insight – slowly, because my stakeholders keep asking me to only do simulation. We have paid for the tool so much - we have to use it to get the money back!
Summing up: techniques in context
So I’ve got digital photo camera doing everything for you in automatic mode. So I don’t care about painting, should I? Should I remove to a trash all the paintings from my walls and replace them with photos which are more realistic? Photos and painting have got both it’s niche in my life.
Story which I suffer
First of all thanks Alexander for his feedback on my last blog. I realized that I did something wrong in my blog. It was about developer’s attitude to what I’m doing not about what I’m doing. So let me tell you an abstract example. Developer say: “your test results are interesting, but we will ignore them unless you will 1) get our management to document exact performance requirements i.e. exact expected load 2) replicate exactly the same with your load tests 3) make sure your environment is as close as possible to production. Your results are meaningless until then.”
For example, if I say “my calculations shows that we will need 4 boxes instead of one to sustain the load” they answer “we don’t want your calculations we want tests to be run”. It takes a lot of time to run tests as functionality is instable at that moment – and all to only prove my initial statement…
The only technique promoted by tool vendors
I still count The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling published in 1991 as one of the best performance testing books existing. Even the title of a book all alone…. Book describes different techniques, but – listen carefully – it says you can’t relay on a single technique however. You need to use at least one more to validate your results.
It is not surprise that tool vendors promote the “best practice” to use simulation to gather results and Measurements (live production) to validate them.
Performance testing as an art
One more thanks Jason Gorman for his art of painting metaphor. When I started performance testing I did it following a certain pattern supported by performance tool we had purchased. Through years I begin to see the reasons why I’m doing what I’m doing, and begin to apply that insight to new situations. I became quite proficient in using the tool and still I faced a lot of “failures” as a performance tester: projects with unsatisfied customers and stakeholders, even cancelled performance testing projects. In internal projects, where I had more freedom I tried experimenting different techniques. And I found them to be useful – in certain cases. And I’m learning an insight – slowly, because my stakeholders keep asking me to only do simulation. We have paid for the tool so much - we have to use it to get the money back!
Summing up: techniques in context
So I’ve got digital photo camera doing everything for you in automatic mode. So I don’t care about painting, should I? Should I remove to a trash all the paintings from my walls and replace them with photos which are more realistic? Photos and painting have got both it’s niche in my life.