A minor performance problem?
Submitted by Alexander Podelko on Tue, 02/01/2007 - 05:43.
non-functional testing | performance testing | performance testing patterns
Returning to Yury's comment:
At the same time in my world, typically, under load some transactions fail or have a longer response time. Nevertheless my clients are happy.
I want to elaborate the subject a little further. Yes, it indeed is very typical and probably the biggest problem I see in performance testing for now.
Unfortunately clients (for example, development teams) are happy to finish the project, move it in production, get bonus, etc., and move to next project. Most developers are not very exited by perspective of debugging code for small memory leaks or hunting for a rare error that is difficult to reproduce. So the development team becomes very creative in finding "explanations". For example, for growing memory and periodic long-running transactions in Java - that's a garbage collection. False in most cases. In few cases when it's true – let's tune garbage collection and prove that the problem went away.
Another typical approach is to say that the requirements are that 99% of transactions should be below X seconds, and for this test we have less than 1% of failed transactions. So everything is fine. Well, doesn't look so for me. It may be okay in production over time consider network and hardware failures, OS crashes, etc. But if the performance test was run in controlled environment and no hardware/OS failures were observed, it looks pretty much as a serious bug. For example, it can be a functional problem for some combination of data.
And in corporate environment performance testers quite often are eager to accept these "explanations". They have their own pipeline of projects, they don't have time to dive into details. And they have a good excuse: they were told that it is not a problem.
I am never comfortable when under load some transactions fail or have a longer response time in controlled environment and we don't know why. That indicates that we have a problem (or a few). And while customers are happy at the moment, it could change. When we have an unknown problem, why not to trace down and fix it in the controlled environment? What if that few failed transactions are a view page for your largest customer and you won't be able to create any order for that customer? Say, the view page fails when you have more than x orders for that customer?
So quite often performance testing starts from validation testing at the end of the project. And when found issues are not an absolutely evident showstopper, quite often some kind of "explanation" is suggested saying that it is not a problem. Sounds meaningless for me because we don't know what IT is at that point. First we need to find what IT is and only then decide whether it is a problem. Instead of doing some investigation the project is wrapped up.
Maybe that approach has background in functional testing where as soon as you found problem you usually can figure out how serious it is. Not the case for performance testing: usually you have no idea what cause the observed symptoms and how serious it is, quite often the original explanations turn out to be wrong.
With great pleasure I found that Michael Bolton in More Stress, Less Distress (see the end of the article) wrote that idea down very concisely. Can't fight temptation to add the illustration from Michael's article:
As Richard Feynman said in his appendix to the Rogers Commission Report on the Challenger space shuttle accident, when something is not what the design expected, it's a warning that something is wrong. "The equipment is not operating as expected, and therefore there is a danger that it can operate with even wider deviations in this unexpected and not thoroughly understood way." When a system is in an unpredicted state, it's also in an unpredictable state.
At the same time in my world, typically, under load some transactions fail or have a longer response time. Nevertheless my clients are happy.
I want to elaborate the subject a little further. Yes, it indeed is very typical and probably the biggest problem I see in performance testing for now.
Unfortunately clients (for example, development teams) are happy to finish the project, move it in production, get bonus, etc., and move to next project. Most developers are not very exited by perspective of debugging code for small memory leaks or hunting for a rare error that is difficult to reproduce. So the development team becomes very creative in finding "explanations". For example, for growing memory and periodic long-running transactions in Java - that's a garbage collection. False in most cases. In few cases when it's true – let's tune garbage collection and prove that the problem went away.
Another typical approach is to say that the requirements are that 99% of transactions should be below X seconds, and for this test we have less than 1% of failed transactions. So everything is fine. Well, doesn't look so for me. It may be okay in production over time consider network and hardware failures, OS crashes, etc. But if the performance test was run in controlled environment and no hardware/OS failures were observed, it looks pretty much as a serious bug. For example, it can be a functional problem for some combination of data.
And in corporate environment performance testers quite often are eager to accept these "explanations". They have their own pipeline of projects, they don't have time to dive into details. And they have a good excuse: they were told that it is not a problem.
I am never comfortable when under load some transactions fail or have a longer response time in controlled environment and we don't know why. That indicates that we have a problem (or a few). And while customers are happy at the moment, it could change. When we have an unknown problem, why not to trace down and fix it in the controlled environment? What if that few failed transactions are a view page for your largest customer and you won't be able to create any order for that customer? Say, the view page fails when you have more than x orders for that customer?
So quite often performance testing starts from validation testing at the end of the project. And when found issues are not an absolutely evident showstopper, quite often some kind of "explanation" is suggested saying that it is not a problem. Sounds meaningless for me because we don't know what IT is at that point. First we need to find what IT is and only then decide whether it is a problem. Instead of doing some investigation the project is wrapped up.
Maybe that approach has background in functional testing where as soon as you found problem you usually can figure out how serious it is. Not the case for performance testing: usually you have no idea what cause the observed symptoms and how serious it is, quite often the original explanations turn out to be wrong.
With great pleasure I found that Michael Bolton in More Stress, Less Distress (see the end of the article) wrote that idea down very concisely. Can't fight temptation to add the illustration from Michael's article:
As Richard Feynman said in his appendix to the Rogers Commission Report on the Challenger space shuttle accident, when something is not what the design expected, it's a warning that something is wrong. "The equipment is not operating as expected, and therefore there is a danger that it can operate with even wider deviations in this unexpected and not thoroughly understood way." When a system is in an unpredicted state, it's also in an unpredictable state.