Agile Performance Testing?
Submitted by Alexander Podelko on Tue, 24/05/2005 - 20:25.
agile | development methodology | exploratory testing | extreme programming (XP) | non-functional testing | performance testing | test driven development
Neill,
You touched two very interesting points here. I have formulated them in a different way for myself, but probably it is the same (if I got it correctly).
The first point, as I got it, perhaps can be named “start performance testing early”. Everybody talking about it, but not much done. Mainly, the only thing that a performance tester (as it is usually defined now) can do until some functionality will work is to collect requirements/use-cases/scenarios.
Looks like the role of performance/load tester should be re-defined, I’d prefer to named it “performance engineer” (accidentally it is my current title, although so far it is rather a title than what I mean here). Your point about monitoring early/unit tests looks very valuable for me. Usually developers don’t monitor what is going on. The problem here is that monitoring one-user load can be useful only for really pathological cases. You need multi-user load to make it meaningful.
Probably we need to talk here about early unit performance/load testing (using some kinds of test harnesses). Mainly done by developers with help of a performance engineer.
I’d say that it will create a bridge between “performance analysis of design” (Software Performance Engineering (SPE) approach by Dr. Connie U. Smith, build around modeling software performance on early stages of development,
Performance Engineering for Systems in the Early Stages by Sudha Paidipati) and “classic” system performance/load testing.
As for the second point, I believe that performance testing is somewhat agile by definition, more like scientific research. I guess the only case when you really can do it in a formal way is when you test a well-known system (some kind of performance regression testing). You really can’t plan much – you never know at what load level you face problems and what you would be able to do with them. So it becomes an iterative process involving tuning and close work with development. Moreover, I am not sure that the term performance/load testing is good for that whole process at all: really it is a mix of testing, tuning, and diagnostics.
You can check my paper "Effective Load Testing" (see p.33-39) – I discussed these points there.
What Scott Barber’s idea do you refer to? UCML? I am a little confused here, perhaps I missing your point. I’d say that documenting load is useful for any kind of performance/load testing, formal or agile, and don’t see as it add more agile-ness.
You touched two very interesting points here. I have formulated them in a different way for myself, but probably it is the same (if I got it correctly).
The first point, as I got it, perhaps can be named “start performance testing early”. Everybody talking about it, but not much done. Mainly, the only thing that a performance tester (as it is usually defined now) can do until some functionality will work is to collect requirements/use-cases/scenarios.
Looks like the role of performance/load tester should be re-defined, I’d prefer to named it “performance engineer” (accidentally it is my current title, although so far it is rather a title than what I mean here). Your point about monitoring early/unit tests looks very valuable for me. Usually developers don’t monitor what is going on. The problem here is that monitoring one-user load can be useful only for really pathological cases. You need multi-user load to make it meaningful.
Probably we need to talk here about early unit performance/load testing (using some kinds of test harnesses). Mainly done by developers with help of a performance engineer.
I’d say that it will create a bridge between “performance analysis of design” (Software Performance Engineering (SPE) approach by Dr. Connie U. Smith, build around modeling software performance on early stages of development,
Performance Engineering for Systems in the Early Stages by Sudha Paidipati) and “classic” system performance/load testing.
As for the second point, I believe that performance testing is somewhat agile by definition, more like scientific research. I guess the only case when you really can do it in a formal way is when you test a well-known system (some kind of performance regression testing). You really can’t plan much – you never know at what load level you face problems and what you would be able to do with them. So it becomes an iterative process involving tuning and close work with development. Moreover, I am not sure that the term performance/load testing is good for that whole process at all: really it is a mix of testing, tuning, and diagnostics.
You can check my paper "Effective Load Testing" (see p.33-39) – I discussed these points there.
What Scott Barber’s idea do you refer to? UCML? I am a little confused here, perhaps I missing your point. I’d say that documenting load is useful for any kind of performance/load testing, formal or agile, and don’t see as it add more agile-ness.