testingReflections.com - performance testing patterns

CMG News

Thu, 29 May 2008 14:42:12 -0500

It is still not late to submit a paper to the CMG conference. Although the abstract deadline passed, the enforced deadline is only the paper deadline – June 13.

I believe that it is the best conference about computer performance. Conference proceedings from 1997 through 2005 are available to the public – probably the largest collection of performance-related papers (and most of them are practical).

Multiple Dimensions of Performance Testing

Thu, 24 Apr 2008 20:18:00 -0500

Almost all experts agree that pre-deployment "waterfall" performance testing (which, with the record/playback method, confused by many as the performance testing itself) is not enough - too little, too late. Actually it is just one very specific way of performance testing - with a full spectrum of other approaches, which are used so infrequently (at least as intentional performance testing techniques) that I don't recall finding any good classification. Thinking about it, I see several dimensions of performance testing which, although definitely correlated, probably might be considered somewhat independently - of course, just a raw idea for the moment, just an effort to order thoughts a little.

Pitfalls of the "Waterfall" Approach to Performance Testing

Mon, 24 Mar 2008 16:23:17 -0500

Looks like the pre-production validation approach to performance testing becomes typical for large corporations (if there is any at all):

-get the system ready
-develop all scripts requested (sometimes offshore)
-run them all together
-compare with the requirements provided
-allow some percentage of errors according to the requirements
-involve the development team if requirements were missed

Re: Performance testing and coverage

Tue, 11 Mar 2008 01:38:38 -0500

Can't leave Ainars Galvans' posting unanswered. I think that it touches very important issues I am fighting for long time with, so this post is going beyond just a comment.

It is very interesting that I completely agree with Ainars on most items except final conclusions – which I completely disagree with. I suspect that it is rather terminology difference. So let's start with what I disagree.

CMG Opens Its Content to the Public

Tue, 12 Feb 2008 13:30:21 -0600

The Computer Measurement Group (CMG) is making its conference proceedings from 1997 through 2005 available to the public. I believe that CMG holds the best practical conference in performance analysis, capacity planning, and related areas. In addition to the areas listed below, I'd definitely add performance testing. Here is the official mail CMG sent:

Performance Requirements

Sun, 06 Jan 2008 17:08:27 -0600

My performance requirements paper was published in the January issue of Software Test & Performance (pp.18-24).

It was simple: I just sent a draft – and now I am reading it printed. With a new name - or even two: it is referred as You Can Gauge Performance Without Requirements in one place and Gauging Performance in The Absence of Measures in another. Not to mention other minor improvement.

Qualitative and Quantitative

Fri, 16 Nov 2007 14:36:07 -0600

A few thoughts inspired by a question about qualitative and quantitative information reporting performance test results in Yahoo's LoadRunner group.

My understanding of qualitative and quantitative in the context of reporting performance test results is:

Quantitative is what we directly measure or straightforwardly calculate from direct measurements (like response times or resource utilization)

What Skills Performance Testers Need and How to Get Them?

Mon, 15 Oct 2007 09:31:39 -0500

From time to time I see questions on different forums asking what skills are necessary for performance testers. There were pretty interesting discussions. Looks like most experts agree that performance testing requires more skills than just knowledge about how to create a script for a particular load testing tool. While it is still possible to imagine a performance tester in a large corporation with deep specialization who only creates scripts and mechanically runs them while other performance experts monitor the system and analyze results, I don't see many perspectives neither for this person, nor for the approach. Systems become so complicated now that the sum of specialized expert views doesn't give the whole performance picture.

Classify Performance Tests: IVECTRAS

Sat, 25 Aug 2007 17:45:02 -0500

This is the second installment of a currently unknown number of posts about heuristics and mnemonics I find valuable when teaching and conducting performance testing.

Other posts about performance testing heuristics and mnemonics are:


I have struggled for over 7 years now with first figuring out and then trying to explain all the different "types" of performance tests. You know the ones:


Well, I finally have an alternative.

IVECTRAS

CMG 2007 Call for Paper

Sun, 11 Mar 2007 10:58:19 -0500

This year performance (load, stress) testing will be a focus track at the CMG 2007 conference. It is the best conference, by my opinion, about performance-related topics. Below is the official text:

The Computer Measurement Group (CMG) calls for papers and presentations for the 33rd International Conference to be held in San Diego, California, December 2nd through 7th, 2007. The 2007 CMG conference will cover load and stress testing, benchmarking, performance optimization, software performance engineering, resource management, capacity analysis, simulation and analytic modeling, and cost management with special emphasis on Virtualization, System Oriented Architecture (SOA), IT Service Management and IT Infrastructure Library (ITIL), and the technology implications of globalization.

A minor performance problem?

Mon, 01 Jan 2007 22:43:10 -0600

Returning to Yury's comment:
At the same time in my world, typically, under load some transactions fail or have a longer response time. Nevertheless my clients are happy.

I want to elaborate the subject a little further. Yes, it indeed is very typical and probably the biggest problem I see in performance testing for now.

WOPR8: Call for Presentations (CFP)

Tue, 19 Dec 2006 22:59:23 -0600

Theme: Critical Incidents in Effective Performance Testing

Conference Location and Time
MITRE Bedford MA, USA

April 12 – 14, 2008

Major Questions of Interest
Questions we are interested in exploring at WOPR8 include:

What critical Incidents lead to effective performance testing? By sharing and attempting to understanding the critical incidents leading to effective performance testing we will explore how our craft can make an important contribution to all IT and Development projects.

CMG Conference

Tue, 12 Dec 2006 22:55:30 -0600

Attended CMG 2006. It was really good ! I am too lazy to write down everything that was there, but there are others: here is one detailed blog. Of course, it is just one view: there was up to 12 different tracks at each timeslot. Unfortunately papers are available to members only, but Measure IT newletter is free (requires free registration).

Risk-based performance testing. A different practice?

Wed, 25 Oct 2006 06:02:25 -0500

With this post I want to continue to attack performance testing with the only goal to validate performance requirements. To me it is testing happy-path only. Some publications and even tutorials claim this to be solved by adding risk-analysis during performance requirements gathering. I’m afraid this is not my practice and not the practice of many performance testers as much as I could judge.

Once again about the difference

Thu, 04 Oct 2007 14:09:10 -0500

Listening to Denise Arruda's presentation at CT CMG I found another point of view for my thoughts about performance testing. Simplified performance testing process flow looks like

1) Run test(s)
2) Meet requirements? Yes -> end
3) No-> modify the system and go to 1

Only 1) and 2) is really testing. The 3) represents performance analysis and system tuning (optimization). Functional testing doesn't have that part at all while it requires some specific skills (like some understanding of internals and debugging) and probably a different attitude (more help to look for the problem, not only report it).