Classify Performance Tests: IVECTRAS

Absolutely the same tests are sometimes used for investigation and validation. They sole distinction is in how the results/data/information/etc. are reported/communicated/used. In my experience, knowing before I "press the run button" whether the expected "result" is pass/fail or lots of data to analyze to help folks understand is useful. Knowing that, for example, may impact the degree of logging I use or what things I monitor.

As I define Spike testing, it would be an I(E/C)(T/R)S, where E/C and T/R depend entirely on context. Endurance could fall in any IVECTRAS class... which is part of what I like about the classification system.

Basically, I want the team to decide what they would like the performance testing to accomplish and trust me to design a test that will accomplish it. When a team or client tells me to do X testing, it almost never adds the value they were hoping for unless I am smart enough to schedule a meeting (that usually lasts longer than it would have taken to both design and execute the test in the first place) to figure out what they are hoping to accomplish.

Thus far, asking clients or teams to completely dispose of the terms that folks like you and I have struggled to understand, define and explain and instead use this, have caused them to think about intent and desired outcome BEFORE asking me to run a test, which means that when they do ask me for a test, I've gotten almost all of the information I've needed to create and execute a test that will assist them without needing additional meetings, without tempers flaring, without frustration, and without anyone feeling like an idiot.

I'm afraid that adding more letters will get counter productive.

Cognitively, human beings find it relatively easy to hold 7 items in a list in their head at once. It gets dramatically more challenging to remember the list starting at 8 items. An 8 item list with a mnemonic
is approximately the same difficulty to remember as a 7 without. At 9, with or without a mnemonic, it would be unreasonable to expect anyone who didn't use the list on a virtual daily basis to remember it. For that reason alone, I'd rather keep the list at 8 items knowing that it covers most of what seems to be relevant and can be remembered by most people with relative ease than to add letters and end up with a more "complete" taxonomy that the majority of folks won't use because it's too hard to remember.

--
Scott Barber
Chief Technologist, PerfTestPlus
VP Operations & Executive Director, Association for Software Testing
sbarber@perftestplus.com

Scott,

Performance tests may be described from different points of view and the problem of traditional terminology is that it is not well defined and, moreover, there are conflicting definitions.

For example, when I use terms "load" and "stress" test, I mean "Anticipated" and "Stressful" conditions respectively. Not all people do. So any explisit specification is better that just using vague terms that may be interpreted differently. IVECTRAS is a great step in this direction.

I am a little confused how you classify "Investigation" and "Validation". In my understanding sometimes the same test may be used for validation and for investigation depending on the context.

Some old terms describe the test from other points of view. For example, spike or endurance test. First tells us about load pattern and the second tells us about the length. They probably get into "Investigation", but it doesn't give us information about load pattern or length. Should we add more letters to IVECTRAS?

Alex