Trends, Papers, and SOA
Submitted by Alexander Podelko on Wed, 06/08/2008 - 15:56.
non-functional testing | performance testing
I quickly found that my capacity of what I can do in addition to regular work is pretty limited: while I was writing a paper for the upcoming CMG conference and then helping a little with the conference organization as CPE (Computer Performance Evaluation) Subject Area Chair (CMG is a volunteer organization), I wasn't able to blog or update my site (probably more energy than time). Now this is mainly over: the CMG preliminary agenda was just published. It is time to summarize and move forward.
While the sample of the papers I worked with was small (CMG papers submitted to the CPE subject area, including performance/load testing), I believe that I noticed some interesting trends.
The number of papers about performance testing and performance engineering (often it is difficult to separate them) is growing. That confirms my observations (from my engagements with different clients) that performance knowledge centers often form around performance testing nowadays, not around performance analysts and capacity planners as it was before.
It is difficult to write a good paper about performance testing. Probably the best way to get a performance testing/engineering paper accepted is to write a good experience report. Here is what the problem was – here is what we found – here is how we fixed it. Usually it is interesting enough and few objections arise (if nothing fundamentally wrong was done). Still it is not very interesting for me to write about one case: when you have a new case each several weeks rarely one particular is so interesting to devote a paper to it. A desire to generalize the experience arises. Looks like it is not only my problem – many papers try to generalize – and here they get hit by referees (CMG uses a blind referee process). First, it is difficult to make a meaningful generalization. Second, you always may make a comment about validity or area of application. And third, looks like you need some performance testing experience to appreciate even a very useful heuristic – otherwise it may look trivial or unimportant.
An interesting trend is a large number of papers submitted by employees of large off-shore consulting companies. Looks like these companies started to invest into performance-related research labs and projects.
Somewhat unexpected for me was a small number of papers about SOA. From the beginning, there were several good publications. For example, Scott Barber's presentation and article. But most of them actually said what SOA is and what it may mean for performance testing/management/etc. I haven't seen many publications afterwards with more details or any kind of experience reports. From what I have seen probably Hon Wong's article in April 2008 Software Test & Performance magazine pp.14-19 is worth to mention. The author described some possible consequences of SOA and ways to handle them, although rather high-level too. By my opinion, SOA creates significant challenges for both performance testing and performance management and perhaps may finally kill the system of silo experts typical for large corporations. The only way I can explain the lack of papers on the subject is that SOA is still far from mainstream (while virtualization, for example, looks like is mainstream now – discussed a lot, mentioned almost everywhere).
While the sample of the papers I worked with was small (CMG papers submitted to the CPE subject area, including performance/load testing), I believe that I noticed some interesting trends.
The number of papers about performance testing and performance engineering (often it is difficult to separate them) is growing. That confirms my observations (from my engagements with different clients) that performance knowledge centers often form around performance testing nowadays, not around performance analysts and capacity planners as it was before.
It is difficult to write a good paper about performance testing. Probably the best way to get a performance testing/engineering paper accepted is to write a good experience report. Here is what the problem was – here is what we found – here is how we fixed it. Usually it is interesting enough and few objections arise (if nothing fundamentally wrong was done). Still it is not very interesting for me to write about one case: when you have a new case each several weeks rarely one particular is so interesting to devote a paper to it. A desire to generalize the experience arises. Looks like it is not only my problem – many papers try to generalize – and here they get hit by referees (CMG uses a blind referee process). First, it is difficult to make a meaningful generalization. Second, you always may make a comment about validity or area of application. And third, looks like you need some performance testing experience to appreciate even a very useful heuristic – otherwise it may look trivial or unimportant.
An interesting trend is a large number of papers submitted by employees of large off-shore consulting companies. Looks like these companies started to invest into performance-related research labs and projects.
Somewhat unexpected for me was a small number of papers about SOA. From the beginning, there were several good publications. For example, Scott Barber's presentation and article. But most of them actually said what SOA is and what it may mean for performance testing/management/etc. I haven't seen many publications afterwards with more details or any kind of experience reports. From what I have seen probably Hon Wong's article in April 2008 Software Test & Performance magazine pp.14-19 is worth to mention. The author described some possible consequences of SOA and ways to handle them, although rather high-level too. By my opinion, SOA creates significant challenges for both performance testing and performance management and perhaps may finally kill the system of silo experts typical for large corporations. The only way I can explain the lack of papers on the subject is that SOA is still far from mainstream (while virtualization, for example, looks like is mainstream now – discussed a lot, mentioned almost everywhere).