Performance Bottlenecks / Diagnostics

Scott/Alex,

I think you are going to see a huge shift towards the use of Mercury Lifecycle diagnostics because it comes as close to putting you back inside the IDE as possible. Yes, it will be an added expense, and yes, it is more intrusive, but it is at the heart a profiler. All profilers in this category require this. There is no way to get inside the JVM or CLR and tell you which method is the problem, or how garbage collection is working without some type of probe. The thing that will seperate MERQ's product is:

1. Since the engineer has named the transaction whatever they wish (one http request or a combination of many), this can be drilled down and directly correlated to the method calls that cause the problem for that transaction.

2. Unlike other profilers like DevPartner Studio and Appsight, this is intended to be used under load conditions.

With the Visual Studio add-on, you could take your C# code, stick your own defined transactions around it, execute the test from VS (launched the controller, etc), and while the controller was running you could navigate down to the specific methods/calls that are causing the bottleneck. You can then modify code in VS and start over again. I don't know of any way to get any closer than that.

At the same time, it is gathering information between servers (say web and db) to capture those big stored procedures and select statements that may be the problem - not the code.

Mercury is actually changing the diagnostics name to Lifecycle diagnostics for this reason. The focus is changing a bit to compete with the profiler category. You can get as white box as you want with the combination of LR and the Diags.

My experience has been that when properly educated about the value of monitors and probes, its very easy to get them installed on their infrastructure even in a highly secure business, such as financial institutions.

Scott Moore
Loadtester.com

I'm glad you liked the articles!

- Rational had the right idea with their performance testing tool, unfortunately, they were bought by IBM and that tool will be left to wither and die and the new (totally unrelated) tool takes a completely different direction that is not the slightest bit useful if you are not developing in an IBM pure (or nearly so) development environment. I'm very disappointed about this and have been very vocal in my displeasure, but I can't really blame IBM - I do believe they are making a sound business decision as a company, so I guess my displeasure is mostly selfishly driven.

- Yes, I was referring to VSTE (Visual Studio Test Edition) and RPT (Rational Performance Tester - the tool mentioned above that I'm not entirely happy about). I don't disagree that LR and other tools enable users to integrate with other diagnostics programs to go a long way down this path. My point is that this is an added expense, added integration and STILL doesn't tie directly in with the IDE. Don't get me wrong. I LOVE PerformaSure and others, but I've yet (as a consultant) been able to bring both a load generation tool and a diagnostic tool into an engagement. Organizations don't like to give access to their back end servers to consultants. In fact, they are SOO distrustful that I have never even been permitted to put a small piece of monitoring software on those servers to report back things like CPU utilization to the load generation tool.

VSTE & RPT remove that limitation, since all of those components are already there. It does remove the silly bureaucratic walls between tester and developer - unit, integration & system testing - functional & performance testing - testing & debugging, etc. All without added tool cost. And better (IMHO) VSTE is going to come onto the market at close to an order of magnitude cheaper than LR before adding diagnostic tools.

I fully admit that neither of these tools adds ANYTHING to mixed architecture environments that don't develop in either VS or Eclipse, and I hope I didn't imply otherwise. THAT is a hole that I see getting filled by either a) Mercury finally admitting that their tools are dreadfully overpriced or b) Someone like Borland (this is an example, not a speculation) taking the lessons learned from VSTE & RPT and building an equivalent tool that supports mixed architectures and various IDEs. At least that is my hope!

- So I am agreeing with your technical assessment - they add more to the testing process than the diagnostic process in comparison with the environments you mention... HOWEVER, they will dramatically add to the diagnostic process in organizations that couldn't afford the environment you mention, but can afford one of these tools -- assuming they are using a development environment where one of these will add value.


--
Scott Barber
Chief Technology Officer
PerfTestPlus, Inc.
Software Performance Specialist,
Consultant, Author, Educator
sbarber@perftestplus.com
www.perftestplus.com