Performance Tool Comparison: How LoadRunner,OpenSTA and JMeter stack up at runtime -1

[textile]

Comparing features among tools, especially performance testing tools, brings another dimension of complexity. The basic premise behind a performance test is to understand what response times can be expected by simulation of user traffic modeled along user actions. These response times may be TTLB (Time to Last Byte) measurements - and usually exclude browser render time. It is merely the time since the request was issued to the time the last byte of the response flowed into the network card on the machine. From the wire to a nicely formatted page on a browser is something not factored into the test.

This is not something Mercury explains clearly enough in the documentation, which is why the A/LR yahoogroups and the MI Support site are inundated with variations on this question:

* Why are my browser timings and LR timings so different?
* Can I use WR/QTP users in an LR scenario?
* I'm looking at a VB/Delphi/Scoobydoo application - what vuser type do I use?

And essentially, there is just one answer: *Think protocol, not application!*

Still, there is another fundamental question. Are response times recorded by tools accurate ? An observation is a point of view and is dependent on several things:

# Are requests going out for all the elements on a particular page? This addresses accuracy of recording.
# Are these requests in the same sequence? This addresses accuracy of replay.
# Is response time calculated TTLB or TTFB? This addresses the accuracy of the results.

Some time ago, between projects, my research team[1] here at Kanbay, did a rather interesting runtime comparison of *JMeter*, *LoadRunner* and *OpenSTA*. The idea was to address all the points mentioned above. Our target application was the Mercury Tours application that ships with the tool.

The results of recording were rather startling and as can be seen from the table below, no one tool completely recorded all elements. This despite the fact that all three tools using similar recording technology - that of a browser proxy. I have no explanation to offer, since I checked the recording options and matched them as closely as possible across all tools.

|*Transaction*|*Recorded in LoadRunner*|*Recorded in JMeter*|*Recorded in OpenSTA*|
(dark). |Home||||
(dark). ||/mercuryWebTours/|/mercuryWebTours/|/mercuryWebTours|
(dark). ||/mercuryWebTours/header.html|/mercuryWebTours/header.html|/mercuryWebTours/header.html|
(dark). ||/mercuryWebTours/welcome.pl|/mercuryWebTours/welcome.pl|/mercuryWebTours/welcome.pl|
(dark). ||/MercuryWebTours/home.html|/MercuryWebTours/home.html|/MercuryWebTours/home.html|
(dark). ||/mercuryWebTours/nav.pl|/mercuryWebTours/nav.pl|/mercuryWebTours/nav.pl|
(dark). ||/MercuryWebTours/images/login.gif|/MercuryWebTours/images/login.gif|/MercuryWebTours/images/login.gif|
(dark). ||/MercuryWebTours/images/signup.gif|/MercuryWebTours/images/signup.gif|/MercuryWebTours/images/signup.gif|
(dark). ||/MercuryWebTours/images/admin.gif|/MercuryWebTours/images/admin.gif|/MercuryWebTours/images/admin.gif|
(dark). ||/MercuryWebTours/images/sun_swede.gif|/MercuryWebTours/images/sun_swede.gif|/MercuryWebTours/images/sun_swede.gif|
(dark). ||/MercuryWebTours/images/banner_merctur.jpg|/MercuryWebTours/images/banner_merctur.jpg|/MercuryWebTours/images/banner_merctur.jpg|
(dark). ||/mercuryWebTours/images/banner_animated.gif|/mercuryWebTours/images/banner_animated.gif|/mercuryWebTours/images/banner_animated.gif|
(dark). |Login||||
(dark). ||/mercuryWebTours/login.pl|/mercuryWebTours/login.pl|/mercuryWebTours/login.pl|
(dark). ||/mercuryWebTours/login.pl|/mercuryWebTours/login.pl|/mercuryWebTours/login.pl|
(dark). ||/mercuryWebTours/nav.pl|/mercuryWebTours/nav.pl|/mercuryWebTours/nav.pl|
(dark). ||/MercuryWebTours/images/sun_with_logo.gif|/MercuryWebTours/images/sun_with_logo.gif|/MercuryWebTours/images/sun_with_logo.gif|
(dark). ||/MercuryWebTours/images/itinerary.gif|/MercuryWebTours/images/itinerary.gif|/MercuryWebTours/images/itinerary.gif|
(dark). ||/MercuryWebTours/images/in_home.gif|/MercuryWebTours/images/in_home.gif|/MercuryWebTours/images/in_home.gif|
(dark). ||/MercuryWebTours/images/flights.gif|/MercuryWebTours/images/flights.gif|/MercuryWebTours/images/flights.gif|
(dark). ||/MercuryWebTours/images/signoff.gif|/MercuryWebTours/images/signoff.gif|/MercuryWebTours/images/signoff.gif|
(dark). ||/MercuryWebTours/images/banner_merctur.jpg|/MercuryWebTours/images/banner_merctur.jpg|/MercuryWebTours/images/banner_merctur.jpg|
(dark). |ClickFlights||||
(dark). ||/mercuryWebTours/reservations.pl|/mercuryWebTours/reservations.pl|/mercuryWebTours/reservations.pl|
(dark). ||/MercuryWebTours/images/in_flights.gif|/MercuryWebTours/images/in_flights.gif|/MercuryWebTours/images/in_flights.gif|
(dark). ||/MercuryWebTours/images/home.gif|/MercuryWebTours/images/home.gif|/MercuryWebTours/images/home.gif|
(dark). ||/MercuryWebTours/images/continue.gif|/MercuryWebTours/images/continue.gif|/MercuryWebTours/images/continue.gif|
(dark). ||/MercuryWebTours/images/splash_findflight.gif|/MercuryWebTours/images/splash_findflight.gif|/MercuryWebTours/images/splash_findflight.gif|
(dark). ||/MercuryWebTours/images/itinerary.gif|/MercuryWebTours/images/itinerary.gif||
(dark). ||/MercuryWebTours/images/sun_with_logo.gif|/MercuryWebTours/images/sun_with_logo.gif||
(dark). ||/MercuryWebTours/images/signoff.gif|/MercuryWebTours/images/signoff.gif||
(dark). ||||/mercuryWebTours/FormDateUpdate.class|
(dark). ||||/mercuryWebTours/FormDateUpdate.class|
(dark). ||||/mercuryWebTours/CalSelect.class|
(dark). ||||/mercuryWebTours/CalSelect.class|
(dark). ||||/mercuryWebTours/Calendar.class|
(dark). ||||/mercuryWebTours/Calendar.class|
(dark). |SearchFlight||||
(dark). ||/mercuryWebTours/reservations.pl|/mercuryWebTours/reservations.pl|/mercuryWebTours/reservations.pl|
(dark). ||/MercuryWebTours/images/splash_searchresults.gif|/MercuryWebTours/images/splash_searchresults.gif|/MercuryWebTours/images/splash_searchresults.gif|
(dark). ||/MercuryWebTours/images/startover.gif|/MercuryWebTours/images/startover.gif|/MercuryWebTours/images/startover.gif|
(dark). |SelectFlight||||
(dark). ||/mercuryWebTours/reservations.pl|/mercuryWebTours/reservations.pl|/mercuryWebTours/reservations.pl|
(dark). ||/MercuryWebTours/images/splash_creditcard.gif|/MercuryWebTours/images/splash_creditcard.gif|/MercuryWebTours/images/splash_creditcard.gif|
(dark). ||/MercuryWebTours/images/purchaseflight.gif|/MercuryWebTours/images/purchaseflight.gif|/MercuryWebTours/images/purchaseflight.gif|
(dark). |PurchaseFlight||||
(dark). ||/mercuryWebTours/reservations.pl|/mercuryWebTours/reservations.pl|/mercuryWebTours/reservations.pl|
(dark). ||/MercuryWebTours/images/splash_flightconfirm.gif|/MercuryWebTours/images/splash_flightconfirm.gif|/MercuryWebTours/images/splash_flightconfirm.gif|
(dark). ||/MercuryWebTours/images/bookanother.gif|/MercuryWebTours/images/bookanother.gif|/MercuryWebTours/images/bookanother.gif|
(dark). |Logout||||
(dark). ||/mercuryWebTours/welcome.pl|/mercuryWebTours/welcome.pl|/mercuryWebTours/welcome.pl|
(dark). ||/mercuryWebTours/nav.pl|/mercuryWebTours/nav.pl|/mercuryWebTours/nav.pl|
(dark). ||/MercuryWebTours/images/banner_merctur.jpg|/MercuryWebTours/images/banner_merctur.jpg|/MercuryWebTours/images/banner_merctur.jpg|
(dark). ||/MercuryWebTours/home.html|/MercuryWebTours/home.html||
(dark). ||/MercuryWebTours/images/login.gif|/MercuryWebTours/images/login.gif||
(dark). ||/MercuryWebTours/images/signup.gif|/MercuryWebTours/images/signup.gif||
(dark). ||/MercuryWebTours/images/admin.gif|/MercuryWebTours/images/admin.gif||
(dark). ||/MercuryWebTours/images/sun_swede.gif|/MercuryWebTours/images/sun_swede.gif||

fn1. Research team included: _Rajat Satendar, Puja Jaishree, Vikram Chandna, Hemant Joshi, Nithya Seshachalam_ and yours truly.