OpenSTA, the free performance testing tool, versus the big-guns...
Submitted by Antony Marcano on Fri, 30/07/2004 - 02:40.
**NOTE:**The content of this table can be reproduced and referenced freely all whilst the reproduction or reference acknowledges the authors: Andy O'Brien and Antony Marcano of "etest associates":http://www.etest-associates.com; All information was correct to the best of the knowledge of the authors at the time of writing; the information is provided as-is and is used at your own risk
I was recently introduced to Scott Barber by Steven Splain. Scott is a performance testing practitioner and one of the organisers of the Workshop on Performance and Reliability (WOPR) (thanks for the invite Scott).
Before the WOPR came into the equation, our e-mail conversation was about OpenSTA- a powerful opensource web performance, load and stress testing tool. I have used it extensively and contributed to the support group and project strategy through my ongoing discussions with Dan Sutcliffe.
Scott was intending to do a comparison between LoadRunner and OpenSTA so I thought I would help him get it started since Andy O'Brien and I (Antony Marcano) produced one this time last year... we just never got round to publishing it. So, here it is... I hope you find it useful...
| Item | Description | LoadRunner | OpenSTA |
| Protocols | The communication protocols that can be captured, manipulated and replayed by the tool. | Many supported. Protocols are charged per item. Has a multi-protocol recording feature. | HTTP 1.0 / 1.1 / HTTPS (SSL) only. |
| Playback functions | Replaying of the script and script debugging facilities. | Extended logging supports view of parameter values and Server messages. Also view and comparison with ‘recorded’ version of web page view and client response messages. Debugging facilities in script generator, step and breakpoints. | Similar playback facilities, but no integrated comparison function. The debugging functions are in the controller, including set break points and single stepping. |
| Scripting language | The medium used to represent the captured protocol data and manipulate the data for play-back. | Called TSL, it uses standard syntax for “C” et allows C function libraries to be added. Has extensive customised functions for the different protocols supported by the tool. | Called SCL, it uses a proprietary, “BASIC” like language that has special automation scriptingfacilities. Is limited in available functions, such as string manipulation and supports direct DOM addressing. |
| Extensibility | The ability to increase the functionality of the tool. | Additional TSL or “C” function libraries, limited to functional capabilities of the tool. | SCL script modules can be defined in â??Includeâ?? files. Open Source therefore new tool functionality can be added using C++. |
| Scripting Interface | The interfaces supplied by the tool application for the purpose of script editing. | Captures in several modes, high level context based and low level HTTP view. Has both a graphical tree structure et a script view. Script view has function sensitive entry. | Has low-level HTTP protocol view and provides graphical tree representation of the â??Document Object Modelâ?? (DOM) structure. Viewable captured HTML rendering and addressable server-header table. Some language sensitive, syntax colour coding functionality. |
| Correlation | The task of substituting values in dynamic data to enable successful playback. | Automated correlation faculties. Including during recording, after recording and comparing recordings with playback results. Not available for all modes of capture. | Manual correlation using graphical interactive â??DOMâ?? structure. Facility to automatically generate script code to aid variable substitution. |
| Cookie Management | Detection, recording and playback of HTTP cookies. Both tools need additional code to manage JavaScript generated cookies. | HTTP header cookies are managed automatically and can be manipulated manually if required. | HTTP header cookies are managed automatically and can be manipulated manually if required. |
| Parameterisation | Automatically changing dynamic data values for more accurate emulation of real users. Often essential for session management. | Extensive facilities for data entry including wizard interface to DB interrogation. No standard function to lock data sources and maintain uniqueness of concurrently accessed data across distributed tests. | Extensive facilities for data entry including wizard interface to automatically generate test data. Standard functions for sequential, random and pseudo-random data-file access. Has standard common locking facilities for maintaining uniqueness of parameters for an individual load injector or across all injectors on a distributed test. |
| Controller | Application that manages and conducts a test. | Facilities for real-time monitoring. Automatic scenario generation. Individual control of vusers, scripts and groups of scripts. Scheduling, percentage runs et iterations. | Facilities for real-time monitoring. Simple drag et drop multi-scenario test configuration supporting modular scripting and addition of new scenarios/Vus at run-time. No automatic scenario generation. Allows test-wide and user-specific HTTP monitoring and debugging while multi-user load in progress. |
| Monitoring | Resource usage information is captured during execution. It can be shown during execution and used to build performance reports. | Many supported. On-line graphics during execution. Supplied with Apache, Netscape et IIS, other monitors are charged per item. Results used for reporting. New facility to allow remote users to monitor real-time results via a browser based interface. Note: To monitor through a firewall requires TCPIP access through a specific port. Future versions of LoadRunner should use HTTP messaging et avoid this issue. | Supports Integrated real-time, graph view of Windows NT/2000 Performance (Perfmon) et SNMP collectors. Various measurements of test progress including VU-specific, custom status and activity information. Web-relay allows monitors to run on remote machines beyond firewall. On-line graphics during execution, et monitored results used for reporting. |
| Distributed tests | The ability to distribute the generation of load across multiple load-injector machines. | Supports multiple load-injectors managed by a single controller. | Supports multiple load-injectors managed by a single controller. Uses TCP/IP if on the same network or the Web-relay feature uses HTTP to control load-injectors located within remote DMZs. |
| IP Spoofing | The ability to emulate the behaviour of different IP addresses accessing a system. Particularly useful with Load balancing systems. | Supports IP spoofing including automated router updates for IP forwarding. | No inbuilt features to enable IP spoofing. |
| WAN/LAN emulation | The ability to emulate the behaviour of different network infrastructures during a test. | New feature added to version 7.6. Allows the emulation of latency, packet loss, link faults et dynamic routing effects over the LAN used in a test. Requires a special license. | No inbuilt features to enable WAN/LAN emulation. |
| Caching | The ability to emulate the caching of pages as performed by a web browser. | Can control browser cache emulation during playback and control setting for each individual vuser. | No specific faculties, although can be emulated with simple script code. |
| User connection speed emulation | The ability to emulate the different network speeds that can be used by real users. | Can emulate different network speeds during playback | No inbuilt features to emulate user-connection speed emulation. |
| Distributed/Remote load generation | To generate large loads additional load generator machines are required. These machines must be centrally controlled. | Can control multiple load generators and collect results. | Can control multiple load generators and collect results. |
| Can control load generators located at remote networks (through a firewall) if required using HTTP ports. | Can control load generators located at remote networks (through a firewall) if required using HTTP ports. | ||
| Reporting et analysis | The facilities to examine and investigate the results of a test including timers and monitored resources and display the results in graphical format. | Sophisticated et large range of charts and graphs with overlay facilities. -Automatic report generation into MS-Word. Analyser is a separate application that can be distributed to users. | Simple charts and graphs sufficient for analysing key load related statistics and resource usage monitors. Resource usage monitors supports graph overlays. Can be exported to Microsoft Excel. No license restrictions on OpenSTA distribution thus stats can be viewed by any user with access. -Free tools and excel macros available through public user-forums. |
| Scalability | The ability of the tool to generate numbers of virtual users and the corresponding resource usage. Actual resource use depends on the number, size and complexity of the scripts. | Resource limitations are number of threads and RAM. Approx 1 vuser per ½ Mb RAM for NT/W2K. Windows 95, 98 et Unix are less efficient. Max. approx 1,500 VUs per PC. | Main resource used is RAM. Tested for simple ASP pages, reaching up to 3,000 users on load generator of 1Gb RAM on a single P4 processor et W2K. Unconfirmed report of a limit of 1664 Vus per Win2K machine for complex scripts. Suspected thread limitation. No licensing limitations. |
| Initial Costs | Purchase costs of software and licenses, excluding upgrades or support. | From £16,000 for basic package with no virtual users. Additional charge for each protocol and monitored resource and for virtual users. | FREE� download from SourceForge via www.OpenSTA.org. Downloads available: Previous versions; Automatic installer or current source code (with simple build instructions for MS C++ Visual Studio 6). |
| Costs of Virtual users | Most commercial tools charge on the basis of the number of virtual user available. Extra hardware is an additional cost. | Prices vary but for the purpose of expectation: additional vusers cost from £10K for 25 VUs to £66K for 1000 VUs. Also temporary VU days at £3.50 each per day (min 1000). This is not a quoted price. | FREE. No licensing limitations. |
| Support et consultancy | The support services available for the tool and some costs. | By M.I., approx 1/5 of initial cost per annum. Includes upgrades. MI et their partner companies also provide consultancy (including etest associates). | Various independent resources. etest associates charged from £50 per incident for remote tech support. Consultancy rates available on request.Numerous online resources including web and e-mail forums. Upgrades are free (approx every 3-6 months) |
| Training | The training services available for the tool. | MI has a range of courses from around £400 per day per person. Many partners also provide training. | Specialist companies provide tailored training, prices vary. |
| System requirements | The operating systems required to host the tool. (Not the OS of the system under test). | MS windows 2000, NT4 (sp6a),XP-Pro (also 95 et 98 cannot run multi-threaded) - Load generators also support limited generator functionality on: Unix: HP, Solaris, Linux | MS windows 2000, NT4 (sp5+), XP-Pro. |
| Hardware requirements. | The hardware requirements to host the tool. (Not the hardware of the system under test). | Min: Pentium 350 et 128M - Load generators: Pentium 1GHz et 1 MB per VU. | Min : Pentium 200 et 80MB RAM. Pref: Pentium 500MHz+ et 128MB+ RAM. |
| Access to source code | Availability of the source code of the tool itself. | Unavailable. | Open Source GNU public license - ‘C++’ Language. |
| A users opinion. | Opinion of Internet performance engineers who have experience with both tools. | Has a very user friendly user interface and fantastic monitoring et results analysis. Automatic correlation and improved script recording facilities can aid productivity. Very flexible scripting functions et great help documents. Complexity of options and layout of controller are drawbacks. The availability (at a cost) of numerous protocols for recording is a significant strength. | Easy to use interface and excellent scalability. Inbuilt results analysis is slightly limited in comparison to LoadRunner. The captured data is open and can also be exported to Excel in two clicks. Drag et drop scenario set-up and control is very intuitive and easy to interact with, simplifying creation of scenarios with modular scripts. Manual correlation can be a headache but is eased slightly by using third-party ‘Diff’ tools and the inbuilt GUI DOM addressing feature. A relative shortage of standard script language functions but enough to get most HTTP load testing jobs done. If not, with “Includes” support and the availability of the source code, it is very extensible |