WOPR7 In the rear view mirror

[textile] It was a privilege to hang out for 4 days with some of the brightest minds in testing, discussing experience reports on agile performance testing, and sharing personal stories and beers at the local pubs in the evenings. A brief synopsis of the experience reports, in the order delivered (from my understanding; guys, you may want to elaborate):

1. James Bach: Developing a performance testing strategy. Developed a use-case simulation in perl as basis for automation consultant to record scripts.
2. James Dobson: Performance testing in an Agile environment. An example of using continuous integration using JUnitPerf.
3. Julian Harty: Creating synergy between testers and developers. Scrum-based project for improving map point-of-interest searches on mobile phones.
4. Gordon McKeown: Enhancing a load testing tool on-the-fly to test a very chatty java app (built w/ Chordiant java code generator) under agile conditions.
5. Antony Gorman: Designing and developing a fast-performing historical event analysis system for the a UK transportation agency.
6. James Bull: Determine the performance capacity of completely converting a large UK newspaper to a web version. Process for performing agile testing that mimicked Development. Used JMeter, initially recording scripts, later shifting to table of urls when a requirement change removed need to time anything but html; defined min-max thresholds; used Zabbix to monitor system and graph results. Did not test using fully-converted data, and at that point performance ground to a halt. Conclusion: Scrum-like requirements gathering applicable to all performance projects.
7. Richard Florence: Performance testing as member of a MS Solutions Agile team on an audio-streaming app. Web services; difficult to record because of signed packets, expiry times, encryption. Tested using Visual Studio Team System, which calls unit-tests tightly integrated with app via APIs. Unit tests are self-maintaining, checked in with code. His performance tests call these functions; load tests data-driven from data in SQL Server, which also stored results (avoids file-contention issues with csv files).
8. Ainers Galvans: Performance testing an app using LoadRunner java RMI protocol and the application API.
9. Dan Downing: Applying lightweight process for load testing a dynamic and image-intensive ecommerce website using openSTA, quickly resolving the csv file-locking issues that arose from accessing large number of csv files.
10. Andrew Coggins: Managing agile testing project for large wireless provider.
11. Raymond Rivest: Load testing two vendor solutions for a medical application under pressure of time and distance in two days, using SilkPerformer for one and UI-based load testing for the second.

A few things I learned at WOPR7 (still digesting):

1. Rapid testing is a mindset to test quickly and less expensively, with excellent results (thanks James Bach).
2. Expanded universe of performance testing: can (should) begin upstream in development (capital A Agile development); other strategies besides load test phase before deployment i.e., continuous integration testing, unit testing; many open-source tools available (see list below).
3. An experience report is a story about what you did, not what you woulda coulda shoulda done.
4. Agile testing (capital A) is a subset of agile testing (small a); the latter is widely applicable.

Tools to investigate further:

1. Fit and FitNesse acceptance testing framework, where tests are expressed as tables of input data and results; the test is run by a small amount of code, and the pass/fail results are portrayed in the same table as colored cells.
2. Eclipse and Eclipse TPTP (Test & Performance Tools Platform) - a java UI for load testing.
3. Facilita Forecast - a tool to fill gap between open-source and LR for java and .NET apps.
4. Python for illustrating use cases.
5. Zabbix.org open source monitoring tools

Terms & concepts we discussed:

1. Leaky Abstractions - exceptions to a model (see article by Joel Spolsky)
2. Continuous integration - an extreme form of daily build + smoke test
3. Mock Object - Stub code w/ defined return value (e.g., return valid user if send correct user/pswd)

The big group learning: We as testers can influence better software through early, agile collaboration with developers! During the wrap-up, Antony conducted a brain-storming session on testability factors for agile performance testing that could be communicated to developers we work with to enhance the early performance testing process. Julian documented the numerous contributions, and I expect these will be synthesized and published shortly. I add my thanks and appreciation to Antony Marcano for his thoughtful orchestration of content and experience reports, to Paul Holland for tight but light facilitation, to Julian Harty for his essential role in making it all come together, and to Transition Solutions for facilities, food and fun. PS: James, the answer to your 1, 5, 6, 7 puzzle: 6 + 7 + 5 - 1 = 21 (Octal). [/textile]