From "Staplers" to "ScrewDrivers" to "Open Source Enterprise Test-Tools Suite"

[textile]"Mike Kelly":http://www.testingreflections.com/user/view/55 started with [Stapler Testing] and now he moves onto "Screwdrivers":http://www.testingreflections.com/node/view/1447 . The first was an exercise in "Thinking like a Tester", the latter is an interesting metaphore for Test Tools. I will try to hide my envy, as I wasn't able to go to the AWTA due to other commitments. I wish I could have been there as it sounds like it was excellent! In his "Screwdrivers post":http://www.testingreflections.com/node/view/1447 he says...

If you think of your average enterprise level automation tool, it's similar to a mutlihead screwdriver. These tools offer everything under the sun (performance testing, functional testing, test management, defect tracking, data-driven technology, etc...) all you have to do is purchase the tool and start recording!

Having spent three days in Mike's company last September at another workshop I know that this is just a general remark, however, for the more naive reader, it is worth noting that in reality "Enterprise Testing Tools" or "Commercial Testing Tools" do not quite offer "everything". They offer tools that account for the needs of the majority, as perceived by the tools vendors. I also know that he knows that it is rarely as simple as "purchase the tool and start recording"... this is, however, the perception that the vendors would like us to have. Generally, commercial tools will provide you with three key interfaces through which you can test your system: * GUI - (e.g. your typical record/playback tool) * Network Protocol - particularly for load testing * Database - Load Testing and Data manipulation What I am yet to find in a commercial test tools suite is the straight-forward ability to bypass the GUI as described in Brian Marick's article "Bypassing the GUI (PDF)":http://www.testing.com/writings/bypassing-the-gui.pdf There is a shift in perspective on test-automation and increasingly Testers want to drive business logic tests directly into the layer of the software that handles business logic - which is often, not the GUI. This is why I am personally very excited about "Visual Studio Team Test 2005":http://lab.msdn.microsoft.com/vs2005/teamsystem/tester/default.aspx , part of "Visual Studio Team System 2005":http://lab.msdn.microsoft.com/vs2005/teamsystem/default.aspx . It will have unit, functional and load testing test-types out of the box. Its' load testing capabilities will initially be limited to web-applications however. New test-types and plugins can be developed and hooked into its open architecture. This is what "OpenSTA":http://www.opensta.org was meant to be but it just didn't get there and remained a load testing tool for web applications... OpenSTA is undergoing a push in development and is looking for developers to help get OpenSTA ahead of the game again. Volunteers can get involved through the "OpenSTA Developers' Mailing List":http://lists.sourceforge.net/lists/listinfo/opensta-devel . Another area that commercial tool-suites generally don't cover is security. This is an area that is usually handled by specialist providers and Open Source tool-kits. Open Source Test Tools are more like your garage or shed. You have every tool you need for everything from DIY to Gardening, accumulated over time. They are a mix of brands, colours, shapes and sizes - no handle being quite the same. This is the current problem with Open Source Test Tools, in my opinion. It is worth noting that this isn't just a problem with Open Source tools, since many commercial vendors have created their suites by buying companies that have a tool for a specific type of testing or testing activity. Integration between tools within a commercial suite is often far from what you would be lead to believe by the vendor and often, each tool uses a different language. What we need is a common platform into which plugins can be written - or an industry-standard open architecture that is generic and allows you to learn one language that will enable you to test almost anything. In some ways, this appears to be the way that "FITNesse":http://www.fitnesse.org seems to be going and perhaps also "Ruby":http://www.ruby-lang.org/ (albeit in different ways). In the commercial environment, it also appears to be where Microsoft are taking Visual Studio. If "Ruby":http://www.ruby-lang.org/ does become the defacto language for the tester, perhaps we will see ourselves move towards a set of tools that can be combined and used effectively without the learning curve of using disparate tools and languages. Perhaps this will become the unique-selling-point for using the .NET framework when VSTS2005 is available later this year or the reason why Open Source advocates shift to Ruby and/or FitNesse? Whichever way the market goes, I personally hope to see a collaborative effort to bring together the powerful Open Source tools available. Then we can all clear out our sheds, garages and tool-boxes of our pick-n-mix screw-drivers, hammers, mallets, axes, shovels and spades and replace them with one, consistent, community lead Enterprise Suite of Open Source Test-Tools.