testingReflections.com - test automation

STP Online Summit: Achieving Business Value with Test Automation

Tue, 23 Aug 2011 16:11:25 +0100

Due to the overwhelming success and positive reviews of the last STP Online Summit: Business Value of Performance Testing, we've decided to do it again -- only this time, we're going to explore Achieving Business Value with Test Automation.

Join me (while I continue practicing my radio host skills for my emergency back-up career as a sportscaster) and 7 other presenters that I consider to be elite practitioners, teachers, and thinkers in their test automation areas of specialization for 3 half days online to learn their tips and methods for achieving business value with test automation. If you or your organization are using, or thinking about using, automation to enhance or improve your testing, you're not going to want to miss this online summit. I honestly can't think of anywhere else you can get this concentration of relevant and thematically targeted information at a better price, but you be the judge:

When: Tuesday October 11 10:00AM - Thursday October 13 1:30PM PST

Cost: $195 USD before 9/26/11 $245 USD after 9/26/11

Theme: For more than 15 years organizations have been investing in the promise of better, cheaper, and faster testing through automation. While some companies have achieved demonstrable business value from their forays into test automation, many others have experienced questionable to negative returns on their investments. Join your host, Scott Barber, for this three day online summit, to hear how seven recognized leaders in test automation have achieved real business value by implementing a variety of automation flavors and styles for their employers and clients. Learn how to answer the ROI question by focusing on business value instead of testing tasks, and how to implement automation in ways that deliver that value to the business, not just to the development and/or test team.

CloudTest Lite - A Game Changer in the Performance Tool Market

Thu, 21 Jul 2011 03:16:22 +0100

Yesterday, SOASTA announced their new product, CloudTest Lite (Press Release). It's not common that I get excited about a tool product release, but this is different. This product has the potential to change the market for the better.

Scratch that. I'll be shocked if it doesn't change the market for the better.

Why is that, you ask? Consider the following attributes of CloutTest Lite:

It's a fully featured, easy to learn and use, enterprise class, modern, performance testing tool for web & mobile applications
All you need to use it is a reasonably modern machine connected to the internet and a web browser.
- You don't need to buy, install, configure or maintain load generation machines.
- The "license" is tied to your personal credentials, so you can design, create, execute, and analyze your tests from any machine you want without needing to figure out how to point to the license server, or how to get onto the corporate network from your favorite internet cafe.
- You can even do much of the design, test enhancement, and analysis entirely off-line.
You can simulate up to 100 virtual users any time you want. No more scheduling time on the controller days or weeks in advance guessing the app will be ready for your test. No more having to wait until your next scheduled time to re-run your test when you see something 'wonky' in your data.
It's free.
- Yes, I said free.
- As in, you never need to pay a dime. Not today, not when the trial expires, not a year from now to continue your maintenance contract.
- That's right, it is free from now until the sun explodes (or at least until well beyond when anything we're building or planning to build today is long gone and forgotten)

Imagine the implications:

Cool Tools - for customer data etc

Thu, 05 Nov 2009 13:24:07 +0000

Test data is often a bottleneck on testing projects. Security and privacy issues can require complex massaging of existing information, Today, there are tools to make it easy to create it from scratch! I find two in particular quite useful.

I’ve been telling people about generatedata.com for several years. It is an amazing tool for generating data for customers or other items, including items chosen from lists and data ranges. It is free, customizable and can even be run off a USB (after it is customized for the computer and database etc).

Short and long game thinking, tests driving design and CRAP metrics

Thu, 28 May 2009 05:59:56 +0100

Kent Beck recently posted To Test or Not to Test? That’s a Good Question on the complex “theory versus practice” issue of always automating tests, where he states,”Then a cult of [agile] dogmatism sprang up around testing–if you can conceivably write a test you must”. By classifying projects into long game and short game, he argues that ROI becomes a major issue on whether a test stays manual. He says “Not writing the test for the second defect gave me time to try a new feature”, but several people commented that this was a technical debt tradeoff, and Guilherme Chapiewski noted he had done the same thing with a Proof of Concept that went live then he had to rewrite major chunks later. It is interesting that this ROI discussion is reflecting the experiences of the pre-agile functional automation community. Back in November 2001 (Wow! Long time ago!!), I posted to the Agile Testing list some considerations for not automating . While many of these were from the context of two separate development teams and the automaters using expensive test tools, the risks of incomplete automation and insufficient ROI dominate. The benefits of having the same people both develop the code and the tests are great, and beyond my experience when I wrote that post.

I think the ROI issue for code-based tests will go away over time. Much of the creation of code-based tests is mechanical. Just as programming languages replaced assembler and took care of fiddly details (what registers to use, low level comparisons etc) and build utilities replaced simple text file include statements, I think that soon it will be standard practice to have tool-created unit testing to handle mocking, dependency injection and assert-based testing. Mocking was originally very manual, then tools were developed. Dependency Injection was very manual,then tools were developed. For assert-based testing, we’ve already seen Agitar’s tools , zentest and now pex amongst others. I think these tools will become standard, just as coverage tools are now standard in IDEs when they originally were luxuries costing tens of thousands of dollars. Another variation of this is tools like Celerity recently blogged about by Jeffrey Frederick. Celerity is a fast way to run GUI web tests, but could be handled as a mechanical translation not a manual one. Some meta language could generate Celerity and selected browser tests in a single step.

Mechanically generated tests are cheap to produce and overcome ROI issues. However, they only reflect the current code. The benefits of test design infusing the coding approach are missing. If tests are not being automated for whatever reason, some analysis of the refactoring risk should be done, at least to know where and what the error-prone code is. One way of doing this is using the Agitar-created CRAP metric , which Bob Martin recently blogged about as a way to keep design clean. While I currently believe all code should be created test first wherever possible, techniques like the CRAP metric can highlight the complicated bits for refactoring where possible. While it may be a great intellectual challenge, there is no need to refactor a complex industry standard algorithm. [Aside: is there an inherent advantage to doing test first design all the time? Perhaps, just as renaissance masters only painted and sculpted hand and faces and left the rest to their workshop staff, we only need to focus on core functions for test first and do the rest test last?]

As Kent says,”By insisting that I always write tests I learned that I can test pretty much anything given enough time.” Time is often a rare commodity, so Kent argues compromises are often needed in short goal projects. As Ron Jeffries said in a comment on Kent’s post, “My long experience suggests that there is a sort of knee in the curve of impact for short-game-focused decisions. Make too many and suddenly reliability and the ability to progress drop substantially.” I hope that advancements in mechanical generation of tests don’t push us into a short game perspective, impacting the use of hand crafting tests to drive design. At the same time, metrics that can be run as part of the build to highlight areas for refactoring on all projects are proving valuable (and I’m looking forward to state coverage ). By any measure, these are interesting times we live in. Long live long game thinking!

The short and the long of IT: two videotaped presentations

Sun, 07 Jun 2009 05:58:45 +0100

Since the middle of the year, I’ve presented and facilitated about 12 hours of sessions at 2 traditional and 3 open space conferences, plus a Googleplex visit. Two presentations are now on video, both filmed on the other side of the world from my usual location of Melbourne, Australia.

A lightning talk (at the functional tools workshop held before Agile 08 in Toronto, Canada) called Shades of Green discusses how the “green” passing tests of functional automation may not be as green as they seem. Note the static pose to stay within camera range, compensated for by the wildly waving arm. And yes, the audience was not limited to a leg and a foot.

An order of magnitude longer at around 50 minutes, a Google Tech Talk (filmed at the San Francisco Googleplex) called 80:20 Rules! Building Software Smarter looks at formal and informal ways to get significant improvements in creating software, including various puzzles and questions for viewers. I had looked at some tech talks by other people I know, and they had been watched around 1000 times over a year or so. It looks like I may hit that mark only a few days after the video was posted which is great. I hope my talk inspires people to build their software smarter. Can I turn “shades of green” into a similar talk? Probably not!

Smart monkey

Mon, 23 Oct 2006 13:39:35 +0100

For those of you who often test databases, here's an affordable tool that can help you preparing your test data or size up your database.

http://www.sqledit.com/dg/index.html

It does a pretty good job....

Google London Test Automation Conference Talks on Video

Tue, 01 Apr 2008 15:16:25 +0100

As previously mentioned the talks from Google London Test Automation Conference (LTAC) are all now available on Google video

Enjoy!!

Google London Test Automation Conference (LTAC) 2006

Wed, 16 Jan 2008 09:33:32 +0000

I've just spent the last two days (Thurs 7th - Fri 8th Sept 2006) at Google's LTAC. I can safely say, this is one of the best Software Testing Conferences that I've attended or even read the programme for. I'm not just saying this because Google were very generous in ensuring that attendance was free, providing the facilities, organising the event, feeding us, providing unlimited refreshments (not to mention beer on Thursday night). The sheer quality of the talks and topics alone were great. The attendees were also exceptional individuals... the breaks were as enlightening as the talks themselves.
images by Adewale Oshineye

Google LTAC was the brainchild of Allen Hutchison (pictured left). As he put it "the great thing about organising your own conference is that you get to include the topics you want". Well, Allen, you weren't the only one who wanted to hear those talks. There was a real community spirit in the air! It was great to meet some people who I've been in touch with for some time by e-mail but have never met. It was also great to cement relationships with people I've been getting to know over time.

What was especially cool about this conference is that there was a healthy mix of developers who are serious about testing and testers who are serious about developing automated tests. I hope to see more like these!

All the talks will be appearing on Google Video, hopefully by the end of next week. To whet your appetite, here is the programme...

HP to buy Mercury Interactive

Wed, 26 Jul 2006 20:51:34 +0100

On Tuesday 7/25/2006, CNNMoney.com (along with *many* others) broke the news that the rumored HP/Mercury deal is really happening. A summary and my reaction is below. See the entire release here and draw your own conclusions.

July 26 2006: 9:22 AM EDT
NEW YORK (Reuters) -- Hewlett-Packard agreed on Tuesday to buy Mercury Interactive for about $4.5 billion in stock, or $52 per share, in a bid to expand the computer maker's business software operations.

Writing tests that don't depend on your environment

Thu, 16 Feb 2006 09:02:13 +0000

Laurent's latest Incipient.oO{} entry really hit home when I read it. He mentions how the QA team he is helping has been writing tests that work in their own environment but not on someone else's or worse, on the official QA box !

Wait for it...wait for it...

Fri, 02 Dec 2005 19:09:53 +0000

So it has been a couple of weeks, has it already -sure has -feels longer, since I was at STARwest and I am still reflecting on the event; which was to the usual high standards.

*round of applause for the SQE Team*

Well it sure as hell reinvigorated me, is it me or have I become rather chatty once more – was rather quite for a while there, which kind of got a pot and kettle comment from Harry Robinson (I am such a namedropper, it is a rather vain glory, I tend to suffer less well in others).

Software Test & Performance Conference 2005

Fri, 25 Nov 2005 21:33:14 +0000

I really enjoyed the STP conference. I am not afraid to say that it was the best conference that I've ever attended. Almost all the presentations that I attended were very good. I was especially impressed by Hung Nguyen (I've never been to one of his presentations before). His straight to the point presentation was about test automation outsourcing.

Test Automation

Thu, 17 Nov 2005 03:36:20 +0000

[textile] Last weekend we held the November session of the Indianapolis Workshop on Software Testing. The attendees were:

Andrew Andrada
Charlie Audritsh
Michael Kelly
Mike Slattery
Dana Spears
Gary Warren
Chris Wingate

The topic we focused on for the five-hour workshop was automated testing.

Session Based Testing: Automation Not Exploratory Testing

Mon, 01 Aug 2005 17:38:12 +0100

[textile]
I have been an advocate of exploratory testing, recently more of an advocate of good exploratory testing as stated by "James Bach":http://www.satisfice.com in one of his recent blogs, as the basics and ideas are widely described and used. However, I have a strong anchoring basis, as described by "Mike Kelly":http://www.michaeldkelly.com/ in his blog "here":http://www.testingreflections.com to process and rather like "Jonathan Khol":http://www.kohl.ca/blog/ a thinking preference to automation and coded solutions.

Testing an Application in Layers

Fri, 05 Aug 2005 20:25:58 +0100

There is often debate about test automation versus manual testing. When I think about testing, I look at an application in 3 broad layers: the code (on the machine side), the system (where the finished software lives), and the visible...