Google Testing Blog
Google Testing Blog
Website:
Description:
Slogan: If it ain't broke, you're not trying hard enough...
Debugging Sucks; Testing Rocks!
(RSS Feed via FeedBurner of the GoogleTestingBlog.)
Debugging Sucks; Testing Rocks!
(RSS Feed via FeedBurner of the GoogleTestingBlog.)
Last update:
1 year 27 weeks ago
Code coverage goal: 80% and no less!
Submitted by noreply@blogger.com (Alberto Savoia) on Tue, 20/07/2010 - 17:40.by Alberto Savoia
I first posted this article a few years ago on the Artima Developer website; but the question of what's adequate code coverage keeps coming up, so I thought it was time for a repost of Testivus wisdom on the subject.
Testivus on Test Coverage
Early one morning, a young programmer asked the great master:
“I am ready to write some unit tests. What code coverage should I aim for?”
The great master replied:
“Don’t worry about coverage, just write some good tests.”
The young programmer smiled, bowed, and left.
Later that day, a second programmer asked the same question.
The great master pointed at a pot of boiling water and said:
“How many grains of rice should I put in that pot?”
The programmer, looking puzzled, replied:
“How can I possibly tell you? It depends on how many people you need to feed, how hungry they are, what other food you are serving, how much rice you have available, and so on.”
“Exactly,” said the great master.
The second programmer smiled, bowed, and left.
Toward the end of the day, a third programmer came and asked the same question about code coverage.
“Eighty percent and no less!” Replied the master in a stern voice, pounding his fist on the table.
The third programmer smiled, bowed, and left.
After this last reply, a young apprentice approached the great master:
“Great master, today I overheard you answer the same question about code coverage with three different answers. Why?”
The great master stood up from his chair:
“Come get some fresh tea with me and let’s talk about it.”
After they filled their cups with smoking hot green tea, the great master began:
“The first programmer is new and just getting started with testing. Right now he has a lot of code and no tests. He has a long way to go; focusing on code coverage at this time would be depressing and quite useless. He’s better off just getting used to writing and running some tests. He can worry about coverage later.
The second programmer, on the other hand, is quite experienced both at programming and testing. When I replied by asking her how many grains of rice I should put in a pot, I helped her realize that the amount of testing necessary depends on a number of factors, and she knows those factors better than I do – it’s her code after all. There is no single, simple, answer, and she’s smart enough to handle the truth and work with that.”
“I see,” said the young apprentice, “but if there is no single simple answer, then why did you tell the third programmer ‘Eighty percent and no less’?”
The great master laughed so hard and loud that his belly, evidence that he drank more than just green tea, flopped up and down.
“The third programmer wants only simple answers – even when there are no simple answers … and then does not follow them anyway.”
The young apprentice and the grizzled great master finished drinking their tea in contemplative silence.
There, but for the grace of testing, go I
Submitted by noreply@blogger.com (James Whittaker) on Sun, 18/07/2010 - 00:40.By James A. Whittaker
I've had more than a few emails about "antenna-gate" asking me to comment and suggesting clever, stabbing rebukes to a fallen competitor. I might aim a few of those at my own team in the future, some were genuinely funny, but none of them will appear here. Instead I offer first a word of caution and second a reflection that my Mom used to intone whenever disaster occurred around her. It's called "counting your blessings."
First, a caution that those of us who live in glass houses really should keep stones at arms length. The only way anyone can rebuke Apple, without risk of waking up one morning sucking on their own foot, is if they write no software or have no users. Apple does a lot of the former and they enjoy many of the latter. Bugs like this make me sick when they are mine and nervous when they aren't. If any tester in the industry isn't taking stock right now then they either aren't producing any software or aren't in possession of any users, at least ones they wish to keep.
Second, taking stock has made me realize that I enjoy some important blessings that make the infinite task of testing so much more manageable. Indeed, the three blessings I count here are really the reason that testing doesn't fail more often than it does.
The Blessing of Unit Testing
I am thankful for early cycle testing thinning out the bug herd. In late cycle testing major bugs are often masked by minor bugs and too many of the latter can hamper the search for the former. Every bug that requires a bug report means lost time. There is the time spent to find the bug; time spent to reproduce and report it; time to investigate its cause and ensure it is not a duplicate; time to fix it, or to argue about whether it should be fixed; time to build the new version and push it to the test lab; time to verify the fix; time to test that the fix introduced no additional bugs. Clearly the smaller the population to begin with, the easier the task becomes. Solid unit testing is a tester's best friend.
The Blessing of Rarity
I am thankful that the vast majority of bugs that affect entire user populations are generally nuisance-class issues. These are typically bugs concerning awkward UI elements or the occasional misfiring of some feature or another where workarounds and alternatives will suffice until a minor update can be made. Serious bugs tend to have a more localized effect. True recall class bugs, serious failures that affect large populations of users, are far less common. Testers can take advantage of the fact that not all bugs are equally damaging and prioritize their effort to find bugs in the order of their seriousness. The futility of finding every bug can be replaced by an investigation based on risk.
Risk analysis is so important that we've built an internal tool to help guide testers in performing it. Code-named "Testify" this tool streamlines the process of risk analysis, at least the way we do it at Google. We're working on open-sourcing an early prototype in time for GTAC 2010 (I can hear my team cringing now ... "you promised it when?").
The Blessing of Repetition
I am thankful that user behavior is highly repetitive. There are features that enjoy heavy usage across user populations and features that are far less popular. Mobile phones are a good example of this. The phone is constantly establishing connections to networks. Certain features like making and receiving calls, texting and so forth are used more often than taking pictures or searching maps. The popularity of user applications is a matter of hard data, not guesswork. Knowing what users do most often, less often and least often means testing resources can be applied with a commensurate amount of force and that testing itself can be patterned after actual usage profiles.
Testers can gain a great deal from taking the user’s point of view and weaving usage concerns into the software testing process. Focusing on the user ensures that high impact bugs are found early and software revisions that break key user scenarios are identified quickly and not allowed to persist.
Apple may be the company in the news today, who knows who it will be tomorrow. Every company that produces software people care about has either been there or will be there. The job is simply too big for perfection to be an option. But there are key advantages we have that make the job manageable.
Put down the stones and make sure that what few blessing we testers possess are being exploited for everything they are worth. Hopefully, your company will be spared and the next time a company suffers such a bug you won't be the one making excuses. Perhaps you'll be lucky enough to be the one saying, "there but for the grace of testing go I."
I've had more than a few emails about "antenna-gate" asking me to comment and suggesting clever, stabbing rebukes to a fallen competitor. I might aim a few of those at my own team in the future, some were genuinely funny, but none of them will appear here. Instead I offer first a word of caution and second a reflection that my Mom used to intone whenever disaster occurred around her. It's called "counting your blessings."
First, a caution that those of us who live in glass houses really should keep stones at arms length. The only way anyone can rebuke Apple, without risk of waking up one morning sucking on their own foot, is if they write no software or have no users. Apple does a lot of the former and they enjoy many of the latter. Bugs like this make me sick when they are mine and nervous when they aren't. If any tester in the industry isn't taking stock right now then they either aren't producing any software or aren't in possession of any users, at least ones they wish to keep.
Second, taking stock has made me realize that I enjoy some important blessings that make the infinite task of testing so much more manageable. Indeed, the three blessings I count here are really the reason that testing doesn't fail more often than it does.
The Blessing of Unit Testing
I am thankful for early cycle testing thinning out the bug herd. In late cycle testing major bugs are often masked by minor bugs and too many of the latter can hamper the search for the former. Every bug that requires a bug report means lost time. There is the time spent to find the bug; time spent to reproduce and report it; time to investigate its cause and ensure it is not a duplicate; time to fix it, or to argue about whether it should be fixed; time to build the new version and push it to the test lab; time to verify the fix; time to test that the fix introduced no additional bugs. Clearly the smaller the population to begin with, the easier the task becomes. Solid unit testing is a tester's best friend.
The Blessing of Rarity
I am thankful that the vast majority of bugs that affect entire user populations are generally nuisance-class issues. These are typically bugs concerning awkward UI elements or the occasional misfiring of some feature or another where workarounds and alternatives will suffice until a minor update can be made. Serious bugs tend to have a more localized effect. True recall class bugs, serious failures that affect large populations of users, are far less common. Testers can take advantage of the fact that not all bugs are equally damaging and prioritize their effort to find bugs in the order of their seriousness. The futility of finding every bug can be replaced by an investigation based on risk.
Risk analysis is so important that we've built an internal tool to help guide testers in performing it. Code-named "Testify" this tool streamlines the process of risk analysis, at least the way we do it at Google. We're working on open-sourcing an early prototype in time for GTAC 2010 (I can hear my team cringing now ... "you promised it when?").
The Blessing of Repetition
I am thankful that user behavior is highly repetitive. There are features that enjoy heavy usage across user populations and features that are far less popular. Mobile phones are a good example of this. The phone is constantly establishing connections to networks. Certain features like making and receiving calls, texting and so forth are used more often than taking pictures or searching maps. The popularity of user applications is a matter of hard data, not guesswork. Knowing what users do most often, less often and least often means testing resources can be applied with a commensurate amount of force and that testing itself can be patterned after actual usage profiles.
Testers can gain a great deal from taking the user’s point of view and weaving usage concerns into the software testing process. Focusing on the user ensures that high impact bugs are found early and software revisions that break key user scenarios are identified quickly and not allowed to persist.
Apple may be the company in the news today, who knows who it will be tomorrow. Every company that produces software people care about has either been there or will be there. The job is simply too big for perfection to be an option. But there are key advantages we have that make the job manageable.
Put down the stones and make sure that what few blessing we testers possess are being exploited for everything they are worth. Hopefully, your company will be spared and the next time a company suffers such a bug you won't be the one making excuses. Perhaps you'll be lucky enough to be the one saying, "there but for the grace of testing go I."
Test Driven Integration
Submitted by noreply@blogger.com (Philip Zembrod) on Fri, 09/07/2010 - 12:58.
Testivus, Testability and Dr. Jekyll and Mr. Hyde
Submitted by noreply@blogger.com (Alberto Savoia) on Thu, 01/07/2010 - 21:49.by Alberto Savoia
Note: Apparently, there were lots of downloads of the Testivus booklet and I hit some kind of quota on my personal account. If you have problems with reaching the original link below, please try this new download link or this one.
A major topic at this year's GTAC conference is going to be testability: "We also want to highlight methodologies and tools that can be used to build testability into our products." That's great!
Testability is one of the most important, yet overlooked, attributes of code – and one that is not discussed enough. That's unfortunate, because by the time the issue of testability comes up in a project it's usually too late. As preparation and seeding for GTAC, I though it would be fun and useful to get some discussions on testability going. So here we go, feel free to chime in with your thoughts.
A few years ago, after watching one too many episodes of Kung Fu, I was inspired to write a pretentious and cryptic little booklet about testing called "The Way of Testivus" (PDF).
Testivus addresses the issue of testability in a few places, but I would like to start the discussion with this maxim:
To me, "Think of code and tests as one" is the very foundation of testability. If you don't think about testing as you design and implement your code, you are very likely to make choices that will impair testability when the time comes. This position seemed obvious and non-controversial to me at the time I wrote it, and I still stand by it. Most people seem to agree with it as well, and more than one person told me that it's their favorite and most applicable maxim from all of Testivus. There are however three groups of people who found issue with it.
Some of the people, mostly from the TDD camp, think that my choice of words leaves too much wiggle room: "Thinking about the tests is not enough, they should be writing and running those tests at the same time."
Others think that code and tests should not be thought of as one at all, but they should be treated independently – ideally as adversaries: "I don't want code and tests to be too "friendly". Production code should not be changed or compromised to make the testing easier, and tests should not trust the hooks put in the code to make it more testable." Most of the people in this camp are not big fans of unit/developer testing in the first place, but not all. One person, a believer in developer testing, told me that he gets the best results with a Dr. Jekyll and Mr. Hyde approach. He assumes two different roles and personalities based on whether he's coding or testing his own code. When coding, he's the constructive Dr. Jekyll who focuses on elegant and efficient design and algorithms – and does not worry about testability. When testing, he turns into the destructive Mr. Hyde; he tries to forget that it's his code or how he implemented it, and puts all his energy and anger into trying to break it. Sounds like it could work quite well – though I don't think I'd want this person as an office mate during the Mr. Hyde phase.
A third group, thought that the maxim was fine for unit tests, but not applicable to other types of tests that were best served by an adversarial black-box approach.
What are your thoughts? Is it enough to think about testability when designing or writing the code, or must you actually write and run some tests in parallel with the code? Does anyone agree with the position that code and tests should be designed and developed in isolation? Are there other Dr. Jekylls and Mr. Hydes out there?
Alberto
Update! GTAC 2010 Keynote Speakers
Submitted by noreply@blogger.com (Patrick Copeland) on Tue, 29/06/2010 - 16:44.We are thrilled to announce that GTAC 2010 keynotes are finalized. We are very fortunate to have three world-renowned software testing icons: Robert Binder, Dr. Jeff Offutt and Dr. James Whittaker. Robert, Jeff and James bring to GTAC a powerful combination of practical and theoretical experience to help us address key aspect of this year's theme: Test to Testability. Robert will kick-off Day 1 with a keynote on the critical role of testability. On Day 2, Jeff will share his experiences on Automatic Test Generation from source code and about a technique he invented, Bypass Testing, for black-box testing of web applications. James will close the conference with a keynote focusing on the Test(ing) challenges and how to get ahead of them. More details on the specifics of their talk coming soon!
Here are brief bios of our esteemed speakers:
Robert Binder
Robert V. Binder is a software entrepreneur and technologist with over 34 years of systems engineering experience. His 1994 analysis of software testability http://portal.acm.org/citation.cfm?id=184077 has had a continuing influence on research and practice in this area. As principal of System Verification Associates, he leads teams that deliver advanced IT assurance solutions. He was recently awarded a U.S. Patent for a unique approach to model-based testing of mobile systems. He is a member of the Editorial Board of Software Testing, Verification, and Review and internationally recognized as the author of the definitive Testing Object-Oriented Systems: Models, Patterns, and Tools. Robert holds an MS in Electrical Engineering and Computer Science from the University of Illinois at Chicago and a MBA from the University of Chicago. He is an IEEE Senior Member.
Dr. Jeff Offutt
Dr. Jeff Offutt is Professor of Software Engineering at George Mason University. He has invented numerous test strategies, published over 120 refereed research papers, and is co-author of the textbook Introduction to Software Testing. He is editor-in-chief of Wiley's journal of Software Testing, Verification and Reliability; steering committee chair for the IEEE International Conference on Software Testing, Verification, and Validation; and program chair for ICST 2009. He has consulted with numerous companies on software testing, usability, and software intellectual property issues. Offutt is on the web at http://www.cs.gmu.edu/~offutt/
Dr. James Whittaker
Dr. Whittaker is currently the Engineering Director over engineering tools and testing for Google's Seattle and Kirkland offices. He holds a PhD in computer science from the University of Tennessee and is the author or coauthor of four acclaimed textbooks. How to Break Software, How to Break Software Security (with Hugh Thompson) and How to Break Web Software (with Mike Andrews). His latest is Exploratory Software Testing: Tips, Tricks, Tours and Techniques to Guide Test Design and he's authored over fifty peer-reviewed papers on software development and computer security. He holds patents on various inventions in software testing and defensive security applications and has attracted millions in funding, sponsorship, and license agreements while a professor at Florida Tech. He has also served as a testing and security consultant for dozens of companies and spent 3 years as an architect at Microsoft.
Reminder: Call for proposals
If you would like to present at this year’s GTAC please remember to submit your proposal by the July 9th deadline. Please visit http://www.gtac.biz/call-for-proposals for details.
Sujay Sahni for the GTAC 2010 Committee
GTAC: Call for Attendance & Proposals
Submitted by noreply@blogger.com (Patrick Copeland) on Mon, 17/05/2010 - 06:59.Google Test Automation Conference (GTAC) 2010
Call for Attendance & Proposals
We are happy to announce that the application process is now open for Attendance and Proposals for the Fifth Google Test Automation Conference (GTAC), to be held in Hyderabad, India on October 28 - 29th.
As in previous years, GTAC is an invitation only conference where we enable sharing of great ideas and active participation to challenge and refine our thoughts and experiences. As such the the application process expects you to share your ideas and insights that you would bring to the conference and how these would further the discussion about this year’s theme of Test to Testability. This information will help the committee select a balanced audience of seasoned practitioners, students and academics.
Also this year, we are introducing a participant-driven format that will give the power to the attendees to select and voice their opinion on the speakers and the content! To make these changes, we are opening up proposals and attendance applications simultaneously. Once the initial set of participants are finalized, we will conduct online viewing and voting by the participants for presentations.
How to apply
For Attendance: Please visit http://www.gtac.biz/call-for-attendance
For Proposals (to present): Please visit http://www.gtac.biz/call-for-proposals
Deadline
The due date for both categories of applications is July 9th, 2010.
Registration Fees
There are no registration fees. Please check the FAQ page for more information.
Further information
General website: http://www.gtac.biz/
Call for proposals: http://www.gtac.biz/call-for-proposals
Call for attendance: http://www.gtac.biz/call-for-attendance
FAQ: http://www.gtac.biz/faq
Questions: Email us at gtac-2010@google.com
We look forward to your applications and a great GTAC!
Finally we would appreciate your help in helping us spread the word about this event.
Regards
Sujay Sahni on behalf of the GTAC 2010 Committee
Do Know Evil
Submitted by noreply@blogger.com (Christopher Semturs) on Thu, 06/05/2010 - 11:10.Web Application Exploits and Defenses
by Bruce Leban in Google Kirkland
http://google-gruyere.appspot.com/
If you want your application to be as secure as possible, you need to learn how Evil People think. And you'll want to use that knowledge to do penetration testing: attacking your own application to try to find bugs.
To help you understand how applications can be attacked and how to protect them from attack, we've created the “Web Application Exploits and Defenses” codelab. The codelab uses Gruyere, a small, cheesy, web application that is full of real world bugs.
In the codelab, you'll learn how to:
Attack a web application to find and exploit common web security vulnerabilities.
Avoid and fix these common bugs.
Gruyere is chock full of cool features, and the more features an application has the larger the attack surface. Your application probably has features just like these:
Can you match each feature to the vulnerability that it exposes and the exploit it enables?
Feature
New template language
HTML allowed in snippets
File upload capability
AJAX
Web-based admin console
-
Vulnerability
Cross Site Scripting (XSS)
Cross Site Request Forgery (XSRF)
Cross Site Script Inclusion (XSSI)
Path traversal
Client-state manipulation
-
Exploit
Information disclosure
Elevation of privilege
Denial of Service (DoS)
Spoofing
Code execution
Ha! Tricked you! Each of these features introduces multiple vulnerabilities. And each vulnerability can be exploited in multiple ways. The codelab walks you step by step through each vulnerability, with progressive hints guiding you on how to find them, how to exploit them and how to avoid them.
Here are some examples of fictitious attacks against Google applications. Do you recognize them? (answers below)
-
http://www.gmail.com/?search=in:spam+%3Cscript%3EmoveToInbox(selectAll())%3C/script%3E
http://www.blogger.com/delete-blog.g
http://www.picasa.com/../../../../../../../etc/passwd
http://www.youtube.com/admin?v=Vr0oK3gMzK&action=rickroll
http://checkout.google.com/buy?order=4815162342&total=0.01
Are you sure that your application isn't vulnerable to similar attacks!?
Check out the Toilet-Friendly Version for the answers
GTAC!
Submitted by noreply@blogger.com (James Whittaker) on Fri, 30/04/2010 - 16:20.By James A. Whittaker
Yes I know, I've been quiet. Seriously heads down shipping products and developing what I think are some pretty cool new testing ideas and tools. Perhaps GTAC will be the chance for you to judge that for yourselves. Perhaps it will be worth the wait.
I hope I am invited to speak at GTAC. (Is this an appropriate forum for such lobbying? Should I open it up to a vote on whether I should or should not be there? I am happy not going as India is a long trip, but this is GTAC we are talking about!) There are a number of things that will be ready to either debut or, if I am lucky, open source by then.
Would you like to see a Chrome extension to display bug, test case, coverage and other information as an overlay on top of your web app UI? Imagine being able to see bugs at their exact location on the UI, report bugs by simply right-clicking the errant part of the web page, see footprints where test cases (automated and manual and across multiple testers) have been and lots more useful information. Are you tired of querying databases to see these things and just want your test data as a cellophane wrapper around your web app UI? If you were at STAR this week, you got a preview. But that presentation is already out of date.
Would you like to be able to write automated test cases that can control your web app, your browser and the operating system they are running on? Well if that stack contains Chrome and Chrome OS, you can do it with a new web test framework we are developing. Would you like to do all of this with Java Script? Sound like magic? Well I think the Web Test Framework is appropriately named: WTF.
Would you like to see a record and playback facility that records directly to Java Script and is actually built-in to your browser? A R/P tool that handles AJAX and won't get confused by self-updating web pages? That stores recordings directly into a test case management system that is accessible to the world?
Would you like to hear about the extensive library of testing tours we have developed and how our manual testing strategy across the web-app/Chrome/Chrome-OS stack is shaping up?
These are some of the things that have kept me from this blog. Forgive me. I will report on them here and, with a little pressure directed toward Sujay The Decider perhaps demo them at GTAC.
Yes I know, I've been quiet. Seriously heads down shipping products and developing what I think are some pretty cool new testing ideas and tools. Perhaps GTAC will be the chance for you to judge that for yourselves. Perhaps it will be worth the wait.
I hope I am invited to speak at GTAC. (Is this an appropriate forum for such lobbying? Should I open it up to a vote on whether I should or should not be there? I am happy not going as India is a long trip, but this is GTAC we are talking about!) There are a number of things that will be ready to either debut or, if I am lucky, open source by then.
Would you like to see a Chrome extension to display bug, test case, coverage and other information as an overlay on top of your web app UI? Imagine being able to see bugs at their exact location on the UI, report bugs by simply right-clicking the errant part of the web page, see footprints where test cases (automated and manual and across multiple testers) have been and lots more useful information. Are you tired of querying databases to see these things and just want your test data as a cellophane wrapper around your web app UI? If you were at STAR this week, you got a preview. But that presentation is already out of date.
Would you like to be able to write automated test cases that can control your web app, your browser and the operating system they are running on? Well if that stack contains Chrome and Chrome OS, you can do it with a new web test framework we are developing. Would you like to do all of this with Java Script? Sound like magic? Well I think the Web Test Framework is appropriately named: WTF.
Would you like to see a record and playback facility that records directly to Java Script and is actually built-in to your browser? A R/P tool that handles AJAX and won't get confused by self-updating web pages? That stores recordings directly into a test case management system that is accessible to the world?
Would you like to hear about the extensive library of testing tours we have developed and how our manual testing strategy across the web-app/Chrome/Chrome-OS stack is shaping up?
These are some of the things that have kept me from this blog. Forgive me. I will report on them here and, with a little pressure directed toward Sujay The Decider perhaps demo them at GTAC.