testingReflections.com - heuristics

A refactored assertion-based test oracle........for US presidents???

Thu, 06 Nov 2008 23:29:53 -0600

An Assertion-based (also known as programmer tests in Test Driven Development) Test Oracle (an information source of what makes a test correct) was developed 27 years ago, well before TDD was thought of but probably around the time test oracles were thought of. That was also long before refactoring was thought of, simplifying the logic to its simplest form.

It was designed by US history professor Allan Lichtman and mathematician Volodia Keilis-Borok. It is 13 assertions (that is refactored all right) that can each be true or false, and has picked every US federal election result since 1984. If more than half of the assertions are true, the governing party will win, otherwise they will lose. The majority of the assertions first went false in early 2006, at which point Lichtman said “Long before the nomination contest unfolded, the Democrats could take a name out of a phone book and still win.” I guess they must have used a Chicago phone book, or maybe even a Brazilian one! (where there are now multiple Barack Obamas) This year the test oracle has evaluated False, but if we look at it vice-versa, True or even Yes we can! [grin]

If the test oracle is correct, newspapers will print this mock headline on Wednesday the 5th of November 2008. (There’s another agile word, mock!)

June Issue of AST Update Now Available

Fri, 22 Aug 2008 12:55:01 -0500

June 2008 Issue of the Association for Software Testing Magazine and Newsletter Now Available

Latest Column -- Inspired by taking AST's Bug Advocacy Class

Wed, 06 Aug 2008 01:18:53 -0500

Software testing is improved by good bug reporting

I recently completed (successfully, I might add) the second of the Association for Software Testing's all online, free to members Black Box Software Testing course. Each of these courses is four weeks in length. I've been involved with this program since years before it became a program, and I am an instructor for the first course in the series, called Foundations. For this course, called Bug Advocacy, I was a student.

Grey-box techniques for excluding tests

Mon, 30 Jun 2008 07:33:18 -0500

Don’t test the reused code tested previously – you will have more time to test new code. This heuristic for a black-box tester is basically all the idea of this blog – one I was creating for a week … I started this as blog about techniques, but wanted to be clear about grey box. A research on the term ended quite a long story below. If you don’t care about it – scroll down to the last chapter describing the techniques.

Linguistic heuristics

Sun, 22 Jun 2008 06:18:35 -0500

Searching. I’m currently testing a product that has a search feature. I’ve tested search functionality before but not a multilingual search engine that utilizes two different search engines based on the language selected. Nor have I previously worked with a search engine that uses stemming and stop words.

I almost didn’t want to write about this – figured I would wait until I resolved my challenges before writing about it – but it occurred to me why? It’s not as though in software testing I haven’t learned that first understanding the complexities of a problem is an essential starting point.

Testing Lessons From Civil Engineering

Sat, 21 Jun 2008 01:13:14 -0500

Below is the paper I submitted as a prologue to an experience report, discussion, and (hopefully) additional research that I'm presenting for the first time during:

Reliability is not only about overload: RANDOMize your system state

Thu, 24 Apr 2008 11:15:18 -0500

I never used mnemonics. Never say never... Here is my own mnemonic. Double RANDOM stands for ReadRights AalternativeAccess, NoNetwork, DesynchronyzedData, OutOf(whatever resource), MinMax(installation). Depending on context (e.g. reliability requirements, etc.) I spend more or less time trying to emulate system states unexpected by developer and thus unhandled by code.

ET thoughts: The Seeker (CKA) heuristic

Sun, 20 Apr 2008 02:18:28 -0500

Some people are more into mnemonics than others. I can recall walking along with James Lyndsay one day. This in itself was unusual because we are normally on opposite sides of the earth. We were discussing how there are many great mnemonics for test ideas, but neither of us was able to recall the items, just the mnemonics! I think this strongly influences our exploratory testing approaches. Alan Richardson has blogged about this here . Because it is general, It doesn’t really need a mnemonic, but I will give it an acronym. Seeker, CKA, Challenge Key Assumptions. While any tester can do this, the experienced tester will be more skilled at identifying key assumptions, and be more effective at using it. If you have less experience, or can remember mnemonics (!), maybe try the standard ones! You’ll find some links by searching for “test ideas” at my testingspot.net site

Training for Performance Testers in Brighton UK, May 08.

Sun, 06 Apr 2008 22:14:39 -0500

Scott Barber has teamed up with Rosie Sherry of DrivenQA to bring PTSS Training to Brighton & Hove, UK.

Sign-up now for Performance Testing for Managers or Performance Testing for Software Systems; A Heuristic Approach.

Pushing boundaries at airports

Fri, 18 Jan 2008 06:19:13 -0600

Not all bugs are created equally. One of the most common bugs is all about equality, too much or not enough. It is caused by code being “off by one”, where a boundary is meant to be at a particular point but is one value too high or too low, for example >=5 (5 or greater) instead of >5 (greater than 5).

Two incidents at airports had me inadvertently testing for boundary bugs. I had dropped off someone at the airport, waited for their delayed boarding and returned to the car park. Parking was $10 for up to an hour, then $18 dollars for up to 2 hours. I paid my ticket at a machine at 61 minutes, and paid $18! Damn. That boundary was spot on.

That wasn’t as painful as another time, getting into the checkin queue a few minutes before the half hour baggage cutoff. I waited a minute or so, and the attendant started to check me in then apologized that the system was preventing her completing it because the half hour boundary had just been crossed. I ended up bumped to the next flight leaving in another 2 hours. Even worse, she explained that if I had checked in at a kiosk instead of queuing, she could have checked my bags in and got me on that flight. At another time I might have been interested in my inclusive interpretations of their exclusive boundary (queue before cutoff versus be at counter before cutoff vs. be checked in manually by cutoff vs. kiosk checkin). That was beyond the bounds of common decency, but the program overruled any staff sympathy, though there was empathy. I just wandered off into a fog of unbounded apathy, which had no equal. I found no bugs but both situations certainly bugged me! [grin]

Read a great article about boundary testing here

Kipling, context & the quandary of the curious quotes

Sun, 09 Nov 2008 04:35:43 -0600

Back in 1905, Rudyard Kipling wrote

I keep six honest serving-men (They taught me all I knew);

Their names are What and Why and When, And How and Where and Who.

Also known as the Reporter’s Creed (for getting a complete story), the 5 Ws and an H or (bizarrely) the six Ws, these words are a simple way to discover context

Multi-edit heuristic

Wed, 07 Nov 2007 01:32:31 -0600

When you test client-server application does not forget to test cases when two users work with the same data simultaneously: one update business item while other deletes it, or one add a new child to a parent moved by another person, or both does rename to the same item...

Model Workloads for Performance Testing: FIBLOTS

Sat, 25 Aug 2007 17:44:04 -0500

This is the third installment of a currently unknown number of posts about heuristics and mnemonics I find valuable when teaching and conducting performance testing.

Other posts about performance testing heuristics and mnemonics are:


For years, I have championed the use of production logs to create workload models for performance testing. During the same period, I've been researching and experimenting with methods to quickly create "good enough" workload models without empirical data that increase the value of the performance tests. I recently realized that these two ideas are actually complimentary, not exclusionary, and that with or without empirical usage data from production logs, I do the same thing, I:

FIBLOTS.

Classify Performance Tests: IVECTRAS

Sat, 25 Aug 2007 17:45:02 -0500

This is the second installment of a currently unknown number of posts about heuristics and mnemonics I find valuable when teaching and conducting performance testing.

Other posts about performance testing heuristics and mnemonics are:


I have struggled for over 7 years now with first figuring out and then trying to explain all the different "types" of performance tests. You know the ones:


Well, I finally have an alternative.

IVECTRAS

Heuristics for Test Question Generation

Fri, 13 Jul 2007 19:37:51 -0500

[textile]
Today at the Workshop on Open Certification we came up with the following (non-ordered) heuristics that might be useful in test question creation:

1) Plausible buzzwords
2) True but irrelevant
3) Write but for the wrong reason
4) Some fool said it
5) My boss will believe it
6) Two conclusions from the same reason
7) Incomplete reason