For the second assignment of a class called 'Software Practice' at the University of Utah in Fall 1998, the students were given a specification, a header file, and an executable.

"No source code! The students did not feel they had control of the assignment without seeing the source code. In industry, objective, independent test groups that never see source code often do black box testing. So, the experience prepared the students for the future.

Shortly after the students received their assignment and code, they needed to produce a test plan to validate that the code actually performed according to the specification. Their test plan outlined a test case and the expected results when the test case was run. At assignment completion, they needed to hand in a revised test plan with all test cases, the expected results, and the actual results. They also needed to hand in a document enumerating all defects they detected in the code and fully-documented, nicely-structured test code.

An important learning experience of this assignment was due to the intentionally poorly-written specifications. The students asked questions about what the specs meant, what happens in error situations, what happens when a certain set of arguments is given, etc. By the time we evaluated the assignment, they had learned the benefit of well-written specifications. Having been treated to poor specifications, they realized that they would not want to force bad specifications on any other programmer."

Kessler, R. R. & Williams, L. A. (1999). If this is what it's really like, maybe I better major in English: Integrating realism into a sophomore software engineering course. 29th ASEE/IEEE Frontiers in Education Conference. San Juan, Puerto Rico, November 10-13, 1999.

On Tuesday, June 28th, Colorado Springs Agile welcomes Ron Jeffries, experienced XP author, trainer, coach, and practitioner. Ron is author of Extreme Programming Adventures in C#, the senior author of Extreme Programming Installed, and was the on-site XP coach for the original Extreme Programming project. He'll be presenting a talk entitled "Why Software Development Is Easy," unless he can't figure out by the meeting why it is, in which case the adjective may be changed.

DETAILS
The meeting will be held at Colorado Springs Fire Station #14 (capacity: 68), 1875 Dublin Blvd. The fire station is located behind The Oak Place at Dublin and Academy. The meeting is free and open to software professionals and students; please share this announcement with your colleagues. For those of you closer to Denver, Ron will also present this topic at Agile Denver the night before.

6:00 - 6:30 PM Networking and refreshments
6:30 - 6:40 PM Extreme Tuesdays review
6:40 - 7:50 PM Ron Jeffries
7:50 - 8:15 PM Door prize giveaway, open discussion

Companies told researchers that recent grads lacked group work skills. The researchers tried introducing project courses and group work, but that didn't solve the problem. Students showed a bias against collaboration which was not overcome with group work.

Over a three year research project with 130 student interviews by ehtnographic observers, they found 3 techniques which helped foster collaboration:

• The conversational classroom
  Professor facilitation of discussion rather than lectures. The professor yields control over the information to the students, showing the students that they must participate to explore the issues.

• Group decision-making
  Each student contributes a specification. Specifications are distributed to the entire class. Then each student generates a design based on a specification (either their own or another person's).

  Next class, the group selects the criteria for choosing the best design. Over the next couple days, the students assign weights to the criteria and evaluate each design based on the criteria. Finally, as a class, they chose the best design to use for their assignment.

  Once the design is chosen, changes were only permitted if everyone agreed that the change would increase previously accepted criteria values.

• Assignment devaluation
  If the individual work is highly rewarded, it will be valued more than group work. So, individual assignments were not given as much weight on the students' grades.

(Side note: The conversational classroom sounds similar to Problem-Based Learning.)

Effectively, if you want to teach collaboration and group work, you have to integrate it into the very fiber of the class. You have to make everything collaborative - the classtime learning, the decision making concerning assignments, and the student grades. The results of the study showed that after strong resistance, the techniques led to better performance and increased student satisfaction.

Waite, W. M., Jackson, M. H., Diwan, A., & Leonardi, P. M. (2004). Student culture vs group work in computer science. Proceedings of the 35th SIGCSE Technical Symposium for Computer Science Education (pp. 12-16). New York: Association for Computing Machinery Press.

For the second assignment of a class called 'Software Practice' at the University of Utah in Fall 1998, the students were given a specification, a header file, and an executable.

"No source code! The students did not feel they had control of the assignment without seeing the source code. In industry, objective, independent test groups that never see source code often do black box testing. So, the experience prepared the students for the future.

Shortly after the students received their assignment and code, they needed to produce a test plan to validate that the code actually performed according to the specification. Their test plan outlined a test case and the expected results when the test case was run. At assignment completion, they needed to hand in a revised test plan with all test cases, the expected results, and the actual results. They also needed to hand in a document enumerating all defects they detected in the code and fully-documented, nicely-structured test code.

An important learning experience of this assignment was due to the intentionally poorly-written specifications. The students asked questions about what the specs meant, what happens in error situations, what happens when a certain set of arguments is given, etc. By the time we evaluated the assignment, they had learned the benefit of well-written specifications. Having been treated to poor specifications, they realized that they would not want to force bad specifications on any other programmer."

Kessler, R. R. & Williams, L. A. (1999). If this is what it's really like, maybe I better major in English: Integrating realism into a sophomore software engineering course. 29th ASEE/IEEE Frontiers in Education Conference. San Juan, Puerto Rico, November 10-13, 1999.

An experiment compared the work of 5 individuals to 5 groups of pair-programmers. Using a 2-tailed t-test, researchers found the teams significantly outperformed the individuals, had more confidence in their solution, and finished 40% faster.

Nosek, J. T. (1998). The case for collaborative programming. Communications of the ACM, 41(3), 105-108.

An article from Game Developer magazine about preventing cheating in online games can offer insight about security testing in general. Here are the rules outlined in the article:

• If you build it they will come - to hack and cheat.
• Hacking attempts increase with the success of your game.
• Cheaters actively try to keep developers from learning their cheats.
• Your game, along with everything on the cheater's computer, is not secure. The files are not secure. Memory is not secure. Services and drivers are not secure.
• Obscurity is not security.
• Any communication over an open line is vulnerable to interception, analysis, and modification.
• There is no such thing as a harmless cheat or exploit. Cheaters are incredibly inventive at figuring out how to get the most out of any loophole or exploit.
• Trust in the server is everything in a client-server game.
• Honest players would love for a game to tip them off to possible cheating. Cheaters want the opposite.

Now, read the list again, but replace the word 'game' with 'application'.

Pritchard, M. (2000) How to hurt the hackers: The scoop on internet chearing and how you can combat it. Game Developer Magazine, July 24, 2000. Retrieved from here.

In psychology, for many years academics operated under the assumption that people had personality traits. Regardless of whether they are biologically inherent (nature) or culturually developed (nurture), psychologists generally agreed that the traits formed an individual's personality. Some researchers tried to show that people behave differently in different contexts, by administering questionaires about hypothetical examples. The results were mixed. Then, Pervin (1976) asked four people to list some real experiences from their life and describe how they acted and felt. Even with this small sample size, the researcher was able to demonstrate that people act differently in different situations. The difference in approaches was that he did not try to fit the participants' answers into a predetermined list of questions; he let them write about whatever they wanted.

Quantitative research begins with the assumption that the researcher knows exactly what to measure. Qualitative research begins with the idea that the previous quantitative research is measuring the wrong constructs - that the questions on the surveys are not getting to the meat of the issue. The qualitative researcher then takes an adaptive approach to investigate what the real questions should be.

The same investigative approaches can be applied to researching the behavior of software. If you do not agree with the way previous research has been performed (for example, you think a different approach or looking in a different area will find new bugs) then you can investigate the phenomenon with exploratory techniques. The survey (scripted test plan) cannot be written ahead of time, because you don't know what the important questions (tests) are.

But the lack of a scripted test plan doesn't mean you go in blind, 'randomly' testing. You still intelligently probe the system with knowledge of software and how software fails (Whittaker, 2004), the domain and scenarios (Kaner & Bach, 2004, slide 78 & 82), and the team that created the software under test (Kaner & Bach, 2004, slide 63 & 64).

Sources
Kaner, C. & Bach, J. (2004). The nature of exploratory testing. Exploratory & Risk-Based Testing. Retrieved on May 26, 2005 from here. (PDF)

Pervin, L. A. (1976). A free-response description approach to the analysis of person-situation interaction. Journal of Personality and Social Psychology, 34(3), 465-474.

Whittaker, J. A. (2004). Software testing as an art, a craft and a discipline. DoD Software Tech News, 7(2), 9-11.

I was reading the slides from the STAR East 2005 lightning talks. Obviously I am missing a bunch of context just reading the slides, but I was interested in one set by David Gilbert from Sirius Software Quality Associates about Exploratory Testing. His presentation seems to be specifically about demonstrating the value of exploratory testing to those who fund the testing. The slides state that professional presentation of the results of exploratory testing can be labor intensive, requiring screen grabbers, lots of cut and paste, and reams of paper for printing the results. He recommends keylogging, screenshots, and video (Gilbert, 2005).

Reading this reminded me again of the similarities between exploratory testing and qualitative research in academia. Qualitative research results in mounds of thick, rich descriptions of few case studies. Rather than asking 100 people about 4 items on a survey, you might ask 4 people 100 questions. It's a different kind of research based on a different epistemological starting point.

Qualitative research and exploratory testing are also both plagued by the same type of discrimination - people think that quantitative statistics are inherently more accurate than qualitative research. Similiary, in software testing, people think that structured (or even automated) testing is more accurate than exploratory testing. The truth is that each offers different benefits: scripted testing offers more accountability, while exploratory testing offers more adaptability (Kaner & Bach, 2004, slide 13).

Sources
Gilbert, D. (2005). Communicating the value of exploratory testing. STAR East 2005 lightining talks. STAR East. Orlando, Florida, May 16-21, 2005. Retrieved on May 24, 2005 from here, slides 27-33.

Kaner, C. & Bach, J. (2004). The nature of exploratory testing. Exploratory & Risk-Based Testing. Retrieved on May 26, 2005 from here. (PDF)