testingReflections.com - reliability testing

Stability test tool: batch file

Mon, 28 Apr 2008 02:31:17 -0500

This morning I realized that last Friday I did something that seems very simple to me, but may not be for young people not used to DOC (or at least windows95). I run a bat file test.bat containing following lines:

:loop
copy test\*.* import\*.*
sleep 600
goto loop

The same documents get copied into import directory each 10 minutes (600 seconds=10minutes). I was testing server who is able to pick up files incoming into this folder. It is quite typical to have such services to automatically process faxed documents and email attachments. But you could replace copy command with your process execution call or anything.

Reliability is not only about overload: RANDOMize your system state

Thu, 24 Apr 2008 11:15:18 -0500

I never used mnemonics. Never say never... Here is my own mnemonic. Double RANDOM stands for ReadRights AalternativeAccess, NoNetwork, DesynchronyzedData, OutOf(whatever resource), MinMax(installation). Depending on context (e.g. reliability requirements, etc.) I spend more or less time trying to emulate system states unexpected by developer and thus unhandled by code.

Watch this next?: the $1 million data mining quality challenge

Wed, 19 Mar 2008 08:34:38 -0500

Quality is often measured in terms of accuracy (or Accurateness, according to this definition ) . For a shopping site, the closer you can predict what a customer likes, the more you can sell and the more inclined your customer will be to buy more and to stay loyal to you as a supplier. As any gambler knows, prediction is not a science but historical results can provide a best guess. Can you put a price on this? Who knows. One company has offered a prize to make it better though, one million dollars.

The P28 virtual fence: Borderline success or virtual vaporware?

Wed, 05 Mar 2008 04:06:01 -0600

I’ve already noted some of the issues around the virtual fence here. In May 2006, George W. Bush called it “the most technologically advanced border-security initiative in American history”. In a flash of originality, the 28 mile virtual fence in Arizona was called Project 28, or P28 in Chertoff-speak. Micheal Chertoff has now accepted the fence system, on the basis that “all of the defects” in the prototype project were either “cured” or “immaterial.” He also said “In some form or fashion, technology is going to be virtually every place on the border, but it’s not necessarily going to be in the configuration of P28,” The chair of the House Committee on Homeland Security, Representative Bennie Thompson, sees it differently, “The poorly structured contract that prevented the line Border Patrol agents from pointing out obvious flaws and caused an overreliance on contractors has resulted in a system that has been described as providing ‘marginal’ functionality at best,” I wonder if he was tempted to say the Border fence had borderline functionality. [grin]

The technology stack on top of each tower looks impressive . The project is anything but impressive , with many of the issues typical in the IT industry: badly underestimated effort, not involving users, and not enough testing, While there was evidently a push for reuse of existing systems and components, it caused many difficulties: trying to base the system around a law enforcement dispatch system (!) that couldn't scale, trying to use off-the-shelf components that weren't designed to integrate, and a lack of standards for the sensors. The builder of the system, Boeing, has only taken 3/4s of the $20 million fee, granted a $2 million credit, and apparently paid $40 million to get this far on a project that will now finish in 2011, not the end of this year (which was already a year and a half past the original schedule). That will only be phase one!

Qualitative and Quantitative

Fri, 16 Nov 2007 14:36:07 -0600

A few thoughts inspired by a question about qualitative and quantitative information reporting performance test results in Yahoo's LoadRunner group.

My understanding of qualitative and quantitative in the context of reporting performance test results is:

Quantitative is what we directly measure or straightforwardly calculate from direct measurements (like response times or resource utilization)

WOPR8: Call for Presentations (CFP)

Tue, 19 Dec 2006 22:59:23 -0600

Theme: Critical Incidents in Effective Performance Testing

Conference Location and Time
MITRE Bedford MA, USA

April 12 – 14, 2008

Major Questions of Interest
Questions we are interested in exploring at WOPR8 include:

What critical Incidents lead to effective performance testing? By sharing and attempting to understanding the critical incidents leading to effective performance testing we will explore how our craft can make an important contribution to all IT and Development projects.