Archeological Black Swans: tales of observation, deduction and lost treasure

I had dinner with Michael Bolton and James Bach during STARwest, which was a great feast of ideas and insights in addition to the fine food. One of the interesting new ideas that I hadn’t yet heard of was Black Swan Theory , mentioned by Michael Bolton. As an Australian, I could see nothing unusual about the concept of a Black Swan, but in Europe before the discovery of Australia, a black swan was seen as an impossibility since none had ever been seen. The Black Swan theory relates to an unexpected, radical event that alters your perception of reality. As testers. we are often told the European view that black swans don’t exist (this code is fully tested, two weeks of testing will be more than enough, one retest all you need, works on my machine), but we know that our world is full of black swans we call bugs. There is also the other myth that “all tests must be scripted” disproved by the black swan of exploratory testing.

There are two key skills in test execution for finding bugs. There is straight observation where using the software creates a bug that needs to be observed. I believe most computer users can find simple types of these bugs, though skilled testers may find them faster as they are more attuned to the types of behaviours that cause bugs. One of the most difficult things is bugs that are hard to find, or hard to reproduce. (If you are using screen capture software, you will have the screenshots/video once it happens; only the developers have to reproduce it then!) There is the other issue that as feeble humans it is very easy to miss all but the most obvious bugs. (again screenshots/video help out here, allowing us to watch a replay of the test execution to see if there are things we missed)

An interesting Swedish archeological black swan was announced this year, a treasure trove of arabian coins hidden 1100 years ago but found near the Swedish international airport. Despite the tomb being apparently obvious, it took a planned development of the site to trigger the investigation. There were 472 silver coins found with the youngest coin minted in AD 840, and other coins being hundreds of years older than that. The tomb that they were hidden in had been build 1000 years before, so turned out to be an excellent hiding place. The coins were used as treasure rather than as coinage, and most had been cut into pieces. The treasure trove provides evidence that trading was occuring much earlier than previously believed.

The more advanced bug finding skill is deduction where thinking about how the software works or limitations in the way it may have been desgined or coded can lead to more devious and often more spectacular bugs.

Archeology can be similar to (exploratory) testing, looking for concentrations of objects (similar to bug clusters) and trying to get as much information about a site (or software) as possible in a limited time. The notion of coverage is important, always weighing up is the current area of interest worth continuing with or should another area be investigated. One of the greatest archeological black swan discoveries has been made in Turkey, an 11,000 year old temple, twice as old as Stonehenge. The German who discovered it, Klaus Schmidt, had revisited an area that other archeologists had previously dismissed as a medieval graveyard. Schmidt recognized the artificial shape of the hill meant that it had been man made. Digging into the soil revealed rings of large T-shaped stones, built before even pottery was discovered. The Smithsonian magazine states, “The find upends the conventional view of the rise of civilization”. (read the article , then see more pictures here )

Michael Bolton seems to believe that bug clusters are an unprovable concept awaiting a black swan, but as people who have seen my Google techtalk will realize, I think they are much closer to a silver bullet! All observations support bug clustering, and even any black swan (if it existed) would not affect the evidence that we already have.

When Michael mentioned “Black Swan” to me, the first thing that popped to mind was a Spanish treasure ship sunk by the British that was found with 17 tons of Peruvian silver off Portugal, and nicknamed the Black Swan by the American company that found it (the legal battle over who the treasure belongs to could go for decades ). Unless a black swan event occurs, I never expect to see that much silver in my possession, but that doesn’t stop me treasuring every bug my team finds, or better still prevents!