Perl Unit Testing... Test::More (or less) like XUnit

I've spent some time working with a client that does extensive development in Perl. They also use Test::More as their test framework. It wasn't my role to evaluate the use of Test::More, not to mention that there was already a significant investment in using it. So, I dived straight in.

Now, having never written any Perl before and being used to XUnit frameworks (specifically NUnit and JUnit), I was used to a very different style of writing tests.

Test::More, unlike many XUnit Frameworks, is a self-sufficient script that requires no test-runner. I've found that many of unit tests written using Test::More were long sequences of method calls and assertions separated by comments. For example...

	use Test::More 'no_plan';
	use Test::Exception;
	
	my $foo = Test::SillyExample::Foo->new();
	
	# foo starts life with low aspirations
	is( $foo->bar, LOW, 'foo has low aspirations' );
	is( $foo->is_high_achiever , FALSE , 'should be low achiever' );
	
	# foo raises the bar
	$foo->set_bar( HIGH );
	
	is( $foo->bar , HIGH , 'Should now have high aspirations' );
	ok( $foo->is_high_achiever ,  'Should be high achiever' );
	
	# foo can't be demoralised
	throws_ok ( sub { 
		$foo->set_bar( LOW ); 
	} , 
	qr/Should not let others lower the bar/ ,
	'Should not allow lowering of bar');
	
	ok( $foo->is_high_achiever , 'Should still be a high achiever' );

notes:

• assume that I've created constants for the capitalised values
• I've made the example intentionally over-simplistic and abstract so don't get hung up on the assertions, whether I'm 'essentially' testing setters and getters or the specifics of the assertions!

Now, assume that I've built the test and the resulting application code incrementally, applying TDD, and got to the point it all works. The TAP style output for the test looks like this:

  ok 1 - foo has low aspirations
  ok 2 - should be a low acheiver
  ok 3 - Should now have high aspirations
  ok 4 - Should be high achiever
  ok 5 - Should not allow lowering of bar
  ok 6 - Should still be a high achiever

Now, imagine a test for a package with many more behaviours that are more complex. The test-code becomes very difficult to read as it gets bigger. The test-log output from Test::More becomes even more difficult to read as it gets longer (yes, prove makes this tidier by showing you only failures, but with the -v option still shows you the 'ok' messages).

Easier on the eye

The first thing I wanted to solve was readability of the test-code. I found myself using blocks with labels.

	use Test::More 'no_plan';
	use Test::Exception;
	
	my $foo = Test::SillyExample::Foo->new();
	
	TEST: { # foo starts life with low aspirations
		is( $foo->bar, LOW, 'foo has low aspirations' );
		is( $foo->is_high_achiever , FALSE '' );
	}
	
	TEST: { # foo raises the bar
		$foo->set_bar( HIGH );
	
		is( $foo->bar , HIGH , 'Should now have high aspirations' );
		ok( $foo->is_high_achiever ,  'Should be high achiever' );
	}
	
	TEST: { # foo can't be demoralised
		throws_ok ( sub { 
			$foo->set_bar( LOW ); 
		} , 
		qr/Should not let others lower the bar/,
		'Should not allow lowering of bar');
	
		ok( $foo->is_high_achiever , 'Should still be a high achiever' );
	}

Notice that each test is dependent on the resulting state of a previously executed test! I'll come back to that.

This worked well. When I used variables inside my blocks, it forced me to think carefully about the relevance of the variable. I'd have to ask myself Is it relevant to the test in the block or beyond it?

This didn't solve the problem of the log, however. So, I added a method to my test. This made the test-log much more readable, but the way I used it also retained the readability of the tests:

	use Test::More 'no_plan';
	use Test::Exception;
	
	use constant LOG_COMMENT_CHARACTER => '#';
	my $foo = Test::SillyExample::Foo->new();
	
	sub test {
		y $description = shift;
		print "\n" . LOG_COMMENT_CHARACTER ."-----" . $description . "\n";
	}
	
	test ( 'foo starts life with low aspirations' );
	{
		is( $foo->bar, LOW, 'foo has low aspirations' );
		is( $foo->is_high_achiever , FALSE '' );
	}
	
	test( 'foo raises the bar' );
	{
		$foo->set_bar( HIGH );
	
		is( $foo->bar , HIGH , 'Should now have high aspirations' );
		ok( $foo->is_high_achiever ,  'Should be high achiever' );
	}
	
	test ( 'foo can't be demoralised' );
	{
		throws_ok ( sub { 
			$foo->set_bar( LOW ); 
		} , 
		qr/Should not let others lower the bar/ ,
		'Should not allow lowering of bar');
	
		ok( $foo->is_high_achiever , 'Should still be a high achiever' );
	}

The blocks aren't labelled or commented any more because the test-code is now expressive. The logs now group the behaviours too.

	#-------- test foo starts life with low aspirations
	ok 1 - foo has low aspirations
	ok 2 - should be a low acheiver
	
	#-------- test foo raises the bar
	ok 3 - Should now have high aspirations
	ok 4 - Should be high achiever
	
	#-------- test foo can't be demoralised
	ok 5 - Should not allow lowering of bar
	ok 6 - Should still be a high achiever

Where's my setup?

I felt, however, that the test wasn't as expressive as I'd like, partly because each test was entirely dependent on the state of $foo established in a previous test.

One of the first things I missed when having to work with Test::More was a @setup()@ method and @teardown()@ method.

I decided that I preferred to reset $foo at the start of each test block. The outcome was that each test was expressive. Look at the last test. It is now much clearer that once $foo sets the bar high, it is not possible to set the bar low again...

	use Test::More 'no_plan';
	use Test::Exception;
	
	use constant LOG_COMMENT_CHARACTER => '#';
	my $foo;
	
	sub test {
		my $description = shift;
		print "\n" . LOG_COMMENT_CHARACTER ."-----" . $description . "\n";
		set_up();
	}
	
	sub set_up {
		$foo = Test::SillyExample::Foo->new();
	} 
	
	test ( 'foo starts life with low aspirations' );
	{
		is( $foo->bar, LOW, 'foo has low aspirations' );
		is( $foo->is_high_achiever , FALSE '' );
	}
	
	test( 'foo raises the bar' );
	{
		$foo->set_bar( HIGH );
	
		is( $foo->bar , HIGH , 'Should now have high aspirations' );
		ok( $foo->is_high_achiever ,  'Should be high achiever' );
	}
	
	test ( 'foo can't be demoralised' );
	{
		$foo->set_bar( HIGH );
		
		throws_ok ( sub { 
			$foo->set_bar( LOW ); 
		} , 
		qr/Should not let others lower the bar/,
		'Should not allow lowering of bar');
	
		ok( $foo->is_high_achiever , 'Should still be High Acheiver');
	}

One of the developers, Tim Brown, didn't like the layout of the code because the only association between the block and the test description was by convention. He was keen on a closer association... He wanted to write the tests more like this...

	test ( 'foo starts life with low aspirations' , sub {
		is( $foo->bar, LOW, 'foo has low aspirations' );
		is( $foo->is_high_achiever , FALSE '' );
	});

Tear it down why don't you?

Willem van den Ende was visiting the same organisation to run a TDD in Perl workshop and took time to meet with me to see what I'd been up to. He took a liking to the layout and wanted to use it in the workshop. He had a play with the idea and soon realised that although the latest incarnation supported set_up(), it wouldn't support tear_down. I hadn't needed a tear_down yet so hadn't even stopped to think about it...

Tim's layout did support tear-down... The test method now had to receive a sub routine:

	use Test::More 'no_plan';
	use Test::Exception;
	
	use constant LOG_COMMENT_CHARACTER => '#';
	my $foo;
	
	sub test {
		my $description = shift;

		my $block = shift;
		
		print "\n" . LOG_COMMENT_CHARACTER ."-----" . $description . "\n";
		set_up();
		&$block;
		tear_down();
	}
	
	sub set_up {
		$foo = Test::SillyExample::Foo->new();
	} 
	
	sub tear_down {
		#tear down code goes here
	}
	
	test ( 'foo starts life with low aspirations' , sub {
		is( $foo->bar, LOW, 'foo has low aspirations' );
		is( $foo->is_high_achiever , FALSE '' );
	});
	
	test( 'foo raises the bar' , sub {
		$foo->set_bar( HIGH );
	
		is( $foo->bar , HIGH , 'Should now have high aspirations' );
		ok( $foo->is_high_achiever ,  'Should be high achiever' );
	});
	
	test ( 'foo can't be demoralised' , sub {
		$foo->set_bar( HIGH );
		
		throws_ok ( sub { 
			$foo->set_bar( LOW ); 
		} , 
		qr/Should not let others lower the bar/ ,
		'Should not allow lowering of bar');
		
		ok( $foo->is_high_achiever , 'Should still be a high achiever' );
	});

So, now I want to use this again

Finally, I wanted to use this approach in other tests... The test code started to look more like XUnit... I extended Test::More and ended up with a package called Test::More::LikeXUnit

I wanted to characterize it's behaviour, since it had evolved inside one test...

LikeXUnit.t (the test) grew incrementally (as did the code)... and ended up looking like this...

	use strict;
	use lib '..';
	use Test::More::LikeXUnit 'no_plan';
	
	use constant TRUE 
		=> 1;
	use constant FALSE 
		=> 0;
	
	my $is_setup = FALSE;
	
	sub set_up {
		$is_setup = TRUE;
	}
	
	sub tear_down {
		$is_setup = FALSE;
	}
	
	test ('setup test', sub {
		ok( $is_setup, 'should have been setup' );
	});
	
	is( $is_setup, FALSE , 'Should have been torn down');
	
	test ( 'Should be able to do more than one test', sub {
		ok( $is_setup, 'should have been setup' );
	});
	
	is( $is_setup, FALSE , 'Should have been torn down');

The resulting package, Test/More/LikeXUnit.pm, ended up looking like this:

	package Test::More::LikeXUnit;
	
	use base 'Test::More';
	use Test::More;
	@Test::More::LikeXUnit::EXPORT = ( @Test::More::EXPORT, 'test' );
	
	use constant LOG_COMMENT_CHARACTER => '#';
	
	sub test {
		my $description = shift;
		my $block = shift;
		print "\n" . LOG_COMMENT_CHARACTER ."-----" . $description . "\n";
		&main::set_up();
		&$block;
		&main::tear_down;
	}
	
	
	1;

Short and sweet! I'm sure it can be tidied up but this'll do for now. I hope to post it on CPAN at some point.

So now my tests look something like...

As for the example foo test...

	use Test::More::LikeXUnit 'no_plan';
	use Test::Exception;
	
	my $foo;
	
	sub set_up {
		$foo = Test::SillyExample::Foo->new();
	} 
	
	sub tear_down {
		#tear down code goes here
	}
	
	test ( 'foo starts life with low aspirations' , sub {
		is( $foo->bar, LOW, 'foo has low aspirations' );
		is( $foo->is_high_achiever , FALSE '' );
	});
	
	test( 'foo raises the bar' , sub {
		$foo->set_bar( HIGH );
	
		is( $foo->bar , HIGH , 'Should now have high aspirations' );
		ok( $foo->is_high_achiever ,  'Should be high achiever' );
	});
	
	test ( 'foo can't be demoralised' , sub {
		$foo->set_bar( HIGH );
		throws_ok ( sub { 
			$foo->set_bar( LOW ); 
		} , 
		qr/Should not let others lower the bar/ ,
		'Should not allow lowering of bar');
		
		ok( $foo->is_high_achiever , 'Should still be a high achiever' );
	});

Now, that's more like it! Best of both worlds and everyone's happy!

P.S. If I had a lawyer, he'd make me say 'use entirely at your own risk'.

P.P.S. During the workshop, Willem showed this style (and the package) to the developers participating in his workshop and they seemed to take to it. Early on in the workshop, there was a lot more discussion surrounding wider behaviours that we weren't focusing on. Around the same time the attendees started separating the tests out like this, it seemed that they focused on the behaviour relating to the test's description more... but hey, that's probably just coincidence.