Roy Osherove

Principal, Team Agile

www.TeamAgile.com

blog: www.ISerializable.com

Recently, I’ve had the pleasure of speaking at a Microsoft Dev/IT Pro Days conference in Belgium.  I was approached by the organizers asking if I would want to do a session on “Designing for testability”, as part of three talks I was to give there.

The topic was not something I had spoken about before, but definitely something I had thought about, considered and wrestled with many times on many projects and occasions.

I set out to first determine what the definition of a “Testable System” might be, in my eyes. I came to the realization that a testable system is not measured in a vacuum, but its testability has to be “mirrored” through external testing related factors. For example, how easy would it be to write quality unit tests against such a system? And for that question to be answered, one has to ask what “quality unit tests” really means in this context. In this article we’ll try to define what a testable system design really means, and explore some basic design rules to make sure we can keep that testability in the system from the beginning.

Here’s my current definition of a testable system:

“For each logical part of the system, a unit test can be written relatively easily and quickly that satisfies all the following PC-COF rules at the same time:

Partial runs are possible

Consistent results on every test run

Configuration is unneeded before run

Order of tests does not matter

Fast run time”

Here are examples for all five rules. The first and foremost rule is the consistency one.

Consistent results on every test run

The test always has to fail or pass, but never switches between modes until a bug is fixed. That’s all good and well for the test quality, but how does it relate to the testable system? Well, remember, the definition says writing such a test is relatively easy and quick.  Imagine trying to come up with testing a scenario that uses multiple threads. That would be neither easy to write a test for or quick. Not only that, threading-related tests often exhibit a “random” style of behavior, where the test usually passes and only sometimes fails.

That can tell us several things. First, this is probably not a unit test, but an integration test (which is fine and dandy, but is not a replacement for a logic-level unit test) which may not be what we are looking for, assuming we just want to test the logic-level inside a single thread. Second this tells us that we have a dependency that we cannot control – that of the system threads (which is why I call it an integration test – we are testing our logic along with the threading utility). We will either have to remove that dependency or by changing the way we test things. Either way, a threading-related scenario is not a very testable one.

Here’s a simpler example that makes us design the system in a specific way so as to make it more testable. Consider the following piece of code that needs to be tested:

  public bool CanPurchase(Person p)

        {

            if (!(PersonValidator.IsValid(p)))

            {

                return false;

            }

            if (p.SSID != null &&

                p.SubscriptionType != "Canceled" &&

                p.CreditOnFile > 0)

            {

                return true;

            }

            return false;

        }  

The code marked in blue is the logic coded inside our method. The code highlighted in red in in fact an external dependency, in this case on a static method of a the PersonValidator Class. What’s the problem with this code? Why isn’t it considered testable? Consider the following:

·                           Suppose our test passes, that is, the logic is well tested. Then someone decides to put a bug in the IsValid() method of the PersonValidator class. Our unit test suddenly fails, even though the logic in the method works just fine. So, depending on the external method call, our test results may be inconsistent.

·                           In our test we’d like to test for a state where the person is valid, and another test where the person is *not* valid. Assume that the business rules for determining if a person is valid are obscure, hard to write and reside in a database somewhere. We can’t possibly know them while we’re writing the “CanPurchase” method. So how to we send in a person which is valid for one test, and not valid for another? That’s the problem of controlling an external resource such as the IsValid method. We can’t easily control it.

The question for this scenario then is “How easy would it be to write a test that can be consistent in the face of bugs in other logic and can control the Validation result so that the code can be tested under various states of validation?” The answer in this case may be “Not very easy”. We can take this as a sign that the system design is not very testable (indeed, it’s spelled out right there in my definition of a testable system).

            Here’s one way to make a system testable for this scenario: Use Interfaces. Consider the following refactoring of the code to use interfaces and think about the testability of the new code:

              IValidator m_validator;

                public void SetValidator(IValidator validator)

                {

                    m_validator = validator;

                }

                public bool CanPurchase(Person p)

                {

                    if (!(m_validator.IsValid(p)))

                    {

                        return false;

                    }

                    if (p.SSID != null &&

                        p.SubscriptionType != "Canceled" &&

                        p.CreditOnFile > 0)

                    {

                        return true;

                    }

                    return false;

                }

What we’ve done is use an interface on a validator object, which allows us to replace that validator with a fake validator in our tests. That Fake validator can be made to do whatever we want – return true, false or even through an exception if we tell it to. It’s just a simple class in our tests that is a “dummie” object that looks like a validator.

Here’s how a test with a fake object might look like with NUnit:

     [TestClass]

    public class PersonLogicTests

    {

        [TestMethod]

        public void CanPurchase_DefaultPersonValidated_IsFalse()

        {

            MyFakeValidaor val = new MyFakeValidaor();

            val.whatToReturn = true;

            PersonLogic logic = new PersonLogic();

            logic.SetValidator(val);

            Person p = new Person();

            bool result = logic.CanPurchase(p);

            Assert.IsFalse(result);

        }

        public class MyFakeValidaor:IValidator

        {

            public bool whatToReturn;

            public bool IsValid(Person p)

            {

                return whatToReturn;

            }

        }

}

The code marked in blue is the one using the fake validator. Notice how easy it is to control the return values and replace the external dependency in this case. This is a perfect example of how interfaces make code more testable.  Using interfaces in this case it is truly easy and quick to write tests for the coded pieces of logic inside that method.

            “But you had to add a whole new method just to replace the object, and I might not want to do that for my class” you may say. A solution could be to put the code that is there just for testing reasons (the setValidator method) in a conditionally compiled block or use the “Conditional” attribute on it. Antoehr solution is to use a constructor which takes the interface as a parameter instead or use a set property. The main thing is to understand that testing changes our code into testable state, which is not necessarily a bad thing. The design may be more “open” but that’s a “price” you might want to pay so that you know your code works as it should.

Next we’ll take a look at the “Configuration is not needed” part of the equation.

Configuration is not needed

Why is Configuration a part of this article? The ability to configure a class at runtime is an important one for unit tests. If code requires external configuration before it is tested, it will take more time to create tests for that code. It can also make the tests less manageable and not as easy to write. Consider the following piece of code:

        public bool IsConnectionStringValid()

        {

         string connString =

           ConfigurationSettings.AppSettings["conString"].ToString();

            //do some stuff

            //...

            return true;

        }

The code marked in blue uses external configuration files to do its bidding. If we tried to write a unit test for this code we’d have to include a configuration file in our test. That’s not a bad idea sometimes, but then it wouldn’t really be a unit test, but an integration test – a test which uses more than one unit test do the testing of some logic. Integration testing has its place and time, but we’re talking about a system which is easily unit testable.

So how do we overcome this problem of testability? We could introduce an interface into the mix – use a separate class to get the connection string settings, make it use an interface, and replace that class with a “fake” class of your own – providing a connection string of your own that does not need configuration. That’s one way to do it, indeed.

            Here’s another way: Use a virtual method to get the external data you need. If  we use virtual methods, you could then derive and override  the virtual method in a different class, returning whatever value you like. You can then use that derived class in your tests. The trick is to separate into a virtual method only the simplest code possible that uses the external dependency. Don’t’ extract an “if” for example, because you’ll be overriding that logic in your test.

            Here’s the same code refactored to use a virtual method:

Class ConnectionHelper

{

public bool IsConnectionStringValid()

        {

         string connString = getConnectionString();

            //do some stuff

            //...

            return true;

        }

        protected virtual string getConnectionString()

        {

            return ConfigurationSettings.AppSettings["conString"].ToString();

        }

}

The code in blue is the refactored code. By refactoring the code we were able to both keep the original functionality of the code, and make it more testable at the same time. That is the definition of the word “Refactoring” – Change existing code without changing its functionality. You’ve done this numerous times yourself – if you’ve ever change a method name, or extracted a long method into many little method calls.

Let’s see how we’d write a test with that refactoring in place. First, we’ll create a class that in effect derives from the class we’d like to test. That class we’ll call it the “TestableConnectionHelper” class and then in fact write our tests against that testable class. Here’s how it looks:

[TestMethod]

        public void ConfigBased()

        {

            TestableConfigBasedClass myClass = new TestableConfigBasedClass();

            myClass.mConnectString = "bad string";

            Assert.AreEqual(false,myClass.IsConnectionStringValid())

        }

        public class TestableConfigBasedClass:MyConfigBasedClass

        {

            public string mConnectString;

            protected override string getConnectionString()

            {

                return mConnectString;

            }

        }

Note that the testable class only overrides one simple thing, and the rest of the functionality under test stays the same as if this was the original test. This testable class also resides in our test project, and is not an original part of the production code.

            This method is called “Extract & Override” and is very powerful for Refactoring existing code into testability. It does require, however, the availability of a virtual method to override. That’s why I recommend, as a rule, to be virtual by default – make method calls virtual by default, and make sure to always use method calls when getting data from external resources in your logic. That guarantees a way in for writing tests.

We went into this because we wanted it to be easy and quick to write tests that don’t’ use a configuration scheme. That’s been achieved in this scenario.

Fast Run Time

It may not sound like much, but being able to have very fast unit tests can make or break your development cycle. Is half a second per unit test fast enough? Not even close. Imagine having 5000 methods with logic, with at least a couple of unit tests per method. That’s 10,000 tests right there. If each of them took half a second (ok, only 30%) that’s 25 minutes of run time just for those tests. How often would you run a test suite that takes half an hour? Once, maybe twice or three times per day, but no more. If it took 2 minutes to run things would be much different and you’d run it much more often.

But how does this “best practice” relate to a testable system? Well, in a non-testable system you would find it really hard, or really time consuming to write some of the tests against objects and make them run fast. That’s because you’d most likely have objects that do some sort of time=consuming activity using their external dependencies. For example, take the “Validator” example explained earlier.

            if (!(PersonValidator.IsValid(p)))

            {

                return false;

            }

What if the IsValid method calls a web service to do its bidding? Or it reads rules from the database? Or it simply performs a very length processing of in-memory rules that might take a few seconds or half a minute in some cases? Writing a test that runs fast with that method in place would be a non trivial task, as you would need to find a way to either make sure the data put in will not take too long to process, or that you configure the validator at runtime to not do so much processing. Either way you’ll end up writing more code in the test, making it more complicated, and having a harder time than you would have if this was a testable system. To make it testable you can take the same actions explained before: Refactor the code to use interfaces, or use a virtual method to check the validation and return a true/false result, then override it in your “testable” class.

Order Of tests does not matter

If you’re doing database related development, and tried to write “unit tests” for your data layer, you’re already familiar with this issue.

bool Insert(Person p)

{

//insert person to database

}

bool Delete(Person p)

{

//Delete person from database

}

The issue here is that you are relying on external state in the database for the tests to work. SO, in order for the delete tests to work, there has to be some row in the database to delete successfully, and for the insert tests to work there should be no duplicate line in the database. Many people who first start out testing data related actions do the “obvious” thing and simply make sure that the delete test runs after the insert test, and that they both try to insert and delete the same row data. That, of course, is problematic, because most unit test frameworks cannot guarantee the order in which unit tests will be executed. SO just the fact that it runs fine today does not mean tomorrow the tests won’t run based on some other criteria.

So how do you make sure that tests can be run in no specific order? 

In this case that’s very hard. You need to make sure the database’s state is rolled back before and after each test, which can be both time consuming and hard to code.  That means we break at least the “runs fast” rule, and even if we didn’t, we are breaking the “easy to write” rule in the definition.

What does that tell us?

It could tell us that the system is not testable, and I’d agree. A data layer, in my mind, is not testable by itself, and should always be tested along with the Database it represents. That’s a whole other topic, but essentially, you would want to test the “logic” in the data layer, which is, like it or not, highly coupled with that in the database. So, even though the system is not “unit” testable, we still want to test it.

Is making it testable unit interfaces and virtual methods a good idea? My opinion is that for anything *but* data layers, this is a good idea, but in the data layer case, this is a bad one because logic exists in the database as well.

Which can lead to a different conclusion: We’d want to write *integration tests* for the data layer, which look like unit tests and use the unit test framework tools to do their work. Integration tests don’t test a class or method separately, but several items working together.

If we subscribe to the fact that we would rather have integration and not unit tests for our data layer, everything falls into place:

Because these are integration tests than:

·                           It will be harder to write them while making sure they run fast (or impossible)

·                           It will be harder (but necessary) to make sure the order of the tests does not matter

·                           It will be harder to make sure they stay consistent between runs

The reason I’m using this example in here is that the ordering of tests, and the next part – the partial running of tests, both relate to integration-based tests problems, and these “problems” help us define not only what a testable design means, but what a “unit-testable” design means. If test ordering needs to be maintained and couldn’t be rewritten easily to make sure it does not matter, you’re probably writing integration tests anyway.

What’s the design rule here? It’s mostly an observational rule. Make sure you’re writing unit tests, unless you can’t, in which case you’ll have problems with ordering and partiality of the test runs.

Partial Runs of tests are possible

This last case related directly to the latter. When you have external state you’re likely to have a problem if you don’t run all the tests in your suite, since the external state between specific tests may need to be adjusted for the next test (so that a row will be inserted just in time for the delete test, for example).

Again, this problem is solvable, but how easy it is to solve it? Not as easy as writing simple unit tests without external state, that’s for sure. The whole problem of rolling back external state is that of controlling the before and after states and making sure they are the same for each test run.

If you can’t do that easily you’re either not using a testable system design, or you’re not writing a unit test.

Summary

The main design rules I’ve described till now are:

·                           Use interface based design

·                           Make methods virtual by default

In the next part of this article we’ll take a look at more specific examples of dependencies in code and who using these two simple techniques we can overcome them. We’ll talk about what refactoring the code had to undergo in order for these techniques to be effective, and see that refactoring the code for testability is easy to accomplish if you’re looking for the right things.

·                           Refactoring and Testing around singletons

·                           Refactoring and Testing Singletons

·                           Refactoring and testing direct object calls and static methods

·                           More..

One of the most important issues I've come to realize is the various ways of dealing with legacy code, mainly - how to deal with breaking multiple dependencies on an existing class or method that needs to be tested.

There are two basic ways to break dependencies, with various "dialects"  for each one. The both have their pros and cons and I won't explain in detail how to do them, but mainly, which of them you should choose and when.

Mock Objects

The first method of breaking dependencies is that of "Mock Objects".

In short, it relates to how one would send an instance of a "fake" class to be interacted with from the class under test instead of the *real* class. For example. to test a method that calls various Logging methods against a class that logs stuff to the database, you might be inclined to replace the instance of the logging class with that of a fake one, so that any calls made to the log will not actually go to a database, and thus will run much faster and won't need any configurations before the tests are run. That fake class can also tell us whether the "Log" method was called or not and how many times, since it is totally under our tester's control (they created it).

This article will deal mainly with the second main approach to testability, so if you’d like to read more about the Mock Objects method, there is plenty of material available on this over at www.MockObjects.com (also called the “Dependency Inversion” or “Inversion of Control” pattern). Martin Fowler also writes about this pattern in this fine article, though it is less related to testing and more generic in nature.

Extract & Override

The second method of breaking dependencies is much closer to the tested code, and relies heavily on inheritance.

It is usually called "Extract & Override".

The main idea here is that instead of "faking" the objects which are problematic. you "fake" the actual calls to them. An example might be in order.

Suppose we have a class called "MyCalculator" which contains code like this:

public double Calculate(int x, int y)

{

   if(x>1000)

   {

   Logger.StaticLog("Calc":Got an illegal number");//problematic call

   throw new Exception("Illegal number");

   }

}

suppose we want to test this method.  And suppose Logger.StaticLog() takes 10 seconds to do its magic (you'd probably do that asynchronous but this example is good enough). Obviously any tests that run this method will take too long to run, and unfortunately, this is a static method, so we can't easily "inject" or "replace" the Logger with a fake instance since there is no instance to work with. So instead, we will refactor our code to make it testable like this:

public double Calculate(int x, int y)

{

   if(x>1000)

   {

   callLog("Calc":Got an illegal number");//refactored

   throw new Exception("Illegal number");

   }

}

protected virtual void callLog(string text)

{

   Logger.StaticLog(text);

}

the idea is that in our tests, we won't really use this original calc class, but instead inherit from it and override the "callLog" method to not really do anything (which means you'll want to make those overridable methods with as little logic as possible) like this

[Test]

[ExpectedException(typeof(Exception))]

public void Calculate_IllegalNumber_ThrowsException()

{

   TestableCalc c = new TestableCalc();

   c.Calculate(1001, 0);

}

class TestableCalc: Calculator

{

public string callLogText = string.Empty;

   protected override void callLog(string text)

   {

   callLogtext = text;

   }

}

Notice that the second method allows you to break even very hard dependencies in a relatively easy manner.

Pros and Cons of Mock Objects

Pros

-         Most mock object frameworks allow you test test interaction between objects in a very easy manner, by allowing you to add any number of “expectations” and return values to your mock objects that will be tested against. The syntax is clear and all of the hard work of maintaining the state of the cals to the mock object is performed with the mock object system.

-         The last sentence means that tests involving a single mock object will be very readable and easy to write, especially when multiple calls are expected against it, or when parameter values are being tests for method calls (for example, to make sure that a Logger class is called 3 times with different values each time would be very easy to write with a framework such as NMock or NUnit.Mocks)

Cons

-         the more depdencies you need to break on the tested object (legacy systems are usually just the thing that might have that) the mock Mock objects and stub objects you’ll need to create and initialize in your tests, and the more refactoring you will have to do to the system to make it testable (adding new interfaces, constructors, parameters and such..). this may prove almost impossible on some systems where you don’t own all the code and where refactoring needs to be done on code that you don’t own.

-         The more dependencies means also less readable tests because all the initialization is not readable at all when its 20 lines of code or more before the actual test takes place. Even if you refactor your mock initialization code it might still not be very readable.

-         Mock objects don’t help when you try to deal with code that calls static methods, singletons and other types of calls in which an object instance cannot be replaced (there are ways to refactor singletons into testability but you might not have that luxury)

Pros & Cons od Extract & Override

Pros

-         It’s much faster and easier to refactor the system into testability with Extract& Override (perhaps this should be named “Extractoring” ? ) You don’t need to introduce new interfaces into a system and the only requirements are that the class is overridable and that the methods are overridable (protected)

-         It’s faster and easier to write tests for such a system because test setup and object initialization is easier and clearer.

o       To mimic system behavior override a method and return what you want using a publicly exposed member on the inheritor class or throw an exception by checking a public flag

o       To check interaction calls simply check a publicly exposed member (“wasMethodXXCalled” or “GetMethodXXPassedValues()” on the inheritor class

-         You can break any dependency you like right at the source.

o       You can override a Boolean validation check such as if(configurationSettings[“key”]==value) into if(hasKeyValue(value))

o       You can override full calls for static methods such as turning LogClass.StaticLog(“text”) into callLog(“text”)

Cons

-         You have to do all the grunt work of verifying calls to methods, including parameter values, yourself. When we used mock objects, the mock object framework we used did that for us automatically. But when we do this manually  (i.e inherit our own class and override methods) we need to make the code work on our own which can lead to long code inside the overriding class.

-         If you need to override some methods and not others in a dynamic way (i.e different tests override different methods) you need to either implement base-forwariding logic into the overridden methods or create multiple inheriting classes which override different methods. In either case, it can get out of hand quickly.

-         In some types of applications, especially Real-Time, where performance matters a lot, adding many virtual method calls may actually be a hindrance. There are ways around that though, using various conditional-compilation techniques which are not discussed in this article.

Hidden bugs with E&O – biggest hazard of all

-         When you override a method call, you usually end up with at least (hopefully only) one line of code in the actually virtual base method which simply calls some external dependency and may return a value. There is a problem here that may not be recognizable at first. To explain the problem I’ll need a little example.

Here’s the code from the refactored class in the first example:

public double Calculate(int x, int y)

{

   if(x>1000)

   {

   callLog("Calc":Got an illegal number");//refactored

   throw new Exception("Illegal number");

   }

}

protected virtual void callLog(string text)

{

   Logger.StaticLog(text);

}

Now, consider what happens if suddenly, the design changes a bit and the logger takes in another parameter (and designs change with time). The new signature for the Logger.StaticLog now looks like this:

Public static void StaticLog(string text)

Public static void StaticLog(int severity, string text)

There is a new parameter for severity of type int, and this is a new overload for the previous method. Fine and dandy. Now suppose we want to change our code to send a severity each time we call the log instead of using the old method. We might refactor the virtual method to take another parameter such as :

protected virtual void callLog(int severity,string text)

{

   Logger.StaticLog(text);

}

But notice that we don’t actually change the actual call to the static log method. In fact, in our test we would do the same thing as in the first example – override the virtual method and make sure it is called with the correct parameters. So our tests might look like this:

[Test]

public void Calculate_IllegalNumber_LogsSeverity()

{

   TestableCalc c = new TestableCalc();

   c.Calculate(1, 0);

   Assert.AreEqual(100, c.callLogSeverity);

   Assert.AreEqual(“illegal number”, c.callLogText);

}

class TestableCalc: Calculator

{

public string callLogText = string.Empty;

public int callLogSeverity = 0;

   protected override void callLog(int severity,string text)

   {

   callLogtext = text;

callLogSeverity = severity

   }

}

We’re overridng the callLog method to make sure it is called with the correct values.

To make this test pass our code might look like this:

//the method we are testing

public double Calculate(int x, int y)

{

   if(x>1000)

   {

   callLog(100,"illegal number");

   throw new Exception("Illegal number");

   }

}

protected virtual void callLog(int severity,string text)

{

//notice that there is a bug here!!

   Logger.StaticLog(text);

}

Yes – this example will actually pass the callLog test, even though there is an obvious bug in it – we don’t really call the logger correctly!

When you think about it, who’s to say that inside the real virtual method the code actually calls the correct static method? No one. We could make our code call the “callLog” refactored method correctly, but we have no way to assure that the actual line inside the CallLog method actually calls the method correctly. That’s because we actually override that call in our tests. That’s a dangerous place to be and you need to watch out for making mistakes on this. Your interaction test might say that everything is ok, but you may forget to actually call the real methods correctly in the virtual calls.

This type of hazard does not exist with mock objects since with mock objects we don’t override any real implementation – just interfaces and method signatures.

This problem may not be as big as you think – I’ve rarely come across it. It’s kinda hard to find though, and it requires debugging to dig out. It would have been cought earlier with real Mock objects instead of using Mock Methods

Summary

Hopefully, you’ve seen that all good things have a dark side, but that doesn’t mean we can’t use them, we just need to know where the dark side begins and make sure we don’t go there (or that we at least know the way back). Mock objects and E&O(“close refactoring” as I like to call it)  are two great routes to a testable application.

You’ll find the need to use E&O much more in legacy code than you would in new, green-field code, where you get the chance to write it testable from the beginning (Test-Driven code is often a great example of using interfaces and Mock Objects with the possibility of never needing to E&O anything).

The end result is all about being able to produce something out of these tests, and you’ll need to figure out which of these approaches works best for you.

I’d also recommend Michael Feathers’ great book entitled  “Working Effectively with legacy code” for more methods and refactoring patterns to break code dependencies. Indeed, everything I’ve outlined here also appears there and may be considered an extension to some of the books ideas and thoughts.

You can also check out some of the Unit Testing Guidelines I keep adding to over time – there’s a whole category of them in my weblog.

During the past week I've attended a .Net master class with Juval Lowy. One of the last items Juval talked about was his notion of software project management using Staged Delivery. Being a Agile advocate myself, I've outlined several of his points and wrote down several differences and similarities during the talk. I've written my thoughts on these issues in this article as a way to put extract the essence of my thoughts on this amazingly complex issue in writing so that I can better understand and share these things with myself and you, the reader. Please feel free to comment and share your thoughts as a comment or track back.

I’ve outlined the basic values that Juval talked about regarding his staged delivery approach below. Interestingly enough, there is a large correlation between he values of various Agile methodologies such as XP and Scrum to his world view. However there are several differences. I’ve decided to split the various values he talked about into 3 categories:

1. General values
2. Similar values to Agile
3. Values that are different from Agile

I’m not going to pretend that I can give you all the details of the life cycle in a simple short article like this, but I’ll try to give you as much of the highlights as needed to make sense of the points I’m going to make.

General values:

1. Staged delivery is all about iterative development and short release cycles.
2. There are several stages.
   1. First, there is a big up front design process done by the architect. That design process can take up to 30% of the life cycle. The outcome of that design  is the various components, entities and interactions of the system.
   2. Every component has a life cycle:

                                                               i.      SRS document which is review

                                                             ii.      A test plan is generated

                                                            iii.      Do some construction of the component to flush out various difficulties that will come up later. (This is throw-away code)

                                                           iv.      Detailed design which is reviewed

                                                             v.      Development + test development

                                                           vi.      Code review

                                                          vii.      Integration testing

3. This whole life cycle implies that even the simplest component can take on average a minimum of 3 weeks
2. Team involvement:

1. Estimation:

                                                               i.      The team is the best source for estimation purposes. Statistically Juval has found that the team produces the most accurate estimation results using the following process:

                                                             ii.      For each component/feature the whole team tells, one by one, how much they think it’s going to take to build

                                                            iii.      Put the results in an excel spreadsheet and calculate the average and the standard deviation

                                                           iv.      For each team member who thought that it would take more than the standard deviation, ask them why (they might have an insight no one thought about)

                                                             v.      After this, go around the room again and ask for estimation. After a couple of such rounds, the results will even out and you’ve got yourself an accurate as possible estimate

                                                           vi.      Lots of peer review is in the process

4. Requirements
   1. Start with the custom/marketing/domain expert and derive a list of basic requirements
   2. Generate use cases
   3. Generate the domain model & interaction diagrams
   4. Generate architecture diagrams
   5. Generate the SDD documents
   6. Code
   7. Each step can back up to the previous step if things are not what they appear to be

OK. Those were the highlights, and certainly not a comprehensive list.  However, I think the next two categories will highlight many of the finer points in the methodology.

Similarity to the various Agile methodologies

1. you can always change your mind

The whole process is built out of very frequent iterations and releases. At each point we can discover that the requirements have changed. If they do, we can tell the customer that there’s not problem. They should accept, however, that the schedule might be slipped or that other features might not make it into the overall release. It’s their choice.

In agile methodologies, the custom can always change their mid as well. However, this process can only take place *between* iterations (which can very from 2 weeks to 3 months to 6 months depending on the methodology you chose to implement)

2. Daily build

Every day you build the whole source code and automatically run a smoke test. This provides daily integration and knowledge that you always have a release on your hands. Agile development (XP in particular is very specific about demanding this)  also has this notion of “continuous integration” though the daily build cycle is slightly different in that it provide more steps and integration points. I’ll outline these diffs in the next category.

3. Spike solutions

During the component life cycle mentioned by Juval, there is a stage where there is a “throw away” code project that allows the developer to make sure they know what they are getting into. Agile development (XP and Scrum in particular) provide the idea of “spiking”  in order to uncover unknown project risks such as lack of technology knowledge or unforeseen integration problems.

4. Design review

Each component has a design review before implementation. In Agile we also have a continuous design review process where each design change can and should be consulted with other members of the team.

5. Testing
   1. Tests are written in conjunction  with the production code

In Juval’s words :”No line of code goes untested”. That means each little functionality that your class, object or component provides should be tested. Testing is done using Test clients (drivers) that allow the developer to trigger the various component’s actions with various values. Agile development (XP in particular) talks about writing unit tests that do exactly this, only they have no GUI, and are fully automated. More on this in the next section.

2. The test plan is written along side the component design documents
3. Every component has an emulator and a simulator

                                                               i.      Makes the developer provide interfaces early to each component

                                                             ii.      You can test against emulators to get hard coded results and exceptions

                                                            iii.      You can automate smoke tests

                                                           iv.      Every component has 3 versions: Real, emulator and simulator.

In agile development you use “Mock Objects” inside you unit tests to provide this same functionality.

6. Team based estimation

As described earlier, The team is in charge of giving out estimated on development time. This is just like in Agile development, where no one but the development team can decide how much time something is going to take to build. The people on your team are individuals with different strengths and weaknesses. They know (and learn throughout the project) how much time it takes them to build things.

7. Feature ownership

Because people on the team say how long it will take them to build stuff, they also feel more accountable on the products of their work. They now have more vested interest in the success of the product. In agile, all team members must also be committed to the final product delivery and other than estimating they also pick and sign on the tasks they would like to perform on each iteration. Essentially, the team drives itself into the finish line, guided by the things the customer wants the most for the specific iteration.

8. Burn down chart

Just like in Scrum, there’s  a burn down chart that tells us when we expect to happen on the time vs. % done scale.

9. Code
   1. Should be self explaining

That is, if you’re finding yourself writing a comment, most likely you’re not writing the code in a self explanatory way. There should be almost no comments except when absolutely necessary in the code. It should be read like a book.

In Agile, the production code and the unit tests are the best and most recommended form of documentation.

2. This is not so say that there is no outside documentation of the API, but that topic resides in the “difference” category, later on.
9. Client involvement is encouraged.

The more time the client spends with you, your team and the product features decisions, the better. In agile, and XP in particular, Customer-on-site is a must for resolving much of the communication issues between the team and the design decisions, the client and the priorities that have been decided. There is a difference however, in that the customer does not have to be there.

11. One click to build, setup, deploy and test

These processes should be fully automated, so that integration can be as smooth as possible. Goes along with Agile’s notion of CI.

12. Defensive programming

Assert as much as possible all the assumptions in your code. On average, every 5^th line is an Assert. That’s the only way to flush out those bugs early and often. With Agile (XP, Test Driven Development) the Debug.Asserts are replaced/complemented with Unit tests that perform their own asserts.

13. Overtime is evil

As a practice, you should never do overtime. Sure, sometimes its called for but if you’re doing it all the time than it’s not overtime, it’s bad management. Again, Agile supports this notion as well.

14. Buddy Programming

Each developer has an assumed “buddy” developer hey can go to whenever they have a problem or want to get things done which are a little complicated to be doing alone. This “buddy system” is much like pair programming in XP, except it is less strict. When two people work on the same code you also avoid the “If he leaves we’re doomed” symptom. (Funny remark from one of the folks in the class was “Unfortunately buddies usually leave together”..  )

15. Have a coding standard

Coding standards ease maintenance and productivity for developers across the project. Agile supports this notion as well.

16. Demo and show the status of the product early and often.

Iterative development with small releases means that every short period of time we can show the customer a new version or a release of the product with more functionality. This lets them See what they really want out of the application and have a clearer understanding of their own requirements. Agile development also believes in showing progress early to the customer and all stakeholders.

So far it certainly seems as though the process Juval was talking about is Agile in many ways. But is it really subscribing to the four rules that all agile methodologies stand on:

l      Individuals and interactions over processes and tools

l      Working software over comprehensive documentation

l      Customer collaboration over contract negotiation

l      Responding to change over following a plan

Here are the differences (in highlights) that I’ve written down for myself during the talk:

Differences

1. Big design up front

(Note: While th following paragraph details what I got from the master class, Brian Noyes, who works and teaches this class at IDesign, has some clarifications and comments in this post) 

A big design phase done by an architect that can take around 30% of the overall project time is done up front. This is a great departure from the “A simply as possible” approach taken by XP. Agile (XP in particular) believes in Iterative, evolutionist's design approach where the design of the system is done incrementally as the system is being developed.  With up front design you spend a lot of time designing parts of the system that you can’t really tell if they will be integrated together well in the end. Usually, the passage from Design into implementation can be a dramatic stage where a lot of inconsistencies in the design model are uncovered. This isn’t the architect's fault, though. Today’s software architects  try to mimic the work of engineering architects (like electronics, buildings) in that they believe that a design that is good enough can produce predictable work and schedule every time. The difference between engineering and software architecture is that the number of tools a software architect has in their disposal to make sure they are doing the right thing is minuscule in comparison to those given into the hands of the engineer. The engineer has mathematical analysis tools, pressure point simulations and pre-known information of building the exact same thing for hundreds of years. Software architects have peers to check their work and make sure things “seem OK”. As a consequence the design will almost always be less adequate that initially thought out to be and changes will have to take place when implementation time arrives.

2. External API documentation (much like MSDN, NDoc..)

According to this approach you need external API documentation in order for new developers in your project to be easily integrated with your current team. You don’t want to re-explain every time how this class talks to that class, assumptions, calling conventions and so on. In agile, however (XP in particular) the Unit tests provide the perfect documentation o four APIs, assumptions and calling conventions, and there is less need for complicated documentation(and maintenance thereof) such as this.

3. QA runs the automated test clients

The test clients that the developer creates for their components are run automatically by the QA team.  In Agile (XP),all the unit tests are run by the developers before even reaching QA. This allows for bugs to be discovered earlier in the loop.

4. Design for performance

In the suggested methodology, you should always think about performance and design for higher performance than the one requested. That is, if the requested performance(or scalability) required is X, design for 10X (for example, up front design to support 10 times the number of clients required in 3 months). Again, a big departure from Agile where you only design for the simple cases and add scalability / performance when it is needed, and no sooner. Why? Because we want to deliver the best value for the customer, as soon as possible. In fact, many of the things we dim as “Needed” that were not in the requirements are actually never going to be used, and we’re wasting time implementing them when we could have been implemented a required set of features/functionality. Also, we’re making the code more complicated by adding this design up front and the code is then harder to maintain. Also, consider what happens when you spend an extra week building functionality that the client didn’t ask for because you thought “they’ll need it on the next version anyway”. A week later the customer’s business needs have changes the highest priority now becomes some feature which could have been already developed by now if you hadn’t been stuck on the performance improvement no one asked for.  Of course, this is a simple situation. This about times whee you spent months building an sophisticated scalable framework for you application which ended up supporting 20^th of the users you designed it for? (And 20^th of that was the actual required task). What would your manager want more? A 2 month schedule decrease or 2000% performance improvement over the required features? It’s all about delivering value to the client as soon as they need it.

5. The client does not have to get involved if they don’t want to

No so in Agile. In agile the client is key for communication, feedback and prioritizing the features for the next iterations.

6. Documentation contains context maps

Context maps tell the developers and stake holders how the system works in the context of Identity boundaries, threading, tasks, architecture and other things. No one in agile land talk about this, and I think it definitely should be on any project, agile or not.

7. Lightweight XML comments on your Code

Simply for tool tip reasons for your developers, this is a good idea. While agile and XP in particular support the idea of least documentation as possible, I think this is still a good idea.

8. Design is solely by the architect.

In agile the team has much more responsibility. Even in changing the existing design to fit the reality that has been encountered. Of course, this isn’t done alone, but the team has much more power because it’s all about the team, not the process in Agile development.

9. Every class has its own spec

Even more documentation. In agile and XP specifically, the design of a class is done just-in-time using the interested parties using CRC cards.

10. Aim to minimize the rework each component may need

Assuming the design is that good and that requirements never change, the component’s design should not change and work on it will be done just once. Again, a big departure from Agile’s “simplicity first, Refactor later” approach where the component’s iterative design can change the interfaces and implementation if needed. One can arguer however that a component in this sense is a black box with public interfaces, where only the inner implementation can change (think web services). In that case there is a middle ground in that you try to minimize as much as possible the change in the public interfaces of a system for integration purposes.

Conclusion

So what do we have here?  What can we deduct from this?

Obviously, the difference in approach between Agile methodologies and Staged Delivery as Juval portrays it is there, but its not as deep as you might originally assume. In fact Juval’s way of doing things is taking the best of both worlds (in his view) – Agile and Engineering, and making them work together using disciplined developer behavior every step of the way. I’m sure it works great. However, just as someone who believes in the Agile side of things should take a long hard look at this approach, so should the “other” side take a look at the way things are done in the agile camp. Both sides have their wins, just as they do their losses. Both can learn a great deal from each other. One of the things Agile is insistent upon is the ability to change the process until it fits your own development team/organization. This includes “borrowing” from other methodologies to make yourself more successful. If that proves right – aren’t we all just a little happier?

One of the things Juval talked about in relation to XP is that in order to advocate XP you need to be a zealot, religious person. I’d say that sounds a little like FUD. I’m in the agile camp but there’s nothing I like more than to listen to people telling me why they think they have a better way. Do I care that I was wrong as long as the end result is better? Not really. The only thing that I’ve learned so far is that I’m capable of making mistakes, but I’m a great learner from them. To be a zealot is to believe in something in spite of all the hard evidence, and that’s true only for the more extremist side of any bunch. As software developers we are more prone to logical conclusions. That is, if you explain it to us in a way that makes sense, there’s really no way we can say no. The matter of software development is so complex because it’s all about people. And no one gets people, really.  That’s why we all have different ways of looking into managing people processes. That’s why we should all keep an eye open to make sure we’re looking at reality and not just what we want our reality to be. Being open to new ideas and ways of doing things is part of the Agile way, and as such, looking at things that have proved successful should be just as important as looking at things that fail from within.

I’m purposely not trying to “judge” which is “better” or “more appropriate”, but merely trying to see the differences, so that it will be easier to pinpoint the exact locations where the methodologies make themselves stronger.  These “diff” points are what we all should be looking at, and ask each other “Why like this and not like that?”. Who knows, we might find a pleasant surprise.

• A custom attribute class, which is just a fancy name for your intentions but does not actually implement anything
• a custom test case, which overrides the standard Nunit test case behavior and lets you inject your on implementation along the way (this is where the real work is done)
• a test case builder class which simply returns an instance of your new test case.

• You create only one simple attribute class that inherits from a base class
• You override two methods : "BeforeRunTest" and "AfterRunTest"

1. Create a new class library project
2. Add a reference to NUnit.Framework and Nunit.Core
3. Download this base class and add it to your test project
4. Start adding your own derived classes like shown above.
5. That's it.

By Roy Osherove

Summary:

In this article I’ll show what problems we encounter today when we perform unit tests against database related component, and the various ways used to solve these problems. I’ll also show a new method to solve these problems, which is far easier and simpler than any of the others. At the end of this article you will have the tools necessary to do simple and easy unit tests against database related components without the need to write messy cleanup code after your tests.

Update:

You will find more information on issues and solutions regarding this approach in the following article: Issues and solutions for database unit testing with enterprise services. You may also find XTUnit helpful in this regard.

Introduction

Test Driven Development makes life easier. Unit tests are the lifeblood of a good Test Driven Development approach (and thus, are also the lifeblood of most of the agile methodologies out there such as XP, Scrum and others). Therefore it stands to reason that one should strive to create unit tests with the least amount of work and maintenance, which will in turn allow the developer to focus on the business problem at hand and the various design and algorithms needed to create a working product/component.

For the past couple of years I’ve been struggling with Unit tests, That is, I’ve used them in major production projects and have seen the good and the bad that can come from a fully test Driven approach. I’ll let the “good” part be taken care of by the numerous other articles out there that talk about the benefits you get from Unit Tests and TDD, and will focus on one of the most problematic parts of Unit tests in a real world project – testing against the database.

Testing Data Access layer components (DALs for short) is problematic in a number of ways. The number one reason f is this:

Testing the various CRUD(Create, Retrieve, Update, Delete) operations on a single class can result in a massive amount of “garbage” data residing on the test database (You are using a test database, aren’t you?). This creates several problems:

1.      Problem: Garbage Data

Your database will eventually be full of junk data which you’ll need to get rid of sooner or later

2.      Problem: Affecting other tests

Worst, you may break one of the golden rules of unit testing:  No test should be dependent on another test to perform correctly. That is, you should be able to run your tests in any order you choose or maybe just one of them at a time, and they should all still work. That means that every test needs to start with a known state on which it will act. For DALs this usually means a known state of records in the database.

But guess what? When you test CRUD operations you’re actually changing the state of the database, so the next test that will be run that might depend on a specific record being there might fail because you just deleted that record in your current test. Anything can happen.

3.      Problem: starting tests from a known state

This is the other side of the previous problem: My test needs to have something in the database in order to perform correctly. For example – it needs a specific “category” entity in the Category tables in order to test inserting a child “Product” into the Products table. I need to make sure I insert these values into the database before I perform the test, but I also want to make sure that data is no longer there after I finish my test.

Dealing with the problems

So how do you deal with all this garbage data that’s making your database and tests unstable? Obviously, you need to make sure that before and after each test you leave the database in the same state that you got it in. That is – you need to “undo” your CRUD operations after testing them. Here are some of the most prevalent methods used to achieve this:

1. Remove it manually

This is the most obvious and in simple cases one of the simplest to achieve. To do this, after each test (at the end of it) you execute the opposite actions than the ones you took in the test. For example, if you inserted a record in the test – you delete it at the end.

Some problems and questions that arise from this approach:

o       If I inserted the record into the Database using my objects’ Insert() method which I was testing – should I remove it from the database using the object’s Delete() method (which might have not been tested yet or may not even exist yet!) or should I do it against the database using direct calling of stored procedures or direct ADO.Net classes?

o       If I choose to remove the records I inserted directly from the DB – this involves some serious amounts of extra code, residing in my test! And that code can have bugs as well.

o       If I choose to use my object to delete and Insert as well - what do I test first? (Chicken and the egg kind of question..)

o       I can also choose to remove all the garbage data at the end of all the tests run (TestFixureTearDown method). But then I’d need lots of data and custom code and …. It’s just complicated.

o       I can ignore all my garbage data and restore my database from backup before running all the unit tests. This might solve the problem partially but may still leave a few unit tests that mess with other test’s data. A big no-no.

2. Use the transaction object in ADO.Net

In his very interesting and insightful book, “Test Driven Development with Microsoft.Net”,  James Newkirk addresses this same problem with a seemingly simply solution: Execute all your actions inside a transaction and rollback that transaction when you’re done with each test. This is a great solution and can definitely be implemented, but also has its problems:

• You need to worry about components that manage their own inner transaction but still will need to use outside transactions initiated by the unit tests.
• To workaround this problem James came up with an implementation of the Command pattern inside the unit tests (the book has a good example). However –even if it accomplishes the task nicely, after using this pattern we are left with more complicated unit tests that we should have, and all this just to be able to support rollback functionality.

A simpler way

The previous two approaches are the most used, and if you’ve used either of them you’d know that they are still hard to achieve with massive amount of custom work just to make the database data more reliable.

When I was reading Jame’s book, in the chapter that discusses implementing transactions, I came across this little sentence (abbreviated):

“We want to achieve the same functionality as “RequiresTransaction” for each tested component, just like in COM+”

This sentence triggered an interesting thought : why work so hard? Why not let COM+(Or, “Enterprise Services” in .Net) do the work of coordinating the transactions for us? Thus – I came up with another way of implementing database rollback functionality :

Using Enterprise Services

What we want is to be able to achieve the transaction functionality without all the overhead of writing our own custom Transaction manager class as prescribed in James’ book. How do we do that? With COM+ Enterprise services.  Here is the technical gist of the method:

Our simple efforts start with this simple base class

using System;

using NUnit.Framework;

using System.EnterpriseServices;

namespace TransactionTesting

{

[TestFixture]

[Transaction(TransactionOption.Required)]

public class DatabaseFixture:ServicedComponent 

{

[TearDown]

      public void TransactionTearDown()

      {

            if(ContextUtil.IsInTransaction)

            {

                  ContextUtil.SetAbort();

            }

      }

}

}

Here are some important pointers about this class:

• This class will be the base class that your DAL unit tests will inherit from. It allows for automatic transaction enlistment using the COM+ architecture through the EnterpriseServices namespace in .Net.

for that end :

• It inherits from ServicedComponent
• It has the following attributes on it:

1. [TestFixture]
2. [Transaction(TransactionOption.Required)]

The second attribute specifies that whenever this class is instantiated or invoked, COM+ will automatically create a transaction context for it to run under, or enlist it into an existing transaction context.

• It has a simple TearDown method which will call ContextUtil.SetAbort(). This causes the following flow to occur:

1.      Before each test a transaction context is opened (or enlisted into an existing transaction)

2.      The test is performed

3.      TearDown is called and the transaction is rolled back thus

We get the following benefits from this approach:

1.      it is dead simple and easy to implement

2.      You need no code whatsoever to roll back any database changes your components have made throughout the test. They will be rolled back automatically.

3.      You do not need to manage transaction by yourself.

A few more settings and we’re done

We’re still not finished but we’re awfully close to the finish line.

All serviced components need to have strong names. This one is no exception.

• Use the SN.exe tool with the –k option to create a key file.
• In your test project open AssemblyInfo.cs and set the following attribute (add it if it is not there):

[assembly: AssemblyKeyFile(@"..\..\..\test.snk")] 

Notice that I’ve put here a relative file path. This is the file I generated from sn.exe and it is located in the same directory as my .sln file. That way I can put this same attribute on all the other projects that need it as well.

• Put a static Version attribute in your test project:
  • Look for this in the AssemblyInfo.cs file :[assembly: AssemblyVersion("1.0.*")]
  • And change it into [assembly: AssemblyVersion("1.0.0")] or whatever version you like.

This step is important because by default when the attribute is at its default state Visual Stuido.Net will automatically increment the assembly version with each build. That’s a bad practice to have on any project, but when you’re dealing with a COM+  component (A ServicedComponent is just that) you need to realize that for every separate version of the test assembly that you build, you will have created a different COM+ component in the COM+ catalogue of the testing machine. SO – after you’ve run the tests 50 times a day your COM+ catalog might turn out to look pretty scary and we don’t want that. Setting the version to a fixed version saves us this little headache.

• Since your test assembly now has a strong name, every assembly that it references must have a strong name as well. This is easily done: put this same attribute in each of the referenced projects set in your test project.

That’s basically it. You’re done.

Here is a simple example of a test fixture that derives from this class to achieve a perfectly transparent data rollback after each test. This class tests a simple class that does category insert and delete against the Northwind database:

using System;

using NUnit.Framework;

using System.Data;

using System.Data.SqlClient;

using Microsoft.ApplicationBlocks.Data;

using NorthwindDAL;

namespace TransactionTesting

{

      public class CategoryTests:DatabaseFixture

      {

            string CONN = @"Server=localhost;Database=Northwind;Trusted_Connection=True";

            [Test]

            public void Insert()

            {

                  CategoryManager mgr = new CategoryManager();

                  int newID = mgr.Insert("Test category");

                  VerifyRowExists(CONN, newID,true);

            }

            [Test]

            public void Delete()

            {

                  CategoryManager mgr = new CategoryManager();

                  int newID = mgr.Insert("Test category");

                  bool result = mgr.Delete(newID);

                  Assert.IsTrue(result);

                  VerifyRowExists(CONN, newID,false);

            }

            private void VerifyRowExists(string connectionString, int existingRowID,bool shouldExist)

            {

                  string SELECT = "Select * from Categories where categoryID=" + existingRowID.ToString();

                  SqlDataReader dr = SqlHelper.ExecuteReader(connectionString,CommandType.Text,SELECT);

                  Assert.AreEqual(shouldExist,dr.HasRows);

                  dr.Close();

            }

      }

}

 As you can see there is no clean up code anywhere. That’s how simple this technique  really is.

Just so you can see there are no tricks up my sleeve – here is the CategoryManager tested class (in VB.Net – just for the heck of it):

Public Class CategoryManager

    Private CONN As String = "Server=localhost;Database=Northwind;Trusted_Connection=True"

    Public Function Insert(ByVal CategoryName As String) As Integer

        Dim sql As String = "Insert into categories (categoryName) values('" + CategoryName + "');Select scope_identity()"

        Dim result As Integer = NorthwindDAL.Data.SqlHelper.ExecuteScalar(CONN, CommandType.Text, sql)

        Return result

    End Function

    Public Function Delete(ByVal id As Integer) As Boolean

        Dim sql As String = "delete from categories where categoryID=" + id.ToString()

        Dim result As Integer = _

            NorthwindDAL.Data.SqlHelper.ExecuteNonQuery(CONN, CommandType.Text, sql)

        Return (result = 1)

    End Function

End Class

No tricks, no hidden cards. The database actually gets updated, only the updates are rolled back.

Conclusion

In this article I’ve show a way to define simpler, more maintainable unit tests against the database access layer components using the powerful and easy to use services of COM+.  Hopefully using this technique database unit testing can become a much less painful experience, and thus allow more developers do more tests on their database code – and find those bugs as soon as possible.

Download the source for this article: Direct link

Summary

In this article I’ll be looking into a trickier part of unit testing– testing the functionality of classes who cannot be instantiated on their own, abstract classes.

I’ll show how to overcome this obstacle using “Mock Objects”, a technique which has a whole methodology behind it all on its own. We’ll see how mock objects provide us with abilities that let us query what’s happening inside our derived class.  

Introduction

The problem with abstract classes

Mock objects

A simple project idea

Planning our first test

Our first test

Creating our Mock object

Creating our abstract class

Making the test fail

Making the test work

Conclusion

References

Introduction

Unit testing is all good and well as a theoretical nirvana. “Yeah, we do unit tests” is a great answer to have when someone asks you, but achieving this is a process just like any other software development process. The overall idea looks great, but you come across problems that don’t fit your original plans, or things you can’t deal with the way you’re use to. Unit testing has several of these obstacles routed at the core of the mythology.  One of those obstacles is the testing of objects that cannot be instantiated on their own (abstract classes are one of several manifestations of this problem). As I’ll demonstrate, this too is possible using Mock objects, objects that exist for the sole purpose of helping us in our task of testing.

The problem with abstract classes

So what’s the definition of the problem we are facing? Well, to run a test against a class, we need to have an instance of that class to work on. We need something to call methods on and get values back if needed. We need a real live object. In the case of abstract classes there’s no way to do this.

We could make the class non-abstract for the sole purpose of testing it, but that would violate one of the most important concepts of Unit Tests – they should not alter the behavior or the data of the tested application. Making changes to the design of an object model for no reason other than tests is not the course of action we want to take.  

We need to ask ourselves what we want to test in an abstract class. Usually abstract classes contain the plumbing required by classes that will derive from them. What we want to know is if the derived classes, as clients to the services the abstract class provides, are getting all the services they need. This method means that we are essentially wanting to do a “black box” test on the abstract class and make sure that any derived class that’s going to use it will have what we want it to have.

So this shifts our focus a bit. How do we test that any “client” to the base class is receiving the services it needs? This could be quite a complex problem to test, because another important concept here is that we want to test only one thing at a time. If we test derived classes of the base class (which might be a design problem in itself) we are essentially testing the functionality of the base classes as well, not to mention we’ll have to build those classes for our tests to work at all!

One of the most elegant solutions to this problem is actually very simple in concept – we’ll build objects just for our tests!

Mock objects

Mock objects are a very handy technique to tests objects that are “mediators”. Instead of referring to a real domain object we call the Mock object which pretends to be the real object. The Mock object is used to validate any assertions and expectations we have of that object and we can fully interact with it while at the same time still have total control over the results of our object’s method calls. There’s a whole Mock Objects testing framework, which is an alternate view of unit testing.

You can learn more about Mock objects by starting from this link.

Mock objects come in very handy when we either want to test an object that “changes stuff” in our application and we want to stop it control it, or when we want to test an object that uses other objects in order to do its work. A data layer object that uses the database would be a good example of when we want to control such things (we never want our unit tests to corrupt live data, that’s rule #1, ask your DBA).

In this case we can use a Mock object to derive from our class and make sure that it, as a “client” to our class, receives all the necessary “services” it expects.

A simple project idea

In order to explain this in more “close to home” terms, I’ll make up a simple project task.

We’ve designed a simple object model in which we have an abstract class “Task” which will contain a “start” method. The class will be used to derive other tasks from it, but will provide each derived class with abstract methods that will need to be implemented: BeforeStart(), OnStart(), and AfterStart(). Each derived class will be able to use these methods to perform initialization before a task begins, the task itself when needed, and cleanup after the task has finished (think ServicedComponent type events).

Planning our first test

Ok. We have no code yet, but what do we test first? We want to make sure that for any derived class, calling the Start() method, actually triggers the BeforeStart(), OnStart(), and AfterStart()  methods inside it. To do that, we’ll need an instance of a class derived from Task (which does not exist yet).

This is a perfect candidate for a Mock object. We can use a Mock object that will derive from out base class and will let us know if its inner methods were called.

We’ll want to used a very simple mechanism here.

Our first test

In our first test we’ll assume we have an object derived from Task and well call it’s Start() method. Then we’ll assert that all three inert methods were called.

[Test]

public void TestOnStartCalled()

{

      MockTask task = new MockTask();

      task.Start();

      Assert.IsTrue(task.OnStartCalled);

      Assert.IsTrue(task.BeforeStartCalled);

      Assert.IsTrue(task.AfterStartCalled);

}

This code won’t compile because we don’t have any MockTask class defined.

Creating our Mock object

Our mock object will derive from Task. Notice we’re building it assuming Task already exists.

public class MockTask:Task

{

      public bool OnStartCalled=false;

      public bool BeforeStartCalled=false;

      public bool AfterStartCalled=false;

}

The beauty of this is because this is our mock object we can make it do whatever we want. In this case we are simply adding flags to it. Those flags will have to change somehow later on, but that’s not our problem now. Now we’re concerned about making the code compile.

Our code still won’t compile because we haven’t created out Task class yet.

Creating our abstract class

public abstract class Task

{

      public void Start()

      {

      }

}

Notice we’re making the class as simple as possible, just making our code compile.

Making the test fail

If we run the test now out code will compile, but the test will fail miserably. The derived class had no OnStart ,BeforeStart and AfterStart methods defined. Therefore it’s “called” flags will always remain false.

Making the test work

Let’s make the test work by adding the required functionality to our base class:

public abstract class Task

{

      public void Start()

      {

            BeforeStart();

            OnStart();

            AfterStart();

      }

      //all base classes must implement this method

      protected abstract void OnStart();

      protected abstract void BeforeStart();

      protected abstract void AfterStart();

}

public class MockTask:Task

{

      public bool OnStartCalled=false;

      public bool BeforeStartCalled=false;

      public bool AfterStartCalled=false;

      protected override void AfterStart()

      {

            AfterStartCalled=true;             

      }

      protected override void BeforeStart()

      {

            BeforeStartCalled=true;

      }

      protected override void OnStart()

      {

            OnStartCalled=true;

      }

}

Conclusion

“Be-a-utiful!” as Bruce almighty often remarks.  We now have a repeatable test that makes sure our base class calls all the methods we require from it.

References

·         MockObjects testing framework

·         What is a Mock object?

·         Download the source for this article: Direct link

Every once in a while in my development career I realize just how much I’ve been missing out on some technique that I’ve just discovered. Such was the case with learning about Design patterns, as was the case with learning how to write self-documenting code, and countless other occasions where I felt I’ve “discovered a whole new world” and from that moment on tried to stick to the new things I’ve learned. Doing Test-Driven development (TDD) was definitely one of those techniques.

I touched it first when I discovered the notion of XP.  XP is a software development methodology that has some hardcore guidelines on how to achieve success in development projects but it’s a very steep learning curve, and you can find plenty of books nowadays that talk about this subject. TDD is one of the trademarks of XP (you might have heard about other, more “provocative” notions such as pair-programming) but it wasn’t invented there. There’s a whole movement that supports what is known as “Agile methodologies”. You might find it interesting to explore this whole subject. A good start would be Martin Fowlers’ site, one of the granddaddies of everything that is agile.

But I digress. The whole notion of TDD is just that. Development is based on testing. Does that sound weird? That’s because we’re used to it being the other way around. Usually we’ll code something and throw it over to QA for testing. Sometimes we’ll hack out some tests after the code base is done, just to make sure that “everything appears to be working” . TDD takes this whole process and turns it upside down. That means that whenever we are going to code something up, we stop, and first do a test that makes sure our code works. Yep. We don’t have any code yet. But we have a test to prove it.

Here are the bullet points for how to start doing TDD:

1.      Make a test that fails

2.      Make it work

3.      For every new feature goto step 1

It’s that easy. Really. Now, I know this sounds weird at first. How can we test code that does not exist yet? Well, if our code even fails to compile, that means the test failed in our book. So it’s a start. To show just how easy this is, let’s make a pretend project.

Our first task in the project is to create a class named Calculator that contains a method that adds two numbers. Now, given what I’ve just explained, the first task would be to first have one test that fails. So let’s create a new Console Application project. This project will be our testing project. It’s not the real project that we have to do. This project will contain the code that tests our production code.

In our Main() method let’s add a few lines of code that determine whether out code (which does not exist yet) runs ok.

The first thing we want to do is test that we can create a new instance of our calculator class:

 [STAThread]

 static void Main(string[] args)

 {

       Calculator calc = new Calculator();

 }

Now, of course this code won’t compile, but what we have just done is created a test that relies on the fact that a class named Calculator exists. So our next step is now to make the test pass . To do that we’ll finally create our “real” project and add a class named calculator to it. Once we add a reference to that project into out test project, we’ll see that the code compiles and runs just fine. We just made the test pass.

Our next goal? Add an “AddNumbers()” method that takes two numbers and returns the sum of those numbers. Again, what’s the first thing we do?  That’s right. We create a test that fails. So let’s add code to test this to our previous test code:

[STAThread]

static void Main(string[] args)

{

      //test creation

      Calculator calc = new Calculator();

      //test addition

      int retVal = calc.Add(22,20);

      if(42!= retVal)

      {

            throw new Exception("calc.Add(2,2) did not return a value of 4");

      }

}

That was simple. Again, this code won’t compile. We have to create the code that does this addition operation. Once we’ve added a simple method for Add() we can now re-run the test and make sure that it works. Let’s make this a little more interesting though. What happens when we send in a null value in one of the parameters? Suppose that the design requires us to throw a CalcException when a null value is sent to this Add() method.  If want to add this feature, we first need to test for it:

//test exception

try

{

      int retVal = calc.Add(null,22);

}

catch(CalcException ce)

{

      //test passed

}

catch(Exception e)

{

      throw new Exception("calc.Add(null,22) did not throw a calcException");

}

Now the code won’t compile again, We don’t have a CalcException class defined. Once we define it, we can run the test, but it will fail again, since we’ll get a standard exception and not a calcException from the AddMethod. So we’ll change our code to throw that exception…. And so on…

As you can see this process is pretty easy. What I’ve done at every step is define a goal, and than make sure I pass it. What we have achieved at the end of this session is a piece of code that is thoroughly tested. Not only that, we get several added bonuses:

·        Anyone who looks at our tests can understand what out intention and purpose is of each method

·        We can make changes to our code and know “what we broke” with the click of a mouse and fix it just as fast

·        If our testing covered all of our code like this, we could find bugs in our programs at build time that would have taken a very long time at the customer’s site

But some things are definitely missing from our current solution:

·        No automation. If I wanted to run a build and get the results of the tests that were performed I’d have a long day coming up with a solution that traces the messages and output them.

·        No re-use. I’d have to re-write any output handling from scratch every time I want to test a project

·        No decoupling. Code that runs a test must be totally decoupled from other code that runs a test. I always want my testing code to run within a given context, a known set of known values for parameters and so on. I don’t want other tests messing up my state when they change stuff. There’s no framework that allows me to have a separate state for each testing code without significant work every time.

NUnit

What’s needed here is a framework that allows us to write tests and not worry too much about how we’re going to get back their results. The de-facto framework for .Net unit testing is NUnit. Currently at version 2.1, NUnit provides us with a set of base classes and attributes that allow us to abstract away our unit tests and concentrate on the code that actually does the testing. The beautiful thing is that moving from our current coding/testing style to Nunit style requires little learning and is very easy to master.

Nunit allows us to separate out testing code into what is logically know as tests, test fixtures and test suites. The concept is very simple.  

·         You write a test to test a single piece of functionality.

·         You group one or more tests inside a test fixture that allows you to easily add repeatable state for each test (I’ll explain shortly)

·         You group one or more fixtures inside test suites to logically separate the test and their meaning

So how do we turn our code into using Nunit style?

·         Download and install Nunit

·         Add a reference to the nunit.framework.dll to our testing project

·         Add a using clause for NUnit.Framework workspace in a new class file

Now we’re ready to start working.

Change the class’s name to MyTestingClass .

This class will hold the fixture for our tests. We also need to let the Nunit framework know that this class is a fixture, so we simple add a [TestFixture] attribute on top of the class name.  You can remove the default constructor from the class (but don’t make it private!). Once we’ve done that, we have a class that looks like this:

[TestFixture]

      public class MyTestClass

      {

      }

Easy enough we just have to start adding tests to the class. We’ll use the code from our previous example to test against Calculator.

A test in a fixture is defined as a public method that returns void and accepts no parameters, marked with the “[Test]” attribute and with one or more assertions inside. So let’s add the first test.:

 [TestFixture]

 public class MyTestClass

 {

       [Test]

       TestAddition()

       {

//test addition

             int retVal = calc.Add(22,22);

             Assert.AreEqual(44,retVal,

"calc.Add() returned the wrong number");

}

 }

As you can see it’s the same code as before only now it’s sitting in a method of its own .The method is decorated with the [Test] attribute .Also, instead of having to manually throw an exception I’m using the Assert class which is part of the Nunit framework. You’ll get to know this class a lot as this is the main instrument you’ll use to verify your code. The Assert class will fail the current test if the condition passed to it is false. It contains only static methods that allows you to make sure that a value is not null, equals other values and just send in any Boolean expression you like. You can also send in messages that explain the meaning of this failure.

Now we need to build our testing project so we can move on to the next step.

Test suites

Test suites in the new versions of Nunit are derived directly from the namespaces that the test fixtures reside in. If you have two fixtures in separate namespaces (i.e. one is not contained inside the other) they’ll be considered as residing in two separate test suites.

So what now?

GUI

Well, now it’s time to run your first Nunit test. When you install Nunit you get two choices on how to run your unit tests: Either a GUI based version of the Nunit test runner, or a console based one. The Gui one is located in Start->Programs->Nunit V2.1->Nunit-Gui . When you open it you get a pretty “not beautiful” but very functional interface that allows you to select an assembly with compiled unit tests inside it and run all the tests that are there.

·         Select File->New project

·         Select Project->Add assembly and select your compiled tests assembly.

Once you’ve selected your assembly you’ll see the tree on the left fill up with namespaces, with the names of any test fixtures inside them and the names of any tests inside them. Now you can see why it’s important to put those attributes on our classes and tests. It’s how we make out testing GUI find them and run them.

Make sure the top node of the tree is selected and click “Run” on the right side of the form. You see the progress bar very quickly turn green to signify success. If the bar is red, it means that test has failed and you can go back and make it succeed.

I won’t go into too much detail here on how to use all the features in the Nunit GUI but you can learn all you need by reading the documentation for it.

Feel free to close the GUI, it will remember the last assembly you loaded in it next time.

On important thing to note here is that once one test inside a test suite fails, all other tests will not run.

Console

Besides the GUI version of the Nunit test runner, you also get a Console test runner. This is especially good for when you have an automated build procedure that runs unattended. You can make it call the console version of Nunit which outputs directly into the stdOutput and have it log all results.

To make the console do the testing, you need to switch to [Nunit program files folder]\Bin . From there you can run Nunit-Console.exe providing the name or full path of the assembly to test against. I urge you to put that path inside the global PATH environment  variable so that you can use the console easily from anywhere.

More testing goodies we get

·        Another attribute you can put on a test is the [Ignore(reason)] attribute, Use this to skip certain tests , but the reason for their skip will be displayed inside the GUI.

·        You can have a [Setup] and [TearDown] method inside your fixture. The [Setup] runs before each test in the current fixture is run, and the [TearDown] runs after each test.  These methods are very useful for when you want all your tests to use the same set of clean initialized data. In there you can initialize global variables, delete or create  needed files and so on. This of [Setup] as an implicit contructor for each test, and of [TearDown] as a destructor for it. Methods that are marked by these attributes should not be marked as tests as well!

·        You can have a [TestFixtureSetUp ] and [TestFixtureTearDown]  methods as well. These methods will be run only once for each test fixture tested. Use them for global initialization and cleanup of resources that can be shared by all tests in that fixture.

·        Another excellent attribute we get is the [ExcpectedException] attribute. When a test method is decorated with this attribute and no exception of the type specified in the attribute is thrown inside the test, thw test has failed. This is perfect to check that your components throw exceptions at the right moment, such as bad user input and so on. We’ll use this attribute to add another test to our fixture, which test for the CalcException:

[Test]

[ExpectedException(typeof(CalcException))]

public void  TestException()

{

      int retVal = calc.Add(null,22);

      Assert.IsTrue(true);

}

As you can see it couldn’t be easier.

The Nunit-Addin

Now that you understand the basics of writing unit tests with Nunit, it’s time for me to introduce one of the coolest gadgets related to this subject – the Nunit Addin.

This add-in allows you to ,instead of re-opening the Nunit-Gui every time you need to make sure your tests pass, to just right click on the project or class you wish to test and hit “Run Test(s)”.  You’ll get all the information inside VS>NET’s output window.

This add-in allows more than just this functionality, however. It allows you to test a single method from inside the code editor. Just click anywhere inside the code of that method and hit “Run test”.

Another very powerful feature allows you to do what is called “Ad-Hoc testing”. You can create any method, and not even put a [Test] attribute on it. Than, inside that method right click and hit “Test with”-“Debugger” and you immediately step into that method without needed to create a separate project that calls this method. Indeed, very powerful. You can also debug using different versions of the .Net framework or even Mono. This add-in is a must have for quick incremental development.

A word before we finish

The technique I’ve shown here means very little if not pursued diligently. Remember – the first thing you ever do is write the test, not the code.  If you keep this up you’ll eventually end up with a system that is fully testable and with fewer bugs. You’ll also find that you think about your component’s design more responsibly, because you’re looking at them from a different perspective.  Once you get the “Zen” of it, you’ll start to even have more fun doing it. You’ll also gain confidence in changing your code. You’ll get instant feedback if something broke and you can squash bugs at their inception point.

Another thing that needs to be known– Nunit is the unit testing framework for .Net, but like Nunit there are many others, for practically any semi-popular programming language out there. If you program in C++ for example, take a look at CPPUnit. There’s also a JUnit out there. In fact, Nunit is a port from JUnit into .Net. There are also commercial frameworks and addins that try to provide added functionality through add-ins for .Net unit testing: Some of those include HarnessIt,csUnit and X-Unity.

Most of the non-.Net frameworks all support the same kind of logical notions of test case, fixture and suite, but each one might provide different means of expressing them. Attributes are unique to .Net but in other OO laguages you might have to derive a class from  TestFixture to declare it  as a fixture and so on. You can find the complete list of frameworks over at www.xprogramming.com (look for the “downloads” link).

Advanced issues

This article is just a first in this series. In the next articles I’ll talk more about real world problems facing a developer who wants to test real-world applications. Some of these issues include:

·         Testing abstract classes?

·         Testing complex object models and dependencies

·         Testing database related features

·         Mock objects and their use

·         Testing GUI interactions