In the last post, I described the extended detour that my attempt to implement CIFactory took me on.  Its worth reiterating, just in case it wasn’t clear from my last post, that I consider the detour needed to implement CIFactory (and by extension, modify the Vault ‘package’ for CIFactory to be a little more flexible) entirely worth the effort.  The results (a comprehensive CI environment with built-in support for NDepend for code-dependency analysis, NCover for unit test code coverage analysis, Simian for code-similarity analysis, and a host of other capabilities) are perfectly wonderful and offer a host of feedback mechanisms to ensure that my project not only builds but passes all its unit tests and reports back to me exactly where my coding has grown sloppy or lazy

But this lead to another ‘gotcha’ that in turn lead to another somewhat lengthy detour.

Attributes on Unit Tests

To understand the problem, a little background is needed first.  Most .NET developers are familiar with the notion of applying attributes to code.  Essentially, this is a way to add capabilities to code declaratively rather imperatively (the distinction being that declarative code says what you want to do and imperative codes says how you want to do it).

Nearly everyone who uses unit test frameworks for .NET are familiar with applying attributes to classes to declare them as ‘TestFixtures’ and to methods to declare them as ‘Tests’.  The canonical example of a test fixture with a test looks like…

[TestFixture]
public class MyTestFixture
{
    [Test]
    public void MyTest()
    {
        int somevalue = 1;
        Assert.AreEqual(1, somevalue);
    }
}

…where the [TestFixture] and [Test] attributes are used to tell the unit test runner that it should consider this MyTestFixture class to have one or more tests and that MyTest( ) is a test method to run.

MbUnit (our unit testing framework of choice) defines several other attributes for our use in attributing our tests and fixtures as well, FixtureCategory, and Author.  Used to organize test fixtures into categories and to identify the author of a test fixture respectively, these attributes can be used to selectively run (or not) tests when using either the MbUnit console test runner or the MbUnit GUI test runner.  MbUnit also allows running tests (or not) based on the namespace of the test fixture as well, but there isn’t an actual attribute for this of course, its just the namespace for the test fixture class that is used to identify these to the test runner.

A Method for Standardizing on FixtureCategory Attribute Values

While we really don’t need or use the Author attribute to identify tests, at my company we do make use of a collection of standardized test fixture categories to aggregate our tests together so that anyone running them can know what kind of tests they are dealing with.

For example, this is how the FixtureCategory attribute looks when applied to a test-fixture class…

    [TestFixture]
    [FixtureCategory("DatabaseDependent")]
    public class DataProviderTest
    { ...

The trick (among other things) is that the string "DatabaseDependent" is entirely case-sensitive (so "DatabaseDependent" doesn’t equal "databasedependent" and of course doesn’t equal "Database-Dependent").  How do we ensure our whole project team is being consistent in their use of categories for unit tests?

To ensure that we get consistent strings used for the categories, we developed a utility class with some static members that the developer just needs to reference into their project so that they can use the class members in place of user-entered strings so that everyone uses the same categories.

    [TestFixture]
    [FixtureCategory(UnitTestType.DatabaseDependent)]
    public class DataProviderTest
    { ...

Once everyone’s test fixtures use a consistent set of categories to describe them, it becomes possible to automate the running of one category of tests vs. another category of tests from the NAnt build scripts (and thus from the CI server since it runs the build scripts to do its work).

Or so I thought.  This assumption lead to my next somewhat time-consuming detour.

The trouble with MbUnit’s NAnt task

When I went to the build script that drives the CI server for this project to inspect the part of the NAnt script that drives the unit testing, I discovered that the following is more or less the prototype syntax for the MbUnit NAnt task..

<project default = "tests">
  <target name = "tests">
    <mbunit
        report-types = "Html"
        report-filename-format = "myreport{0}{1}"
        report-output-directory = "."
        halt-on-failure = "true">
      <assemblies>
        <include name = "FizzBuzz.Test.dll"/>
      </assemblies>
    </mbunit>
  </target>
</project>

Stare at it as hard as you can and you won’t see anywhere to specify the filters to use to select the tests to run. 

A check of the CIFactoy newsgroup confirmed my discovery…and that to get around it I would need to talk to the MbUnit folks to see how their NAnt task handles fixture filters.

This lead to my next disappointing discovery: an answer to my post in the MbUnit newsgroup confirmed that there is no support in their NAnt task for fixture filters.

Extending the MbUnit NAnt task to handle FixtureFilters(or ‘open source to the rescue, part 2’)

When I asked the question of the MbUnit newsgroup re: support in the MbUnit NAnt task for fixture filters, it was suggested to me by the MbUnit team that while they were actively working on an alpha release of MbUnit v3 (code-named ‘gallio’…I really hate what Microsoft did for code-naming software projects~!), they were also accepting patches to MbUnit 2.4.1 for an upcoming 2.4.2 maintenance release and I might be able to do this work for them.

Flush with my prior initial success in contributing to an Open Source Software project with the Vault Package for CIFactory, I decided to take up the opportunity to try to add this support for the MbUnit NAnt task myself and contribute the changes back to the MbUnit project for eventual inclusion in a subsequent release.

There were just a few (significant) hurdles to overcome in this process:

1. I’d never written a NAnt task, had no idea what’s involved, and had no idea what the requirements were or the API looks like
2. MbUnit uses Subversion for their SCC system and I had about zero Subversion experience since I left the world of open source SCC systems back when CVS had the mindshare so never had a real need to familiarize myself with it
3. I had never looked at even a single line of the MbUnit codebase and so had no idea how the filters work or even where the code was that drove the MbUnit.Tasks.dll that encapsulates all the MbUnit extensions for NAnt

Despite these impediments, I decided to give it a shot.  After all, the worst that could happen would be that I find myself unable to figure it all out and I waste some of my own time 

Kudos to the MbUnit team

I am pleased to report, however, that my experience interacting with the MbUnit development team was every bit as pleasant as that of my CIFactory experience.  Their posts to the newsgroup in response to my queries for additional information, design direction, and feedback on several of my intermediate strategies for accomplishing what I was after were very helpful and insightful.

My first attempt at an implementation was to support just a single include and a single exclude filter speficied directly in the <mbunit…> node of the build script…

    <mbunit
        report-types = "Html"
        report-filename-format = "myreport{0}{1}"
        report-output-directory = "."
        halt-on-failure = "true"
        include-category ="SomeCategory "
        exclude-category="SomeOtherCategory">
      ...
    </mbunit>

I got this working quite quickly, but it would only support categories and only a single include and exclude category name.  Back to the drawingboard, but this proved to me that it was at least conceptually possible to do what I was after.  The second attempt supported multiple filters and looked much like this…

<project default = "tests">
  <target name = "tests">
    <mbunit
        report-types = "Html"
        report-filename-format = "myreport{0}{1}"
        report-output-directory = "."
        halt-on-failure = "true">
      <assemblies>
        <include name = "FizzBuzz.Test.dll"/>
      </assemblies>
      <includeCategory name="SomeCategory" />
      <includeCategory name="SomeOtherCategory" />
      <excludeCategory name="StillAnotherCategory" />
    </mbunit>
  </target>
</project>

Better (and supports multiple filters), but starts to look like messy XML (repeating multiple nodes over and over).  After several other iterations and several rounds of feedback from members of the MbUnit newsgroup, I settled upon the following approach (which is both well-organized XML and clearly understandable)…

<project default = "tests">
  <target name = "tests">
    <mbunit
        report-types = "Html"
        report-filename-format = "myreport{0}{1}"
        report-output-directory = "."
        halt-on-failure = "true">
      <assemblies>
        <include name = "FizzBuzz.Test.dll"/>
      </assemblies>
      <fixtureFilters>
        <filter type = "category" action = "include" value = "Unit"/>
        <filter type = "category" action = "exclude" value = "Integration"/>
        <filter type = "author" action = "include" value = "sbohlen"/>
        <filter type = "namespace" action = "exclude" value = "Microdesk.SkillPortal.Test.LongRunning"/>
      </fixtureFilters>    
    </mbunit>
  </target>
</project>

This approach supports an unlimited number of include and exclude filters, allows filtering of fixtures based on category, author, and/or namespace (basically all the attributes one can assign to an MbUnit TestFixture), and also has the benefit of being completely extensible if new filter types are added in the future (though as the MbUnit team has pointed out to me, the highly-anticipated ‘gallio’ release will use a completely different method for filtering tests so this is unlikely).

The following is the function I wrote that does the bulk of the work to construct the filter based on the settings in the NAnt build file…

        private FixtureFilterBase BuildFilter()
        {
            FixtureFilterBase filter;

            if (IncludesSpecified())
                filter = FixtureFilters.Current;
            else
                filter = FixtureFilters.Any;

            foreach (FixtureFilterElement element in filters)
            {
                switch (element.FilterType)
                {
                    case "category":
                        {
                            if (element.FilterAction == "include")
                                filter = FixtureFilters.Or(filter, FixtureFilters.Category(element.FilterValue));
                            else if (element.FilterAction == "exclude")
                                filter = FixtureFilters.And(filter, FixtureFilters.Category(element.FilterValue, true));
                        }
                        break;

                    case "author":
                        {
                            if (element.FilterAction == "include")
                                filter = FixtureFilters.Or(filter, FixtureFilters.Author(element.FilterValue));
                            else if (element.FilterAction == "exclude")
                                filter = FixtureFilters.And(filter, FixtureFilters.Not(FixtureFilters.Author(element.FilterValue)));
                        }
                        break;

                    case "namespace":
                        {
                            if (element.FilterAction == "include")
                                filter = FixtureFilters.Or(filter, FixtureFilters.Namespace(element.FilterValue));
                            else if (element.FilterAction == "exclude")
                                filter = FixtureFilters.And(filter, FixtureFilters.Not(FixtureFilters.Namespace(element.FilterValue)));
                        }
                        break;

                    default:
                        break;
                }
            }
            return filter;
        }

To be sure, there were other modifications to the MbUnit.Tasks.dll source code needed to support this new NAnt task syntax, but the NAnt Task API’s own reliance on attributes to declaratively spec out its behaviors made the work much more straightforward than I expected.  Like all development projects that result in a positive outcome, the designing of the solution and the deciding of the right course of action ultimately took longer than the actual coding work

After a little quick research into how to create a DIFF to send back to the MbUnit team for their consideration for inclusion in the next MbUnit 2.x release, I was finally ready to get back to coding.

Or so I thought.  The last detour on this project will be detailed in my next post.