« Unit Testing in C++ and Objective-C just got ridiculously easier still | Main | The iPad reconsidered »
Tuesday
Dec282010

Unit Testing in C++ and Objective-C just got easier

Day 133-365 : Catching the bokeh.jpg

Back in May I hinted that I was working on a unit testing framework for C++. Since then I've incorporated the technique that Kevlin Henney proposed and a whole lot more. I think it's about time I introduced it to the world:

Introducing CATCH

CATCH is a brand new unit testing framework for C, C++ and Objective-C. It stands for 'C++ AdaptiveAutomated Test Cases in Headers', although that shouldn't downplay the Objective-C bindings. In fact my initial motivation for starting it was dissatisfaction with OCUnit.

Why do we need another Unit Testing framework for C++ or Objective-C?

There are plenty of unit test frameworks for C++. Not so many for Objective-C - which primarily has OCUnit (although you could also coerce a C or C++ framework to do the job).

They all have their strengths and weaknesses. But most suffer from one or more of the following problems:

  • Most take their cues from JUnit, which is unfortunate as JUnit is very much a product of Java. The idiom-mismatch in C++ is, I believe, one of the reasons for the slow uptake of unit testing and TDD in C++.
  • Most require you to build libraries. This can be a turn off to anyone who wants to get up and running quickly - especially if you just want to try something out. This is especially true of exploratory TDD coding.
  • There is typically a certain amount of ceremony or boilerplate involved. Ironically the frameworks that try to be faithful to C++ idioms are often the worst culprits. Eschewing macros for the sake of purity is a great and noble goal - in application development. For a DSL for testing application code, especially since preprocessor information (e.g. file and line number) are required anyway) the extra verbosity seems too high a price to pay to me.
  • Some pull in external dependencies
  • Some involve a code generation step

The list goes on, but these are the criteria that really had me disappointed in what was out there, and I'm not the only one. But can these be overcome? Can we do even better if we start again without being shackled to the ghost of JUnit?

What's the CATCH?

You may well ask!

Well, to start, here's my three step process for getting up and running with CATCH:

  1. Download the headers from github into subfolder of your project
  2. #include "catch.hpp"
  3. There is no step 3!

Ok, you might need to actually write some tests as well. Let's have a look at how you might do that:

[Update: Since my original post I have made some small, interface breaking, changes - for example the name of the header included below. I have updated this post to reflect these changes - in case you were wondering]

#include "catch_with_main.hpp"

TEST_CASE( "stupid/1=2", "Prove that one equals 2" )
{
    int one = 2;
    REQUIRE( one == 2 );
}

Short and to the point, but this snippet already shows a lot of what's different about CATCH:

  • The assertion macro is REQUIRE( expression ), rather than the, now traditional, REQUIRE_EQUALS( lhs, rhs ), or similar. Don't worry - lhs and rhs are captured anyway - more on this later.
  • The test case is in the form of a free function. We could have made it a method, but we don't need to
  • We didn't name the function. We named the test case. This frees us from couching our names in legal C++ identifiers. We also provide a longer form description that serves as an active comment
  • Note, too, that the name is hierarchical (as would be more obvious with more test cases). The convention is, as you might expect, "root/branch1/branch2/.../leaf". This allows us to easily group test cases without having to explicitly create suites (although this can be done too).
  • There is no test context being passed in here (although it could have been hidden by the macro - it's not). This means that you can freely call helper functions that, themselves, contain REQUIRE() assertions, with no additional overhead. Even better - you can call into application code that calls back into test code. This is perfect for mocks and fakes.
  • We have not had to explicity register our test function anywhere. And by default, if no tests are specified on the command line, all (automatically registered) test cases are executed.
  • We even have a main() defined for us by virtue of #including "catch_with_main.hpp". If we just #include that in one dedicated cpp file we would #include "catch.hpp' in our test case files instead. We could also write our own main that drives things differently.

That's a lot of interesting stuff packed into just a few lines of test code. It's also got more wordy than I wanted. Let's take a bit more of a tour by example.

Information is power

Here's another contrived example:

TEST_CASE( "example/less than 7", "The number is less than 7" )
{
    int notThisOne = 7;

    for( int i=0; i < 7; ++i )
    {
        REQUIRE( notThisOne > i+1  );
    }
}

In this case the bug is in the test code - but that's just to make it self contained. Clearly the requirement will be broken for the last iteration of i. What information do we get when this test fails?

    notThisOne > i+1 failed for: 7 > 7

(We also get the file and line number, but they have been elided here for brevity). Note we get the original expression and the values of the lhs and rhs as they were at the point of failure. That's not bad, considering we wrote it as a complete expression. This is achieved through the magic of expression templates, which we won't go into the details of here (but feel free to look at the source - it's probably simpler than you think).

Most of the time this level of information is exactly what you need. However, to keep the use of expression templates to a minimum we only decompose the lhs and rhs. We don't decompose the value of i in this expression, for example. There may also be other relevant values that are not captured as part of the test expression.

In these cases it can be useful to log additional information. But then you only want to see that information in the event of a test failure. For this purpose we have the INFO() macro. Let's see how that would improve things:

TEST_CASE( "example/less than 7", "The number is less than 7" )
{
    int notThisOne = 7;

    for( int i=0; i < 7; ++i )
    {
        INFO( "i=" << i );
        REQUIRE( notThisOne > i+1  );
    }
}

This gives us:

    info: 'i=6'
    notThisOne > i+1 failed for: 7 > 7

But if we fix the test, say by making the for loop go to i < 6, we now see no output for this test case (although we can, optionally, see the output of successful tests too).

A SECTION on specifications

There are different approaches to unit testing that influence the way the tests are written. Each approach requires a subtle shift in features, terminology and emphasis. One approach is often associated with Behaviour Driven Development (BDD). This aims to present test code in a language neutral form - encouraging a style that reads more like a specification for the code under test.

While CATCH is not a dedicated BDD framework it offers a several features that make it attractive from a BDD perspective:

  • The hiding of function and method names, writing test names and descriptions in natural language
  • The automatic test registration and default main implementation eliminate boilerplate code that would otherwise be noise
  • Test data generators can be written in a language neutral way (not fully implemented at time of writing)
  • Test cases can be divided and subdivided into SECTIONs, which also take natural language names and descriptions.

We'll look at the test data generators another time. For now we'll look at the SECTION macro.

Here's an example (from the unit tests for CATCH itself):

TEST_CASE( "succeeding/Misc/Sections/nested", "nested SECTION tests" )
{
    int a = 1;
    int b = 2;
    
    SECTION( "s1", "doesn't equal" )
    {
        REQUIRE( a != b );
        REQUIRE( b != a );

        SECTION( "s2", "not equal" )
        {
            REQUIRE_FALSE( a == b);
        }
    }
}

Again, this is not a great example and it doesn't really show the BDD aspects. The important point here is that you can divide your test case up in a way that mirrors how you might divide a specification document up into sections with different headings. From a BDD point of view your SECTION descriptions would probably be your "should" statements.

There is more planned in this area. For example I'm considering offering a GIVEN() macro for defining instances of test data, which can then be logged.

In Kevlin Henney's LHR framework, mentioned in the opening link, he used SPECIFICATION where I have used TEST_CASE, and PROPOSITION for my top level SECTIONs. His equivalent of my nested SECTIONs are (or were) called DIVIDERs. All of the CATCH macro names are actually aliases for internal names and are defined in one file (catch.hpp). If it aids utility for BDD or other purposes, the names can be aliased differently simply by creating a new mapping file and using that.

CATCH up

There is much more to cover but I wanted to keep this short. I'll follow up with more. For now here's a (yet another) list of some of the key features I haven't already covered:

  • Entirely in headers
  • No external dependencies
  • Even test fixture classes and methods are self registering
  • Full Objective-C bindings
  • Failures (optionally) break into the interactive debugger, if available
  • Floating point tolerances supported in an easy to use way
  • Several reporter classes included - including a JUnit compatible xml reporter. More can be supplied

Are there any features that you feel are missing from other frameworks that you'd like to see in CATCH? Let me know - it's not too late. There are some limiting design goals - but within those there are lots of possibilities!

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (65)

So am I understanding it correctly:

instead of writing multiple test cases using the same fixture as you would in other frameworks, you'd write one test case with multiple sections?

And the setup code is just the bit outside the sections, rather than a separate fixture/class/whatever?

Anyway, looks pretty nice. I think I'll have to try it out. :)

December 28, 2010 | Unregistered Commenterjalf

Boookmarked! I usually do plain C (mathematical coding), but I want to do one or two iPhone apps in some moment of my life and this can come in handy for the Objective C part. Thanks for sharing it!

Cheers,

Ruben
In my blog: The emacs 30 Day Challenge: Using gnus to read mail

December 28, 2010 | Unregistered CommenterRuben Berenguel

@jalf Yes, that's how it works when you use SECTIONS. I think this is the cleanest way to go - but I also fully support class based fixtures too (complete with sef registerting test methods) and free-function based tests (if you need fixtures you can just declare a stack based object).

December 28, 2010 | Registered CommenterPhil Nash

Nice one, Phil!

For the record, I changed the divisions to sections in LHR a while back. The name seemed more natural. I'm going to shorten specification to the obvious spec.

December 28, 2010 | Unregistered CommenterKevlin Henney

Hey,

That's a very good job you've made here :)
I think I will use it for a personal iPhone project.

Anyway I would like to know how you will compare your library to Boost.Unit ?

Thanks :)

December 28, 2010 | Unregistered CommenterQuanteek

About fixtures: Notice that, in C++ at least, you don't need the old JUnit way of having setUp() and tearDown() fix up some test environment for you. A simple struct which you instanciate on the stack will do just fine, its constructor and destructor doing whatever setup/tear-down work need be done. You can instanciate that fixture in several test cases, and you have exactly the same effect as a JUnit TestSuite with its test methods.

December 28, 2010 | Unregistered CommenterCarl Seleborg

Hi, compiling the "stupid" example using -Wall -W gives us:

[acacio@dsv-bridge7-02 catch]$ g++ -Wall -W -I /usr/include/catch/ 01.cpp -o 01 2>&1 | wc -l
49

warnings. Aren't you compiling with these flags or you just don't mind about them at this stage of development?

December 28, 2010 | Unregistered CommenterAcácio

@Acácio
Yes I've been a bit lax about compiling with high warning levels. I'm now building with -Wall and -Wextra.
Fortunately I had a pull request from Michael Mortensen (SocketPuppet) with a load of fixes for warnings, and I applied a few more tweaks of my own to clear the last few up. For me it now compiles cleanly in GCC.

Last time I checked VC++ there were quite a few warnings I was going to get to. Hopefully most, if not all, have been cleaned up by this exercise now. When I can bear firing up Parallels again I'll check it out ;-)

Thanks for the feedback.

December 28, 2010 | Registered CommenterPhil Nash

@Carl
Yes, I have eschewed setup() and teardown() methods when using fixtures in favour of constructors and destructors (except for the Objective-C binding, which doesn't have RAII as an idiom - although it would work in this case).

Note that I support fixtures in several ways.

The simplest is to organically introduce a stack based object within test functions. This does add some extra boilerplate, however.

Next, in C++, you can write a class with test methods and manually register the methods (using METHOD_AS_TEST_CASE) as test cases. Instances of the class are created on the stack immediately prior to running each test method (I'll probably deprecate this method as the next one is superiour with no downsides, AFAICS).

A better variation is where you write a class to act as your fixture, but write the test cases outside the class body (using TEST_CASE_METHOD). The test cases are then self registering, but you can refer to the fixture data as member variables (each test case is actually implemented in a unique derived class).

FInally - my preferred way - is to write a test case with sections. The test case itself becomes the fixture and the sections become the test cases. There is still work to be done in this area (as alluded to in the blog post) - which is why the terminology starts to get a bit screwed here.

In Objective-C you can use SECTIONS or write fixtures as standard Objective-C classes - where each test case method is defined using the OC_TEST_CASE macro. Due to Objective-C's richer reflection support all the tests are self registering.

December 28, 2010 | Registered CommenterPhil Nash

@Quanteek
Thanks.
I believe CATCH improves on Boost.Test in having less boilerplate, more expressive assertions and no external dependencies (Boost.Test requires other parts of boost). It also has several other features that Boost.Test doesn't - such as SECTIONS and richer data generators.
That said Boost.Test is obviously more mature at this stage, and some features are more fully fleshed out. This will rectify over time, of course. In the longer run I don't see Boost.Test having any advantage over CATCH.

December 28, 2010 | Registered CommenterPhil Nash

@Kevlin
I suspected that might be the case - hence the "(or were)" qualification ;-)

December 28, 2010 | Registered CommenterPhil Nash

Obie, did you check out Cedar (http://github.com/pivotal/cedar) before or while writing this? It uses the new-ish block capabilities of Objective C to allow a syntax similar to Jasmine or Rspec. So, your first example could look something like this:

describe(@"one", ^{
it(@"should equal two", ^{
assertThat(@"one", equalTo(@"two"));
});
});

This example uses the popular Hamcrest matchers, as ported to Objective C by Jon Reid (http://github.com/jonreid/ochamcrest).

December 28, 2010 | Unregistered CommenterAdam Milligan

As usual, I fail at markdown. I suspect you can imagine the correct indenting in the previous example.

December 28, 2010 | Unregistered CommenterAdam Milligan

I created a new Obj-C project. Dropped in what I downloaded from git. #include "catch.hpp". Build.

Lots of errors.

I'm sure these are totally minor and the right C++ incantation of using namespace std; or whatever would fix, but I'm mostly just not interested in spending time on it.

Perhaps some hints (beyond the hyperbolic Step 1, Step 2, no Step 3) would be helpful to noobs like me.

Thanks

December 28, 2010 | Unregistered CommenterMe

This looks awesome in so many ways, thanks for sharing. Can't wait to test drive it out once I get back from holidays.

I love that it supports both C++ and Obj-C.

How does it integrate in an XCode/Obj-C workflow? Are there any special steps to take care of?

December 28, 2010 | Unregistered CommenterSteph Thirion

@Me (not me, me):

What sort of errors are you seeing?
You shouldn't need to do anything special (that's sort of the idea).
Did you have a look at the Test project under Test/XCode ?

I appreciate that I haven't yet put up any docs for the Objective-C bindings. I'll be getting to it (I've only just put up the start of some docs for the C++ usage - at https://github.com/philsquared/Catch/wiki/Tutorial, btw).

December 28, 2010 | Registered CommenterPhil Nash

@Me, et al

Regarding the Objective-C bindings: looks like I've left them short of a bit of polish. The test project works but has some boilerplate in it that I had intended to remove.
Will get my polishing brushes out now...

December 28, 2010 | Registered CommenterPhil Nash

It's a bit better now, but needs more.
Anyway, in short, you'll need one .mm file that #includes (or #imports) "catch_objc_main.hpp". Then each of your test classes should be implemented in .mm files that #include (or #import) "catch_objc.hpp".
You can supply optional setUp and tearDown methods. Test case methods are declared using OC_TEST_CASE, but are otherwise the same as TEST_CASE, as described above.

You don't have to use Objective-C classes, of course. You can still write free standing test case functions using TEST_CASE.

I'll write this up properly on the wiki shortly. I've started a stub for it here.

December 28, 2010 | Registered CommenterPhil Nash

JUnit is very much a product of Java

http://en.wikipedia.org/wiki/XUnit. JUnit is based on SUnit (for Smalltalk), so its not very much a product of Java.

December 29, 2010 | Unregistered CommenterYaseen Rauf

@Yaseen
That's not quite what I meant, but point taken. What I meant was that JUnit very much matches the spirit and idioms of Java. That's great for Java, and is probably why it dominates the unit testing scene there so much.

It's not so great when it is ported to a language with very different idioms, abilities and opportunities.

In particular, C++'s lack of reflection capability is usually just glossed over - leaving the user to do more manual work - and templates and operator overloading are often overlooked (usually for the sake of "keeping things simple" - which is fair enough - but if a sprinkling of these things can make life easier for the user - why not?).

December 29, 2010 | Registered CommenterPhil Nash

PostPost a New Comment

Enter your information below to add a new comment.
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>