« Unit Testing in C++ and Objective-C just got ridiculously easier still | Main | The iPad reconsidered »
Tuesday
Dec282010

Unit Testing in C++ and Objective-C just got easier

Day 133-365 : Catching the bokeh.jpg

Back in May I hinted that I was working on a unit testing framework for C++. Since then I've incorporated the technique that Kevlin Henney proposed and a whole lot more. I think it's about time I introduced it to the world:

[Update]
This post is very old now, but is still the first point of contact with Catch for many people. Most of the material here still applies in concept, so is worth reading - but some of the specifics have changes. Please see the tutorial (and other docs) over on GitHub for more up-to-date coverage.

Introducing CATCH

CATCH is a brand new unit testing framework for C, C++ and Objective-C. It stands for 'C++ AdaptiveAutomated Test Cases in Headers', although that shouldn't downplay the Objective-C bindings. In fact my initial motivation for starting it was dissatisfaction with OCUnit.

Why do we need another Unit Testing framework for C++ or Objective-C?

There are plenty of unit test frameworks for C++. Not so many for Objective-C - which primarily has OCUnit (although you could also coerce a C or C++ framework to do the job).

They all have their strengths and weaknesses. But most suffer from one or more of the following problems:

  • Most take their cues from JUnit, which is unfortunate as JUnit is very much a product of Java. The idiom-mismatch in C++ is, I believe, one of the reasons for the slow uptake of unit testing and TDD in C++.
  • Most require you to build libraries. This can be a turn off to anyone who wants to get up and running quickly - especially if you just want to try something out. This is especially true of exploratory TDD coding.
  • There is typically a certain amount of ceremony or boilerplate involved. Ironically the frameworks that try to be faithful to C++ idioms are often the worst culprits. Eschewing macros for the sake of purity is a great and noble goal - in application development. For a DSL for testing application code, especially since preprocessor information (e.g. file and line number) are required anyway) the extra verbosity seems too high a price to pay to me.
  • Some pull in external dependencies
  • Some involve a code generation step

The list goes on, but these are the criteria that really had me disappointed in what was out there, and I'm not the only one. But can these be overcome? Can we do even better if we start again without being shackled to the ghost of JUnit?

What's the CATCH?

You may well ask!

Well, to start, here's my three step process for getting up and running with CATCH:

  1. Download the headers from github into subfolder of your project
  2. #include "catch.hpp"
  3. There is no step 3!

Ok, you might need to actually write some tests as well. Let's have a look at how you might do that:

[Update: Since my original post I have made some small, interface breaking, changes - for example the name of the header included below. I have updated this post to reflect these changes - in case you were wondering]

#include "catch_with_main.hpp"

TEST_CASE( "stupid/1=2", "Prove that one equals 2" )
{
    int one = 2;
    REQUIRE( one == 2 );
}

Short and to the point, but this snippet already shows a lot of what's different about CATCH:

  • The assertion macro is REQUIRE( expression ), rather than the, now traditional, REQUIRE_EQUALS( lhs, rhs ), or similar. Don't worry - lhs and rhs are captured anyway - more on this later.
  • The test case is in the form of a free function. We could have made it a method, but we don't need to
  • We didn't name the function. We named the test case. This frees us from couching our names in legal C++ identifiers. We also provide a longer form description that serves as an active comment
  • Note, too, that the name is hierarchical (as would be more obvious with more test cases). The convention is, as you might expect, "root/branch1/branch2/.../leaf". This allows us to easily group test cases without having to explicitly create suites (although this can be done too).
  • There is no test context being passed in here (although it could have been hidden by the macro - it's not). This means that you can freely call helper functions that, themselves, contain REQUIRE() assertions, with no additional overhead. Even better - you can call into application code that calls back into test code. This is perfect for mocks and fakes.
  • We have not had to explicity register our test function anywhere. And by default, if no tests are specified on the command line, all (automatically registered) test cases are executed.
  • We even have a main() defined for us by virtue of #including "catch_with_main.hpp". If we just #include that in one dedicated cpp file we would #include "catch.hpp' in our test case files instead. We could also write our own main that drives things differently.

That's a lot of interesting stuff packed into just a few lines of test code. It's also got more wordy than I wanted. Let's take a bit more of a tour by example.

Information is power

Here's another contrived example:

TEST_CASE( "example/less than 7", "The number is less than 7" )
{
    int notThisOne = 7;

    for( int i=0; i < 7; ++i )
    {
        REQUIRE( notThisOne > i+1  );
    }
}

In this case the bug is in the test code - but that's just to make it self contained. Clearly the requirement will be broken for the last iteration of i. What information do we get when this test fails?

    notThisOne > i+1 failed for: 7 > 7

(We also get the file and line number, but they have been elided here for brevity). Note we get the original expression and the values of the lhs and rhs as they were at the point of failure. That's not bad, considering we wrote it as a complete expression. This is achieved through the magic of expression templates, which we won't go into the details of here (but feel free to look at the source - it's probably simpler than you think).

Most of the time this level of information is exactly what you need. However, to keep the use of expression templates to a minimum we only decompose the lhs and rhs. We don't decompose the value of i in this expression, for example. There may also be other relevant values that are not captured as part of the test expression.

In these cases it can be useful to log additional information. But then you only want to see that information in the event of a test failure. For this purpose we have the INFO() macro. Let's see how that would improve things:

TEST_CASE( "example/less than 7", "The number is less than 7" )
{
    int notThisOne = 7;

    for( int i=0; i < 7; ++i )
    {
        INFO( "i=" << i );
        REQUIRE( notThisOne > i+1  );
    }
}

This gives us:

    info: 'i=6'
    notThisOne > i+1 failed for: 7 > 7

But if we fix the test, say by making the for loop go to i < 6, we now see no output for this test case (although we can, optionally, see the output of successful tests too).

A SECTION on specifications

There are different approaches to unit testing that influence the way the tests are written. Each approach requires a subtle shift in features, terminology and emphasis. One approach is often associated with Behaviour Driven Development (BDD). This aims to present test code in a language neutral form - encouraging a style that reads more like a specification for the code under test.

While CATCH is not a dedicated BDD framework it offers a several features that make it attractive from a BDD perspective:

  • The hiding of function and method names, writing test names and descriptions in natural language
  • The automatic test registration and default main implementation eliminate boilerplate code that would otherwise be noise
  • Test data generators can be written in a language neutral way (not fully implemented at time of writing)
  • Test cases can be divided and subdivided into SECTIONs, which also take natural language names and descriptions.

We'll look at the test data generators another time. For now we'll look at the SECTION macro.

Here's an example (from the unit tests for CATCH itself):

TEST_CASE( "succeeding/Misc/Sections/nested", "nested SECTION tests" )
{
    int a = 1;
    int b = 2;
    
    SECTION( "s1", "doesn't equal" )
    {
        REQUIRE( a != b );
        REQUIRE( b != a );

        SECTION( "s2", "not equal" )
        {
            REQUIRE_FALSE( a == b);
        }
    }
}

Again, this is not a great example and it doesn't really show the BDD aspects. The important point here is that you can divide your test case up in a way that mirrors how you might divide a specification document up into sections with different headings. From a BDD point of view your SECTION descriptions would probably be your "should" statements.

There is more planned in this area. For example I'm considering offering a GIVEN() macro for defining instances of test data, which can then be logged.

In Kevlin Henney's LHR framework, mentioned in the opening link, he used SPECIFICATION where I have used TEST_CASE, and PROPOSITION for my top level SECTIONs. His equivalent of my nested SECTIONs are (or were) called DIVIDERs. All of the CATCH macro names are actually aliases for internal names and are defined in one file (catch.hpp). If it aids utility for BDD or other purposes, the names can be aliased differently simply by creating a new mapping file and using that.

CATCH up

There is much more to cover but I wanted to keep this short. I'll follow up with more. For now here's a (yet another) list of some of the key features I haven't already covered:

  • Entirely in headers
  • No external dependencies
  • Even test fixture classes and methods are self registering
  • Full Objective-C bindings
  • Failures (optionally) break into the interactive debugger, if available
  • Floating point tolerances supported in an easy to use way
  • Several reporter classes included - including a JUnit compatible xml reporter. More can be supplied

Are there any features that you feel are missing from other frameworks that you'd like to see in CATCH? Let me know - it's not too late. There are some limiting design goals - but within those there are lots of possibilities!

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (70)

Thanks for the tool and helpful blog post! I'm interested in your framework and leaning towards using it in my own development after experimenting this weekend. Is this still under active development? What is the overall interest and uptake? Do you have examples of using CATCH for Obj-C as well? What of the matchers? I saw a reference for assert macros which were designed for matchers, where are these implemented? Are they of the Hamcrest style? Very good work, thanks again!

April 8, 2013 | Unregistered CommenterCliff

Hi Cliff,

CATCH is still very much in active development!
Please see the Google Group at https://groups.google.com/forum/?fromgroups#!forum/catch-forum for discussion. Anything I write here is likely to be out of date quite quickly.
Briefly, though, at present Matchers are still at an early stage. They are somewhat similar to Hamcrest.

There are some basic Objective-C examples in the OCTest project in the CATCH github repos (catch-lib.net).

Both of these are on my radar to be fleshed out in the near-to-mid-term.

The interest and uptake has been much more than I had expected. I don't know real figures as there is no way to track, but extrapolating from the rate of queries I get, and the number of watchers on GitHub, I imagine the active user base must be in the 1000s at this point. Still plenty of room to really take off, of course.
It's being used in some very interesting places, especially many of the banks - and I know that Bloomberg have company-wide approval to use it.

April 8, 2013 | Registered CommenterPhil Nash

HI there,

I'm shopping around for a C++ unit testing framework, and till now I like CATCH the most by far. I t deserves to be in the public mindset.

I'm trying to determine how to use it within Eclipse. Could you give pointers, if at all possible, as to how to integrate CATCH within Eclipse?

April 17, 2013 | Unregistered CommenterWim Rijnders

Like the approach you've taken and I'm going to use this for testing my own work.

A couple of comments:

1. the self-tests fail because of a bug in the TestCaseFilter constructor (shows up on VS2010 - suspect its because of checked-iterators)
2. you expect 289 successful tests, but only run 264

Fix those up and it tests clean.

I've submitted an issue (with attached fix) on github for point 1.

Thanks for all the great work.

K

May 24, 2013 | Unregistered CommenterKevin Shea

Phil:
We want to implement some TDD approaches to our software development cycle and CATCH framework attracts us. In fact we are trying to use this framework for unit testing in our development. Actually we are unable to configure the CATCH in our iOS applications. it gives lots of header path errors. there are two main tutorials available on the internet regarding configuring CATCH in iOS but both are not working for US. Please guide us is there any proper document that describes step-by-setp guide to integrate CATCH.

June 12, 2013 | Unregistered CommenterAdnan Nasir

Best c++ testing framework I have tried to date. Period. And by a considerable difference to the others. The naming can be nice and natural, it supports BDD and it's really easy to setup.


The other two candidates for second and third place are Boost.Test and Google Test.

Google Test is a pain in the a*se in windows, with all the ABI-incompatible stuff for each configuration. It gave me several headaches in combination with CMake.

Boost.Test at least can be compiled in header-only mode and is a bit nicer than Google test in my opinion. That Setup/TearDown weirdness, since there are constructors and destructors in c++, forces me to use pointers for some classes when in Boost.Test I can use just normal automatic variables.

For now I use turtle mock for mocking. It's nice. Maybe some integration into catch would be nice, though it took me little work to do it myself at least for showing messages, but not for error reporting. I don't know how to do it.

May 29, 2014 | Unregistered CommenterGerman Diago

I concur with German Diago. I've been using the framework for over a year now and I think it's time I should say I'm very satisfied with it.

When I started my current project, I took the time to examine the various testing frameworks available. CATCH was a relative newcomer and not widespread at all, but it made the most sense to me. I took the plunge and I must say the experience is good. I don't feel I have to fight it to get it to work; put in another way it doesn't get in the way of my workflow. CATCH deserves more recognition.

May 29, 2014 | Unregistered CommenterWim Rijnders

@German and @Wim - thanks! It always means a lot to hear when people appreciate Catch. I'm particularly glad you've picked up on all the things that I (and others) have worked hard to make key parts of the Catch experience.

@German - re: mocking: that's something I really need to spend time with myself. I'm not a big mockist in general - and rarely use them in C++ (more so in Objective-C, where the frameworks I've used have integrated nicely with Catch). I've heard that HippoMocks works well.

June 2, 2014 | Registered CommenterPhil Nash

Oh wow, your REQUIRE macro is a work of art. Reporting on the *values* of both sides of an arbitrary relational expression is something I would have sworn to be impossible in C++. Then I read catch.hpp:1574.

Enlightenment was left associated, if slow to come.

July 4, 2015 | Unregistered CommenterChris Foster

Thanks Chris,

The original idea was Kevlin Henney's.
You might also want to see my NDC Oslo talk where I break down the whole process: https://vimeo.com/131632252

July 4, 2015 | Registered CommenterPhil Nash

PostPost a New Comment

Enter your information below to add a new comment.
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>