« Unit Testing in C++ and Objective-C just got ridiculously easier still | Main | The iPad reconsidered »
Tuesday
Dec282010

Unit Testing in C++ and Objective-C just got easier

Day 133-365 : Catching the bokeh.jpg

Back in May I hinted that I was working on a unit testing framework for C++. Since then I've incorporated the technique that Kevlin Henney proposed and a whole lot more. I think it's about time I introduced it to the world:

[Update]
This post is very old now, but is still the first point of contact with Catch for many people. Most of the material here still applies in concept, so is worth reading - but some of the specifics have changes. Please see the tutorial (and other docs) over on GitHub for more up-to-date coverage.

Introducing CATCH

CATCH is a brand new unit testing framework for C, C++ and Objective-C. It stands for 'C++ AdaptiveAutomated Test Cases in Headers', although that shouldn't downplay the Objective-C bindings. In fact my initial motivation for starting it was dissatisfaction with OCUnit.

Why do we need another Unit Testing framework for C++ or Objective-C?

There are plenty of unit test frameworks for C++. Not so many for Objective-C - which primarily has OCUnit (although you could also coerce a C or C++ framework to do the job).

They all have their strengths and weaknesses. But most suffer from one or more of the following problems:

  • Most take their cues from JUnit, which is unfortunate as JUnit is very much a product of Java. The idiom-mismatch in C++ is, I believe, one of the reasons for the slow uptake of unit testing and TDD in C++.
  • Most require you to build libraries. This can be a turn off to anyone who wants to get up and running quickly - especially if you just want to try something out. This is especially true of exploratory TDD coding.
  • There is typically a certain amount of ceremony or boilerplate involved. Ironically the frameworks that try to be faithful to C++ idioms are often the worst culprits. Eschewing macros for the sake of purity is a great and noble goal - in application development. For a DSL for testing application code, especially since preprocessor information (e.g. file and line number) are required anyway) the extra verbosity seems too high a price to pay to me.
  • Some pull in external dependencies
  • Some involve a code generation step

The list goes on, but these are the criteria that really had me disappointed in what was out there, and I'm not the only one. But can these be overcome? Can we do even better if we start again without being shackled to the ghost of JUnit?

What's the CATCH?

You may well ask!

Well, to start, here's my three step process for getting up and running with CATCH:

  1. Download the headers from github into subfolder of your project
  2. #include "catch.hpp"
  3. There is no step 3!

Ok, you might need to actually write some tests as well. Let's have a look at how you might do that:

[Update: Since my original post I have made some small, interface breaking, changes - for example the name of the header included below. I have updated this post to reflect these changes - in case you were wondering]

#include "catch_with_main.hpp"

TEST_CASE( "stupid/1=2", "Prove that one equals 2" )
{
    int one = 2;
    REQUIRE( one == 2 );
}

Short and to the point, but this snippet already shows a lot of what's different about CATCH:

  • The assertion macro is REQUIRE( expression ), rather than the, now traditional, REQUIRE_EQUALS( lhs, rhs ), or similar. Don't worry - lhs and rhs are captured anyway - more on this later.
  • The test case is in the form of a free function. We could have made it a method, but we don't need to
  • We didn't name the function. We named the test case. This frees us from couching our names in legal C++ identifiers. We also provide a longer form description that serves as an active comment
  • Note, too, that the name is hierarchical (as would be more obvious with more test cases). The convention is, as you might expect, "root/branch1/branch2/.../leaf". This allows us to easily group test cases without having to explicitly create suites (although this can be done too).
  • There is no test context being passed in here (although it could have been hidden by the macro - it's not). This means that you can freely call helper functions that, themselves, contain REQUIRE() assertions, with no additional overhead. Even better - you can call into application code that calls back into test code. This is perfect for mocks and fakes.
  • We have not had to explicity register our test function anywhere. And by default, if no tests are specified on the command line, all (automatically registered) test cases are executed.
  • We even have a main() defined for us by virtue of #including "catch_with_main.hpp". If we just #include that in one dedicated cpp file we would #include "catch.hpp' in our test case files instead. We could also write our own main that drives things differently.

That's a lot of interesting stuff packed into just a few lines of test code. It's also got more wordy than I wanted. Let's take a bit more of a tour by example.

Information is power

Here's another contrived example:

TEST_CASE( "example/less than 7", "The number is less than 7" )
{
    int notThisOne = 7;

    for( int i=0; i < 7; ++i )
    {
        REQUIRE( notThisOne > i+1  );
    }
}

In this case the bug is in the test code - but that's just to make it self contained. Clearly the requirement will be broken for the last iteration of i. What information do we get when this test fails?

    notThisOne > i+1 failed for: 7 > 7

(We also get the file and line number, but they have been elided here for brevity). Note we get the original expression and the values of the lhs and rhs as they were at the point of failure. That's not bad, considering we wrote it as a complete expression. This is achieved through the magic of expression templates, which we won't go into the details of here (but feel free to look at the source - it's probably simpler than you think).

Most of the time this level of information is exactly what you need. However, to keep the use of expression templates to a minimum we only decompose the lhs and rhs. We don't decompose the value of i in this expression, for example. There may also be other relevant values that are not captured as part of the test expression.

In these cases it can be useful to log additional information. But then you only want to see that information in the event of a test failure. For this purpose we have the INFO() macro. Let's see how that would improve things:

TEST_CASE( "example/less than 7", "The number is less than 7" )
{
    int notThisOne = 7;

    for( int i=0; i < 7; ++i )
    {
        INFO( "i=" << i );
        REQUIRE( notThisOne > i+1  );
    }
}

This gives us:

    info: 'i=6'
    notThisOne > i+1 failed for: 7 > 7

But if we fix the test, say by making the for loop go to i < 6, we now see no output for this test case (although we can, optionally, see the output of successful tests too).

A SECTION on specifications

There are different approaches to unit testing that influence the way the tests are written. Each approach requires a subtle shift in features, terminology and emphasis. One approach is often associated with Behaviour Driven Development (BDD). This aims to present test code in a language neutral form - encouraging a style that reads more like a specification for the code under test.

While CATCH is not a dedicated BDD framework it offers a several features that make it attractive from a BDD perspective:

  • The hiding of function and method names, writing test names and descriptions in natural language
  • The automatic test registration and default main implementation eliminate boilerplate code that would otherwise be noise
  • Test data generators can be written in a language neutral way (not fully implemented at time of writing)
  • Test cases can be divided and subdivided into SECTIONs, which also take natural language names and descriptions.

We'll look at the test data generators another time. For now we'll look at the SECTION macro.

Here's an example (from the unit tests for CATCH itself):

TEST_CASE( "succeeding/Misc/Sections/nested", "nested SECTION tests" )
{
    int a = 1;
    int b = 2;
    
    SECTION( "s1", "doesn't equal" )
    {
        REQUIRE( a != b );
        REQUIRE( b != a );

        SECTION( "s2", "not equal" )
        {
            REQUIRE_FALSE( a == b);
        }
    }
}

Again, this is not a great example and it doesn't really show the BDD aspects. The important point here is that you can divide your test case up in a way that mirrors how you might divide a specification document up into sections with different headings. From a BDD point of view your SECTION descriptions would probably be your "should" statements.

There is more planned in this area. For example I'm considering offering a GIVEN() macro for defining instances of test data, which can then be logged.

In Kevlin Henney's LHR framework, mentioned in the opening link, he used SPECIFICATION where I have used TEST_CASE, and PROPOSITION for my top level SECTIONs. His equivalent of my nested SECTIONs are (or were) called DIVIDERs. All of the CATCH macro names are actually aliases for internal names and are defined in one file (catch.hpp). If it aids utility for BDD or other purposes, the names can be aliased differently simply by creating a new mapping file and using that.

CATCH up

There is much more to cover but I wanted to keep this short. I'll follow up with more. For now here's a (yet another) list of some of the key features I haven't already covered:

  • Entirely in headers
  • No external dependencies
  • Even test fixture classes and methods are self registering
  • Full Objective-C bindings
  • Failures (optionally) break into the interactive debugger, if available
  • Floating point tolerances supported in an easy to use way
  • Several reporter classes included - including a JUnit compatible xml reporter. More can be supplied

Are there any features that you feel are missing from other frameworks that you'd like to see in CATCH? Let me know - it's not too late. There are some limiting design goals - but within those there are lots of possibilities!

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (68)

Ok, antoher question. :)

I assume it's intentional that Catch swallows up the standard output? Is there an option to disable it? Or do you have a macro for just printing information unconditionally (not dependent on test failures)

March 11, 2011 | Unregistered Commenterjalf

It's intentional, yes. The JUnit reporter dumps in as part of its output. I should probably add it to the other reporters too.
In the meantime I believe the WARN() macro will do what you want - although that's obviously a workaround rather than a solution.

March 12, 2011 | Registered CommenterPhil Nash

What's the rationale behind that decision?

I guess it tidies things up when you're just running your tests, but when a test fails, and you want to debug your code, getting your output directly and immediately can be pretty vital in some cases, Any chance of an optional flag to disable this behavior?

March 13, 2011 | Unregistered Commenterjalf

I had a quick look and it should have been dumping the outputs during the reporting but a bug means it only happens in the event of failure. I've fixed that on my branch and will try to merge it across in the next day or so.

March 15, 2011 | Registered CommenterPhil Nash

Has anyone tried Catch on Android with the Android NDK? I may need to test C++ code that runs on both iOS and Android. I actually don't have experience with the Android development at this time, but my understanding is that there are some C++ limitations on that platform.

Thanks,
David

April 19, 2011 | Unregistered CommenterDavid Franzen

I'm surprised nobody else has mentioned this, but the number of unscoped names injected by this framework is extremely problematic. I can understand using `TEST_CASE` instead of the better-scoped `CATCH_TEST_CASE` on the basis of the assumption that there will be only one unit testing framework. Names like `CHECK` and `REQUIRE` are more troubling, with the potential defense that short names are important for the bread and butter of testing. The names `SECTION` and `GENERATE` aren't obviously constrained to the unit testing domain, and conflicts with other macro libraries are entirely possible.

Worst, the main header also creates macros called `INFO` and `WARN`, which makes using this framework a complete nonstarter for me. These are names commonly used by logging frameworks, and conflicts are not just possibly but extremely likely.

It's worth noting that conflicts between real function and class names are annoying but (usually) detectable: the compiler will detect incompatible declarations and the linker will find multiple definitions. Conflicts between preprocessor macros create silent bugs based on inclusion ordering.

Otherwise, it looks like a really good library---I just wish this issue with the names were addressed!

June 20, 2011 | Unregistered CommenterRob

Hi Rob,

I should really do a post on that - or at least a wiki entry.
The idea is that the catch.hpp header is a top-level header that maps prefixed names to "friendly" names. Obviously conflicts will occur from time to time. The intention here is that it's trivial to define your own version of the top-level header that uses different names. I even allow the top-level macro name to be passed down to the lower-level macros so that the name used can be reported in the event of failure.

Admittedly I haven't kept this as straightfoward as it should be (there are a lot of #includes at the top level, and the single-header version includes the top level as well). I shall look at this again shortly.
As you say, you're the first one to mention it - so it's dropped off my immediate sights for a while.

You're right that INFO and WARN are a little *too* generic and common - more likely to have conflicts.
As it happens I'm going through a bit of an overhaul of some of the top-level naming at the moment. Thanks for bringing this up - I'll get to address it at the same time.

The other intention I had, which I've not followed through on as yet, is to provide a couple of alternate naming schemes out of the box (ie, just include a different top-level header).

June 21, 2011 | Registered CommenterPhil Nash

Does this integrate well with rpmbuild/git/cruisecontrol?

How can I help this project?

November 8, 2011 | Unregistered Commenterjdonson

@jdonson I've been using it with CruiseControl.Net. I wrote a custom XSLT to render the XML reporter. Unfortunately the XSLT itself belongs to my current "employer" - but it's not terribly involved.

Alternatively there is a Junit XML reporter which CC/.Net can supposedly use directly.
I say "supposedly" because I have not had a chance to try it out yet. I believe there is an issue logged against it on GitHub so it may need some work.

I'm not quite sure what you mean by git integration?

As for rpmbuild, I have no experience of that - but in any of these cases (or, indeed, any others) you would be very welcome to help out.

You can take your own fork from the github master: catch-lib.net and feed back pull requests - or we could discuss other forms of collaboration if you prefer.

My personal input has stalled a bit recently as I've had to take on addition projects - but hope to get back onto it in the next few weeks.

I'll drop you an email to discuss further.

November 8, 2011 | Registered CommenterPhil Nash

I'm getting an error of "redefinition of ‘Catch::AutoReg <unnamed>::autoRegistrar6’" (with both Intel icpc and g++-4.2) when building a project structured as follows:

main.cpp:

#define CATCH_CONFIG_MAIN

#include "Catch.hpp"

#include "TestsA.h"
#include "TestsB.h"

TestsA.h contains, for example:

#ifndef __TESTS_A_H__
#define __TESTS_A_H__

TEST_CASE( "some_test", "just some text" )
{
REQUIRE( 0 == 0 );
REQUIRE( 1 == 1 );
}

#endif

TestB.h is similar.

I'm building my code as: g++-4.2 main.cpp or icpc main.cpp

Am I doing something incorrect?

Thanks!

February 27, 2012 | Unregistered CommenterBob the Builder

@Bob the Builder

Short answer: You should put your tests in a .cpp file rather than a header.

Longer answer:
Each TEST_CASE generates an instance of a static initialiser object (AutoReg) that must have a unique name. The name is generated by combining a common root (autoRegistrar) with the line number (6, in this case) - which should be unique within the current file (unless you write multiple test cases on one line).
To avoid clashes with test cases on the same line in another file the registrar instance is declared within an anonymous namespace - which causes it have internal linkage only. So different *translation units* may contain files that contain test cases on lines the same as in another translation unit. This is usually all you need as you typically only define your tests cases in a .cpp file - each of which are in a different translation unit.


Is there any reason you needed to write have these tests in headers?

February 27, 2012 | Registered CommenterPhil Nash

I have only recently started unit testing in my XCode projects. One problem that I've come across, is that it seems the macros available to test for bad argument handling in OCUnit seems to expect that bad arguments are handled with an objective-c exception. This is only inferring from the Apple docs, though. Currently I'm just handling bad args in my c++ and objective-c++ code with a standard c assert, I haven't had a problem with that, because production quality code shouldn't call a function with bad args anyway. I admit it's a bit crude...... anyway, when unit testing you're supposed to call functions with bad args and verify that the function is complaining. Does your framework have a way to test bad argument handling? Something like REQUIRE_EXCEPTION or something? It must be an exception type that can be used in both c++ and objective-c++ code.

Anyway, I really like your philosophy for unit testing much better than the OCUnit way.

April 3, 2012 | Unregistered CommenterBalthazar

@Balthazar
Excellent point!
CATCH does catch C++ exceptions. It should catch Obj-C exceptions too, but I just checked and see that it doesn't. I'll have to fix that (I can't remember OTTOMH if Obj-C exceptions can be caught in a C++ catch block - it's possible this "just works" already).
As for asserts - that can be trickier. Assert usually just calls terminate. On some platforms it may be interceptable. I'm not sure about Mac/ iOS. Again will have to check.
I've had it on my list for a while to look at catching seg faults/ access violations where possible - this may play into that.
Thanks for prodding me.

April 3, 2012 | Registered CommenterPhil Nash

Hi

That's great! I just checked that out. It seems on 64 bit, objective-c and c++ exceptions are interchangeable. On 32 bit, they don't mix. I needed only c++ exceptions, because my project has some pure c++ and some objective-c++, no pure objc. So I'm fine. It seems you can catch an exception in an obj-c method compiled as objective-c++, as long as you use c++ syntax. I need to double check that. But there seems to be some problemos on 32 bit, if you have a stack frame of objective-c between your c++ function that wants to catch an exception, and the c++ function that throws an exception:

http://www.cocoabuilder.com/archive/cocoa/212027-exceptions-in-objective-call-stack.html

But I'll just stick to c++ exceptions. I see there's no way a testing framework can handle the standard c assert.... :)

April 3, 2012 | Unregistered CommenterBalthazar

Oh, about this sentence: "It seems you can catch an exception in an obj-c method compiled as objective-c++, as long as you use c++ syntax", that's for 32 bit. For 64 bit you can catch a c++ exception with objective-c syntax and vice versa. If I got it right. Don't take my word for it... :)

April 3, 2012 | Unregistered CommenterBalthazar

Hi again...

Sorry for "spamming" your list, but I can't get it to compile. I have made, in my XCode 4 project, a new target of type command line tool, of subtype Foundation. I have made a completely empty file called main.mm with only two lines:

#import <Foundation/Foundation.h>
#import "catch_with_main.hpp"

When I try to compile, I get an error message, saying "use of undeclared identifier __unsafe_unretained in file catch_objc.hpp, line 174. That is with the LLVM Compiler 2.0. With gcc instead I get "'__unsafe_unretained' was not declared in this scope.

April 3, 2012 | Unregistered CommenterBalthazar

@Balthazar. Looks like it thinks ARC is available when it is not, for some reason.
Could I ask you to post your issue to the issue tracker on GitHub, and specify which version of Xcode too?

https://github.com/philsquared/Catch/issues

Thanks for bringing this up.

WRT catching exceptions make sure you are familiar with the appropriate macros here: https://github.com/philsquared/Catch/wiki/Assertion-Macros

April 4, 2012 | Registered CommenterPhil Nash

Hi!

I have been busy, but will post the issue soon. Currently I only need c++ which works fine. I love the approach to test case setup, which makes me able to write my test cases as specification/documentation... :)

April 5, 2012 | Unregistered CommenterBalthazar

OK, I have posted the issue now... :)

April 6, 2012 | Unregistered CommenterBalthazar

Hi!
I've been looking at CATCH for a while and decided to try it out on one of my private projects. I really liked it! There is, however, one thing that I would like to request (maybe it's already implemented and I simply couldn't figure out how to do it...) Anywhat here goes:

I've read that you encourage the use of test fixtures by doing something along the these lines:

class Fixture { ... };
TEST_CASE("test/1", "...") {
Fixture fixture;
// code for test 1.
}
TEST_CASE("test/2", "...") {
Fixture fixture;
// code for test 2.
}

Correct? Anyway, I personally feel that doing test fixtures like this is a bit cumbersome because of how C++ forces the programmer to declare member variables at one place (the class body) and initialize them at another place (the initialization list in the constructor).
So this bring me to my request: would it be possible to implement the fixture as part of the function body instead of in a seperate class? I'm thinking something like this:

FIXTURE("...", "...") {
// This is a function body -- similar to how TEST_CASE works.
int a = 0; // Variables that are part of the fixture is delcared just like local variables.
std::string s = "hello";
FIXTURE_TEST_CASE("...", "...") { // The actual testcase is implemented like this -- similar to SECTION.
// Code for the test-case goes here.
// When this test-case is done, the fixture is teared down (because the function is returning).
}

FIXTURE_TEST_CASE("...", "...") { // This is the second test-case.
// Code for the test-case goes here.
// When this test-case is done, the fixture is teared down (because the function is returning).
}
}

As far as I undestand SECTION, FIXTURE_TEST_CASE would be very similar -- the only difference would be that the function return between each FIXTURE_TEST_CASE.

What do you think?

September 18, 2012 | Unregistered CommenterTorgny

PostPost a New Comment

Enter your information below to add a new comment.
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>