« Could the Internet please stop changing while I finish this blog post? | Main | Unit Testing in C++ and Objective-C just got easier »
Friday
May272011

Unit Testing in C++ and Objective-C just got ridiculously easier still

Spider web in morning sun

'Spider Web in Morning Sun' by Rob van Hilten

In my previous post I introduced Catch - my unit testing framework for C++ and Objective-C.

The response was overwhelming. Thanks to all who commented, offered support - and even contributed to the code with fixes and features.

It certainly gave me the motivation to continue active development and a lot has changed since that post. I'm going to cover some highlights, but first I want to focus on what has been one of the most distinguishing features of Catch that has attracted so much attention - and how I have not rested but made that even better!

How easy is easy enough?

Back in April I gave a five minute lightning talk on Catch at the ACCU conference in Oxford (I highly recommend the conference). With just five minutes to talk about what makes Catch special what was I going to cover? The natural operator-based comparison syntax? The use of Sections instead of class-based fixtures? Data generators?

Well I did touch on the first point. But I decided to use the short amount of time to drive home just how quickly and easily you can get up and running with Catch. So after a 30 second intro I went to the GitHub page for Catch (now aliased as catch-test.net), downloaded the zip of the source (over a 3G connection), unzipped and copied to a central location, fired up XCode, started a fresh C++ project, added the path to Catch's headers, #include'd "catch_with_main.hpp", wrote an anonymous test case, compiled and ran it, demonstrated how it caught a bug, fixed the bug and finally recompiled and re-ran to see the bug go away.

Phew! Not bad for five minutes, I thought. And from the feedback I got afterwards it really did drive the point home.

Compare that with my first experience of using Google Test. It took me over an hour to get it downloaded and building in XCode (the XCode projects don't seem to have been maintained recently - so perhaps that is a little unfair). There are other frameworks that I've tried where I have just run out of patience and never got them going.

Of course I'm biased. But I have had several people tell me that they tried Catch and found it to be the easiest C++ Unit Test framework they have used.

But still I wasn't completely satisfied with the initial experience and ease of incorporating Catch into your own projects.

In particular, if you maintain your own open source project and want to bundle it with a set of unit tests (and why wouldn't you?) then it starts to get fiddly. Do you list Catch as an external dependency that the user must install on their own? (no matter how easy they are to install external dependencies are one or my least favourite things). Do you include all the source to Catch directly in your project tree? That can get awkward to maintain and makes it look like your project is much bigger than it is. If you host your project on GitHub too (or some other Git based repository) you could include Catch as a submodule. That's still not ideal, has some of the problems of the first two options, and is not possible for everyone.

There can be only one

Since Catch, as a library, is fully header-only I decided provided a single header version that is ideal for direction inclusion in third-party projects.

How did I do this?

Go on guess.

Did you guess that I wrote a simple Python script to partially preprocess the headers so that the #includes within the library are expanded out (just once, of course), leaving the rest untouched?

If you did you're not far off. Fortunately some of the conventions I have used within the source meant I could drastically simplify the script. It doesn't need to be a full C preprocessor. It only needs to understand #include and #ifndef/#endif for include guards. Even those are simplified. The whole script is just 42 lines of code. 42 always seems to be the answer.

The result is https://github.com/philsquared/Catch/blob/master/single_include/catch.hpp

I see no reason why this should not be the default way to use Catch - unless you are developing Catch itself. So I'm now providing this file as a separate download from within GitHub. Think of it as the "compiled" header. The lib file of the header-only world.

Licence To Catch

But Open Source is a quagmire of licensing issues, isn't it?

Well it certainly can be. Those familiar with GPL and similar open source licences may be very wary of embedding one open source library (Catch) within another (their own).

IANAL but my understanding is that, contrary to what might seem intuitive, source code with no license at all can be more dangerous, legally speaking, than if it does have one (and if you thought that sentence was difficult to parse you should try reading a software license).

So Catch is licensed. I've used the Boost license. For a number of reasons:

  • It is very permissive. In particular it is not viral. It explicitly allows the case of including the source of Catch along with the distribution of your own source code with no requirements on your own code
  • It's been around for a while now - long enough, I think, that most people are comfortable with it. I work with banks, who can be very nervous about software licensing issues - especially open source. But every one I have worked at has already got Boost through it's compliance process. I'm hoping that will ease any barriers to adoption.
  • I'm familiar with Boost, know many of it's contributors personally, and generally trust the spirit of the licence. Boost itself is a very well known and highly respected set of libraries - with very widespread adoption. A large part of Boost is in header-only libraries and people are already comfortable including them in their own projects.

So what's the Catch? The catch is that I retain the right to keep using that joke - well beyond its humorous lifetime.

The important bit:

In short: any open source author who wants to use Catch to write unit tests for their own projects should feel very free to do so and to include the single-header (or full) version of the library in their own repository and along with their distribution.

That fully applies to commercial projects too, of course.

What else?

Here's a quick run down of some of the other changes and features that have gone in:
  • Single evaluation of test expressions. The original implementation evaluated the expression being tested twice - once to get the result, and then again to get the component values. There were some obstacles to getting this to work whilst only evaluating the expression once. But we got there in the end. This is critical if you want to write test expressions that have side-effects.
  • Anonymous test cases. A little thing, but I find them really handy when starting a new project or component and I'm just exploring the space. The idea is that you don't need to think of a name and description for your test - you can just dive straight in and write code. If you end up with something more like a test case it's trivial to go back and name it.
  • Generators. These are in but not fully tested yet. Consider them experimental - but they are very cool and very powerful.
  • Custom exception handlers. (C++) Supply handlers for your own exception types - even those that don't derive from std::exception, so you can report as much detail as you like when an exception is caught within Catch. I'm especially pleased this went in - given the name of the library!
  • Low build time overhead. I've been aggressive at keeping the compile-time footprint to a minimum. This is one of the concerns when using header only libraries - especially those with a lot of C++ templates. Catch uses a fair bit of templates, but nothing too deeply recursive. I've also organised the code so that as much as the implementation as possible is included in only one translation unit (the one with main() or the test runner). I think you'll be pushed to notice any build-time overhead due to Catch.
  • Many fixes, refactorings and minor improvements. What project doesn't have them? This is where a lot of the effort - possibly the majority - has gone, though. I've wanted to keep the code clean, well factored, and the overhead low. I've also wanted it to be possible to compile at high warning levels without any noise from Catch. This has been challenging at times - especially after the Single Evaluation work. If you see any Catch-related warnings please let me know.

Are we there yet?

As well as my own projects I've been using Catch on a large scale project for a bank. I believe it is already more than just a viable alternative to other frameworks.

Of course it will continue to be refined. There are still bugs being found and fixed.

But there are also more features to be added! I need to finish the work on generators. I'd like to add the tagging system I've mentioned before. I need to look at Matchers. Whether Catch provides its own, or whether I just provide the hooks for a third-party library to be integrated, I think Matchers are an important aspect to unit testing.

I also have a stub project for an iPhone test runner - for testing code on an iOS device. Several people have expressed an interest in this so that is also on my list.

And, yes, I will fill out the documentation!

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (22)

Hey Phil,

I thought I'd drop by and say thanks for the wonderful work you did with CATCH (I also posted a couple of issues on your git repo :) since I use it quite a bit and find it the easiest testing framework I have had the pleasure of working with. Keep up the good work!

- Sam

June 19, 2011 | Unregistered Commentersamaursa

Thanks for the feedback, Sam - and for the issues (which I believe I resolved ;-) ).

June 21, 2011 | Registered CommenterPhil Nash

Phil,

This is a great framework. Its VERY easy to get everything rolling. I'm considering contributing a reporter that prints out a little more information than the basic one. I'll be forking soon and hopefully making time to work on it. ;)

Jurnell

June 24, 2011 | Unregistered CommenterJurnell

Thanks @Jurnell. What sort of information are you looking for? Updating the basic reporter is on my list anyway - I may be able to accomodate your needs there.

June 29, 2011 | Registered CommenterPhil Nash

@Phil,

Well, I was looking to get a reporter to print out the # of passed tests per test case.

For example (assuming there's 3 test case with 2 tests each):

Test case ("some/case/name/here1") ............... [2/2]
Test case ("some/case/name/here2") ............... [1/2]
** Failed Test Info here **
Test case ("some/case/name/here3") ............... [2/2]

[Testing completed. 5 test(s) passed but 1 test(s) failed in 1 case(s)]

`Failed test info` would be the normal print out provided already provided by the basic reporter for a failed test:
filename:line#: `test expression` failed for `blah`

It's a bit more verbose, but I don't know if everyone would like that. In fact, if we could even allow for an extra flag for the basic reporter to spit out additional information, that would be fine.

Thanks,

Jurnell

July 2, 2011 | Unregistered CommenterJurnell

Isn't it about time the Wiki is updated? It still says that "Currently CATCH should be considered a "developer preview", and "I wouldn't recommend it for commercial projects".

From this blog post, it sounds like you consider it a lot more robust and usable than the wiki implies.

Also, it's possible that this is documented somewhere, but I haven't found it. If I use the single-header version, how do I indicate which translation unit should contain the main function? I assume I need to define some macro in one of the translation units?

July 8, 2011 | Unregistered Commenterjalf

@jalf Good point. I appreciate the apparent discrepency.

The way I see it Catch has proved itself pretty stable and I'm very close to removing the "developer preview" status. I'm holding off just a bit longer as there's a couple of things I want to complete first (fix some issues with SECTIONs - and adding support for tags - which will involve an overhaul of the TEST_CASE and SECTION macros).

As for the single-header main - I did add that to the wiki 2-3 weeks ago: https://github.com/philsquared/Catch/wiki/Tutorial
(#define CATCH_CONFIG_MAIN)

More and better docs are another thing that is really needed to be comfortable opening CATCH up to a wider audience.

July 8, 2011 | Registered CommenterPhil Nash

@Jurnell sorry I missed your reply before.

I intend to improve the basic reporter to give a clearer distinction between number test *cases* (and sections) that have passed and test *assertions*. Currently the count of tests is really the count of assertions, which may not be what you expect.

That's not really what your asking though. Currently running with -s will give you the count of tests passed even if there are no failures - but will also show you the results of all the individual assertions - which is probably too verbose.

Sounds like you want a quieter, but still more verbose, mode. Shouldn't be too hard to accomodate that.

July 8, 2011 | Registered CommenterPhil Nash

Very interesting framework. Is there any instruction for the cases where 'main' does exist? And... Is it possible to control the tests from it (at least to run or not to run the tests)? Is it possible to inject our own stream class for Catch output?

September 26, 2011 | Unregistered Commenterdrom

I have used it for a while now, and it is great.

However, are there any built in fuzzy equality macros? It's very easy to make, I know, I just thought it would be good for convenience completeness. Thanks for a great package! :)

May 27, 2012 | Unregistered CommenterBalthazar

Catch is too easy to use.
I start using this in my current personal project.
The command line options are very useful also!

Thanks for you really great work!

June 30, 2012 | Unregistered Commentermmazzei

Hey,

I appreciate this little tiny swiss knife! Thanks a lot for implementing it.

One wish for future development: If SECTIONs are used, could you modify the output in case of an error. I would like to see, in what section my error actually appears.

Thanks

August 28, 2012 | Unregistered CommenterSebastian

Hi @Sebastian,

Thanks for the feedback.
That's odd about the sections. That is the way it is meant to work - and has done since the start.
However I've also noticed, when using on a client project, that they sometimes don't get logged. Not a chance to triage it from that code-base yet.
But in my self-tests they always come out - and from the code I can't see how they could not!

If you managed to capture a self-contained example to reproduce I'd be grateful if you could post it to the issues list on GitHub at http://catch-lib.net

Thanks

August 28, 2012 | Registered CommenterPhil Nash

You are right. They come out. But I would like to have theme ordered similar to the names f the test cases themselves.

If I have something like
TEST_CASE("1/1","bla") {
CHECK(true)
SECTION("a","bla"){CHECK(true)}
SECTION("b","bla"){CHECK(true);}
}

it says "4 exceptions tested" in 1 test case. Coming from Google test I suspect 2 testcases with 2 exceptions each, named "1/1/a" and "1/1/b".

September 25, 2012 | Unregistered CommenterSebastian

Valgrind gives me some strange output on Mac OS X Lion:. I only create the main routine using the CATCH macro. There exists no test case.

==38788== HEAP SUMMARY:
==38788== in use at exit: 331,566 bytes in 1,952 blocks
==38788== total heap usage: 4,777 allocs, 2,825 frees, 868,007 bytes allocated
==38788==
==38788== 18 bytes in 1 blocks are definitely lost in loss record 298 of 1,165
==38788== at 0xC8E6: malloc_zone_malloc (vg_replace_malloc.c:276)
==38788== by 0xCC58C6: malloc_set_zone_name (in /usr/lib/system/libsystem_c.dylib)
==38788== by 0xCC5DF2: _malloc_initialize (in /usr/lib/system/libsystem_c.dylib)
==38788== by 0xCC5F2B: malloc_good_size (in /usr/lib/system/libsystem_c.dylib)
==38788== by 0x1995E06: __CFStringChangeSizeMultiple (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==38788== by 0x199A0E7: CFStringAppend (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==38788== by 0x19AC59D: _convertToURLRepresentation (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==38788== by 0x1AA3E56: _CFURLInit (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==38788== by 0x19A4F81: CFURLCreateWithFileSystemPathRelativeToBase (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==38788== by 0x19BA388: _CFBundleGetMainBundleAlreadyLocked (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==38788== by 0x19BA2C5: CFBundleGetMainBundle (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==38788== by 0x1A09268: _CFBundleGetMainBundleIfLooksLikeBundle (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==38788==
==38788== 22 bytes in 1 blocks are definitely lost in loss record 304 of 1,165
==38788== at 0xC8E6: malloc_zone_malloc (vg_replace_malloc.c:276)
==38788== by 0xCC58C6: malloc_set_zone_name (in /usr/lib/system/libsystem_c.dylib)
==38788== by 0xBC13F1: dispatch_once_f (in /usr/lib/system/libdispatch.dylib)
==38788== by 0xBBE4CD: _dispatch_continuation_alloc_from_heap (in /usr/lib/system/libdispatch.dylib)
==38788== by 0xBBFA69: _dispatch_barrier_async_f_slow (in /usr/lib/system/libdispatch.dylib)
==38788== by 0xE0C415: _xpc_connection_create (in /usr/lib/system/libxpc.dylib)
==38788== by 0xE0CD99: xpc_connection_create (in /usr/lib/system/libxpc.dylib)
==38788== by 0x19DB9E7: -[NSXPCConnection initWithServiceName:privileged:] (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==38788== by 0x19DB57F: __CFXNotificationCenterSetupConnection (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==38788== by 0x19DB4C0: __CFXNotificationCenterCreate (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==38788== by 0x19DB399: __CFNotificationCenterGetDistributedCenter_block_invoke_1 (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==38788== by 0xBC13F1: dispatch_once_f (in /usr/lib/system/libdispatch.dylib)
==38788==


Either there is a problem with the runtime libs on Mac, or there is a small memory leak in CATCH... or do I miss something?

September 25, 2012 | Unregistered CommenterSebastian

hi!

including the glued catch.hpp works fine but i'm a little afraid to start writing lots of tests if the not-glued version doesn't work

Code:

#define CATCH_CONFIG_MAIN
#include "catch.hpp"

TEST_CASE( "stupid/1=2", "Prove that one equals 2" )
{
int one = 2;
REQUIRE( one == 2 );
}

Compileroutput:

$ g++ -o testFramework testFramework.cpp
In file included from catch.hpp:39:0,
from testFramework.cpp:2:
internal/catch_impl.hpp:19:28: fatal error: catch_runner.hpp: No such file or directory
compilation terminated.

when doing
#include "catch_runner.hpp" instead:


$ g++ -o testFramework testFramework.cpp
In file included from internal/catch_test_spec.h:11:0,
from internal/catch_config.hpp:11,
from internal/catch_commandline.hpp:11,
from catch_runner.hpp:11,
from testFramework.cpp:2:
internal/catch_test_case_info.h:51:9: error: 'Ptr' does not name a type
In file included from catch_runner.hpp:11:0,
from testFramework.cpp:2:
internal/catch_commandline.hpp:107:43: error: expected template-name before '<' token
internal/catch_commandline.hpp:107:43: error: expected '{' before '<' token
internal/catch_commandline.hpp:107:43: error: expected unqualified-id before '<' token
testFramework.cpp:8:1: error: expected '}' at end of input

Platform: Win7, Cygwin 1.7.17

Any help appreciated...

February 4, 2013 | Unregistered Commenteraugomat

This is superb!

April 4, 2013 | Unregistered CommenterDLed

Hi Phil.

I am just investigating this framework to hopefully use in our project(s). What attracted me to it is its simplicity, no-nonsense license, and the overall good gut-feeling about it that I have.

I am still very new to it but have tried a few experiments using your example Factorial function test as detailed in the docs, and it's going well, so a humble thank you for all your good work on this so far.

However, I have query regarding the "--success" command line option.
It may be my misunderstanding or down to the environment I'm using (VS2012 on Win 7, x64), or in fact something else entirely, but I seem to only get the success results given when there are no errors. If there are errors it will only show the errors and not the successes as well.

Thus I was just wondering if this is known/expected behaviour or if it is something on my part that I must be doing wrong?

All help gratefully appreciated, and keep up the great work.

February 13, 2014 | Unregistered CommenterDaniel

Hi Phil.
Apologies for the above post... just as I hit send it dawned upon me that I was using REQUIRES (which aborts the current test case on first failure found and continues to the next test case) rather than CHECK (which continues through the test case even if errors are found and ultimately reports everything).

So indeed it was something else entirely... me not using it correctly!

I am liking the flexibility available with the command line options (when the user knows what he is doing of course!).

February 13, 2014 | Unregistered CommenterDaniel

Glad you got it sorted, Daniel. Thanks for letting me know - and thanks for the kind words.

February 13, 2014 | Registered CommenterPhil Nash

PostPost a New Comment

Enter your information below to add a new comment.
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>