Thursday
Jan192017

Catch Up

Trolley

Stock image from Shutterstock

It's been just over six years since I first announced Catch to the world as a brand new C++ test framework!

In that time it has matured to the point that it can take on the heavyweights - while still staying true to its original goals of being lightweight, easy to get started with and low-friction to work with.

In the last couple of years or so it has also increased dramatically in popularity! That sounds like a good thing - and it is - but with that comes a greater diversity of environments and usage, and more people raising issues and submitting pull requests.

Again, it's great to have so much input from the community - especially in the form of pull requests - where other developers have gone to some effort to implement a change, or a fix, and present it back for inclusion in the main project. So it's been heart-breaking for me that, between this increase in volume and finding my meagre free-time stretched even further, so many issues and PRs have been left unacknowledged - many not even seen by me in the first place.

But two things have happened, recently, that completely change this state of affairs. We're moving firmly in the right direction again.

Firstly, as mentioned in On Joining JetBrains, I've recently changed jobs to one that should give me much more time and opportunity to work on Catch - as well as the opportunity to do so in my home office - with stable internet (as opposed to on the train while commuting to and from work). The first few months were a bit of a wash for the reasons discussed in that post, but, as I also suggested there, this year has seen that change and I've been able to put in quite a lot of work on Catch already.

But that's not really enough. There's a huge back-log - and I'm still only doing this part time - and I want to spend time working on Catch2 as well (more on that soon). I don't want to end up back in the situation where everything is backing up and there's no hope of recovery.

I've been hoping to find someone else to be a key maintainer of Catch for a couple of years now. I've not been very active in this search - for all the same reasons - but it's been on my mind.

But, just last month, after I appeared on CppCast talking about JetBrains and Catch, there was a thread on Reddit about it - with many expressing concern over the Catch situation. I brought the subject up on there again and got the attention of one of the commenters.

I didn't know it at the time, but Martin Hořeňovský has been responsible for a good number of those PRs and issues that had been left unaddressed - as well as an active community member in helping address other people's issues. So it's with great pleasure (and relief!) that I can announce that Martin now has full commit rights to Catch on GitHub and has been prolific in working through the currently outstanding tickets.

Martin seems to really "get" Catch, and the design goals around it - so working with him on this the last couple of weeks has been very rewarding. From some queries I just ran on GitHub it looks like 39 issues have been closed and 38 PRs merged or closed in that time! That's compared to 9 new issues and 7 PRs - about half of which were created by Martin and I in the process. And that's not to mention all the labels we've been using to categorise the other tickets - with many marked as "Resolved - pending review" - which usually means we think it's resolved but we're just waiting for feedback (or a chance for more testing).

With 219 open issues and 41 PRs still outstanding, at time of writing, there's a lot more work to do yet - but I hope this reassures you that we're going in the right direction - and fast!

And we're not stopping with Martin. We have at least one other volunteer that I'll be bringing up to speed soon.

Catch2

I've referred to Catch2 a number of times now, and talked a little about what it will be. The biggest reason for making it a major release, according to Semantic Versioning, is that it will drop support for pre-C++11. For that reason Catch Classic (1.x) will continue to receive at least bug fix updates - but no more new features once Catch2 is fully released. A few major features in the pipeline have been explicitly deferred to Catch2: concurrency support and generators/ property-based testing in particular.

Moving to C++11 provides a very large scope for cleaning up the code-base - which has a significant volume of code dedicated to platform-specific workarounds for compiler shortcomings, missing library features such as smart pointers, and boilerplate that will no longer be necessary with things like range-based-for, auto and others. Lambdas will be useful too, but are not quite so important.

Because taking advantage of C++11 has the potential to touch almost every line of code, I'm taking the opportunity to rewrite the core of Catch - primarily the assertion macros and the infrastructure to support that. This is code that is #included in every test file, and expanded (in the case of macros) in every test case or even every assertion. Keeping this code lightweight is essential to avoiding a compile time hit. There's a number of ways this foot-print can be reduced and the rewrite will strive for this as much as possible.

The rest of the code, concerned with maintaining the registry of tests, parsing and interpreting the command line, running tests and reporting results, will be updated more incrementally.

I already have a (not-yet-public) proof-of-concept version of the re-written code. It's not yet complete but, so far, has only one standard library dependency and minimal templates. The compile-time overhead is imperceptible.

In addition to compile-time, runtime performance is also a goal of Catch2. It's not an overriding goal - I won't be obfuscating the code in the name of wringing out the last few milliseconds of performance - but this is a definite change from Catch Classic where runtime performance was a non-goal. This is in recognition of the fact that Catch is used for more than just isolated unit tests - and will also become more important with property based testing.

I don't have a timeline, yet, for when I expect Catch2 to be ready - and in the immediate term getting Catch Classic back under control is the priority. Despite the partial re-write, and the major version increment, I expect tests written against Catch Classic to mostly "just work" with Catch2 - or require very minimal changes in a some rare cases.

You

As already mentioned many developers have also spent time and effort contributing issues, fixes and even feature PRs over the years. So Catch has really been a community project for years now and I'm very grateful for all the help and support. I think Catch has shown that having a low-friction approach to testing C++ code is very important to a lot of people and I'm hoping we'll continue to build on that. Thank you all.

Wednesday
Dec282016

C++17 - Why it's better than you might think

C++20 Horizon

From Mark Isaacson's Meeting C++ talk, "Exploring C++ and beyond"

I was recently interviewed for CppCast and one the news items that came up was a trip report from a recent C++ standards meeting (Issaquah, Nov 2016). This was one of the final meetings before the C++17 standard is wrapped up, so things are looking pretty set at this point. During the discussion I made the point that, despite initially being disappointed that so many headline features were not making it in (Concepts, Modules, Coroutines and Ranges - as well as dot operator and uniform call syntax), I'm actually very happy with how C++17 is shaping up. There are some very nice refinements and features (const expr if is looking quite big on its own) - and including a few surprise ones (structured bindings being the main one for me).

But the part of what I said that surprised even me (because I hadn't really thought of it until a couple of hours before we recorded) was that perhaps it is for the best that we don't get those bigger features just yet! The thinking was that if you take them all together - or even just two or three of them - they have the potential to change the language - and the way we write "modern C++" perhaps even more so than C++11 did - and that's really saying something! Now that's a good thing, in my opinion, but I do wonder if it would be too soon for such large scale changes just yet.

After the 98 standard C++ went into a thirteen year period in the wilderness (there was C++03, which fixed a couple of problems with the 98 standard - but didn't actually add any new features - except value initialisation). As this period coincided with the rise of other mainstream languages - Java and C# in particular - it seemed that C++ was a dying language - destined for a drawn out, Cobolesque, old-age at best.

But C++11 changed all that and injected a vitality and enthusiasm into the community not seen since the late 90s - if ever! Again the timing was a factor - with Moore's Law no longer influencing single-core performance there was a resurgence of interest in low/ zero overhead systems languages - and C++11 was getting modern enough to be palatable again. "There's no such thing as a free lunch" turns out to be true if you wait long enough.

So the seismic changes in C++11 were overdue, welcome and much needed at that time. Since then the standardisation process has moved to the "train model", which has settled on a new standard every three years. Whatever is ready (and fits) makes it in. If it's not baked it's dropped - or is moved into a TS that can be given more real-world testing before being reconsidered. This has allowed momentum to be maintained and reassures us that we won't be stuck without an update to the standard for too long again.

On the other hand many code-bases are still catching up to C++11. There are not many breaking changes - and you can introduce newer features incrementally and to only parts of the code-base - but this can lead to some odd looking code and once you start converting things you tend to want to go all in. Even if that's not true for your own codebase it may be true of libraries and frameworks you depend on! Those features we wanted in C++17 could have a similar - maybe even greater - effect and my feeling is that, while they would certainly be welcomed by many (me included) - there would also be many more that might start to see the churn on the language as a sign of instability. "What? We've only just moved on to C++11 and you want us to adopt these features too?". Sometimes it can be nice to just know where you are with a language - especially after a large set of changes. 2011 might seem like a long time ago but there's a long lag in compiler conformance, then compiler adoption, then understanding and usage of newer features. Those just starting to experiment with C++11 language features are still very common.

I could be wrong about this, but it feels like there's something in it based on my experience. And I think the long gap between C++98 and C++11 is responsible for at least amplifying the effect. People got used to C++ being defined a single way and now we have three standards already in use, with another one almost ready. It's a lot to keep up with - even for those of us that enjoy that sort of thing!

So I'm really looking forward to those bigger features that we'll hopefully get in C++20 (and don't forget you can even use the TS's now if your compiler supports them - and the Ranges library is available on GitHub) - but I'm also looking forward to updating the language with C++17 and the community gaining a little more experience with the new, rapidly evolving, model of C++ before the next big push.

Thursday
Dec222016

On joining JetBrains

JbOffice

The JetBrains office in St. Petersburg

I recently joined JetBrains as Developer Advocate for their (our) C++ related tools, CLion (a cross-platform C++ IDE), AppCode (for Mac and iOS development - also supporting Swift and Objective-C) and ReSharper C++ (plug-in for Visual Studio).

This was a significant move for me as my previous full-time role had been as a developer at one place for over seven years! In that time I had been able to take time out to do occasional stints of coaching and consulting, as well as travelling to a number of conferences and other events where I always felt like I was more a part of the community. But making that a significant part of my job has been an interesting transition.

JetBrains are a superb software engineering company with a friendly, diverse, set of employees, distributed across a number of offices. I primarily interact with members in St. Petersburg, Russia and Munich, Germany (and have already visited both offices). Personally I work at home - as do the rest of the Developer Advocacy team (their respective homes, that is. There's not quite enough room at my place!)

I'm not going to talk too much more about my role in general, here. I covered it a bit more in an interview published on some of the JetBrains blogs soon after I joined. What I wanted to talk about here is how it affects my other activities - and in particular Catch.

One of the enticing things about taking this role was that I would be expected to continue work on Open Source projects. After all I wouldn't be working on a paid project anymore - but I still need to keep my skills relevant (and not just tuned to small, self-contained, demos). Over the last three months, however, I've found there has been a lot more to get up-to-speed on - as well as opportunity to overbook myself - than I had anticipated. So catching up on Catch has been minimal so far.

But with my first three release-oriented activities under my belt, and a clearer idea of what to expect, I'm looking forward to turning that around early in the new year. I have a lot of issues and PRs to catch up on, and I've started work on Catch2 in parallel, which I'm keen to get to a first release of (I intend to follow-up soon with a post on Catch2). I'm also hoping to find people I can work with (and give commit rights to) to help with all of the above.

Beyond Catch I expect to blog more frequently again (both here and on the JetBrains blogs) and I have some other projects in the pipeline too.

Wednesday
Jul082015

A Game of Tag

One of the tent-pole features of Catch is the ability to write test names as free-form strings. When you run a Catch executable from the command line you can specify a test case by name, to run just that one:

./MyTestExe "a very nice test case"

or you can use wildcards to run a group of test cases (or just one with less typing):

./MyTestExe "*very nice*"

If you want to use wildcards but you're not sure what they'll match you can combine this with the listing option, -l, to see which test cases match the pattern:

./MyTestExe "*very nice*" -l
Matching test cases:
  a very nice test case
  a not very nice test case
2 matching test cases

This is already quite a powerful way to group test cases into ad-hoc "suites". However we don't want to twist our test names into artificial schemes for this purposes (although, early on, that's exactly what I proposed). Instead Catch allows you to add "tags" to test cases.

TEST_CASE( "a very nice test case", "[nice][good]" ) { /* ... */ }
TEST_CASE( "a not very nice test case", "[nice][bad]" ) { /* ... */ }

Now we can run all tests with a certain tag:

./MyTestExe [good]

or combination of tags:

./MyTestExe [nice][good]

also with exclusions:

./MyTestExe [nice]~[bad]

unions are supported with ,:

./MyTestExe [nice],[pleasant]

Very powerful! And this functionality has been around for a while.

More recent, and less well known (mostly because they weren't documented until recently) are a set of "special tags": Instruction Tags, Hiding Tags, Tag Aliases and some automatically generated tags.

Let's see what they're all about.

Instruction Tags

In general all tags that start with a symbol are reserved by Catch (or, put another way, user defined tag names must start with an alpha-numeric character). This allows a nice rich range of namespaces for special tags. Tags that start with the ! character are Instruction tags. They inform Catch something about the test case that they apply to. At time of writing the following are defined:

  • [!hide] This "hides" the test from the default run (i.e. if you run the test executable without specifying any names or tags). This feature was originally introduced with the [hide] tag (note, no: !) - and is still supported, though deprecated. There is also a shortcut form, [.] which we'll revisit in a moment.
  • [!throws] This tells Catch that an exception may be thrown in the course of executing the test - even if it is caught and dealt with. If you've ever tried to track down a rogue exception in your debugger - and so have set the debugger to break on exceptions as they're thrown - you'll know how frustrating all the false positives coming from such tests are! So Catch provides a way to suppress exceptions it is expecting - through the -e or --nothrow options on the command line. This already skips over REQUIRE_THROWS... or CHECK_THROWS... assertions. The [!throws] tag covers you for cases where the exception is caught and handled in the code under test (or your test code).
  • [!shouldfail] This tells Catch that you're expecting this test to fail! Furthermore, if it does fail then it should treat that as a pass!
  • [!mayfail] Rather than explicitly inverting the pass/ fail logic as the previous tag does, this tag just says that the test may fail but that's ok (although it is still reported). It's also ok if it passes.

Hiding Tags

We already looked at [!hide] (and the deprecated [hide]) above, and mentioned that [.] was a shortcut for the same.

It turns that when one of these tags is used it is often combined with another tag that is used when you do want to run the test. The classic example is where you write integration tests in the same executable as unit tests. By default you don't want the integration tests to run as you want the shortest possible path to running just unit tests. So you hide them but also tag them [integration], or something similar (the word "integration" has no significance to Catch). So pairings like, [.][integration] or [.][performance] are frequently found together.

So, as a convenience, Catch now supports . as a tag prefix. The rest of the tag can be completely custom and works exactly like any other normal tag - except that the test is also hidden. Our examples would, thus, be written as [.integration] and [.performance]

One final point to mention about hiding tags is that, due to the way they have evolved through a number of forms (including the severely deprecated "./" name prefix) whichever form is used will not only hide the test, but any of the other forms will match it in a tag pattern. e.g. if you tag a test with [.] you can match it with [!hide].

Tag Aliases

As we saw earlier, tags can be combined in fairly complex ways. While this is powerful and flexible, it can be a bit awkward if you often want to use the same tag expression. Wouldn't it be nice if there was a way of writing the expression once then getting Catch to remember it for you - and associate it with an easier to remember name?

Well there is! You can associate any tag pattern with a name that you can use just like any normal tag - except that it must begin with the @ character.

You create a tag alias, in code, using the CATCH_REGISTER_TAG_ALIAS macro. E.g.

CATCH_REGISTER_TAG_ALIAS( "[@not nice]", "~[nice]~[!hide]" );

This registers a tag alias, [@not nice] which, when expanded will match all tests that are not tagged [nice] but also are not hidden. The second part is important because if you have any hidden tests then they will usually be included any time you use a not expression (~) because the rule is that tests are only hidden if no pattern is specified!

Also did you notice that we had a space in the tag name? Surprised? I never said that tags could not include spaces. Of course they can.

You can register as many aliases as you like and you can put them anywhere you like (as long as catch.hpp is #included). However I recommend keeping them all in your main source file (the one you #define CATCH_CONFIG_MAIN, or equivalent) - simply so you only have to look in one place for them.

Filenames As Tags

The newest special tag form is the result of automatically generating a set of tags. The tags all begin with the # character (I've resisted the urge to call them "hash tags"). The rest of the tag is generated from the name of the source file that the test is implemented in. The full path (as reported by __FILE__) is stripped of its directories and extension - so all tests in /Development/Tests/SquirrelTests.cpp would be tagged, [#SquirrelTests].

At time of writing this feature is only available on the develop branch on GitHub - and must be specifically enabled running with the --filenames-as-tags or -# command line options. It's possible that situation may change by the time it makes it onto master.

The Tag Line

So tags not only provide a rich grouping mechanism in Catch - they also allow you to control some aspects of how Catch runs and treats test cases. Some tags can be generated for you - and some tags can be expanded from simpler forms. We've covered here the complete set of special tags at the time of writing. If you're reading this in the future there may be more - I'll try and be better at keeping the docs up-to-date there. Also any stock price tips you might have from the future would be welcome too.
Friday
May012015

Naming is hard - or is it?

Following Peter Hilton's excellent ACCU talk, at last week's conference in Bristol, "How to name things - the hardest problem in programming", a few of us were discussing some of the points raised - and some not raised.

He had discussed identifier length without any mention of Uncle Bob's guideline, whereby the length of a variable name should be proportional to it's scope (i.e. large or global scopes need longer, descriptive, names whereas in smaller, local, scopes shorter, more concise - even single letter - names are appropriate). This seemed all the more of an omission given that he later referenced the book, Clean Code.

It wasn't that Peter disgreed with Uncle Bob (who doesn't, half the time?) that surprised me but that he didn't even mention it in passing. I thought it was fairly well known. Actually I double checked and it is not discussed fully in the book, which only says, "The length of a name should correspond to the size of its scope". This is expanded considerably in Clean Coders (video) episode 2. Also, of course, this is not really "Uncle Bob's rule". Kevlin Henney recalls that he first heard of it in the 90s and it may well have been kicking around before that. Bob calls it "The Scope Rule".

Kevlin was one of those discussing this afterwards. After initially toying with The Scope Rule in the 90s he came to consider it not particularly useful. This, too, surprised me as I had found it worked quite well for me. Or so I thought. Further discussion with Kevlin led to the conclusion that I had read more of my own interpretation into The Scope Rule than I had realised! So I started musing over exactly what my interpretation was.

A transparent reference

As it happened a concept key to clarifying matters came from another great talk at the same conference just the day before. Didier Verna's "Referential transparency is overrated". In this talk Didier discussed various ways that useful idioms in Lisp required violating referential transparency. At one point he explained how "hidden" variables may be introduced by one macro that were then used by another. This worked because the thing being referred to was named very generically - so both macros agreed on the name. He drew on the term "Anaphora" from linguistics, which is where one part of an expression - usually a pronoun - stands in for a more specialised part - such as a person's name - introduced earlier in the context. For example, just now I used the word, "he" to refer to Didier Verna. It was clear who I was talking about because his was the most recently specified name in the current context. In fact I used this anaphoric term a couple of times - and many, many, times in this article. If I had had to fully qualify "Didier Verna" every time writing would quickly become very cumbersome. Anaphora is used very frequently in natural language - usually to good effect.

Scope Creep

I believe this is key to understanding why and when shorter identifiers can be used too. When I had been talking with Kevlin it had become apparent that we had different interpretations of the word, "scope". I realised that I had subconsciously expanded the specific technical meaning to include a more general idea of "context" - including the anaphoric context.

To make this clear I might write some (C++) code like this:

	std::string s = getNextString();
	if( !s.empty() )
		std::cout << "received string: " << s << std::endl;

Many corporate, or personal, coding standards would balk at such practice! Single character identifiers? Way to obfuscate the code!

But how has it obfuscated anything? Look at it as an anaphoric entity. In this case the variable name 's' is anaphoric. We know it is "the next string" because we saw it being introduced by the function call, "getNextString()". We then use it twice on the next couple of lines. There are no other strings being introduced in the same vicinity to confuse it with, and the context in which it is used is kept small. There is no ambiguity and the full identity is revealed in the immediate vicinity.

Sustainability

But what if we add more code, or move parts of this elsewhere? Certainly code evolves over time in ways that can make things less clear if we don't change them. That's true regardless. Naming of the entities at play should always form part of your consideration when refactoring or otherwise modifying existing code. Does it make this code less "sustainable" (to reference another property that Kevlin likes to talk about)? I don't think so. In the worst case, if you don't immediately notice that a short name has become unclear because it's usage has drifted out of anaphoric range, you'll notice the next time you look at it and, momentarily, think "why is there an free variable called 's' here? What on earth is that. You'll take a moment to find it's original declaration, work out what it is, then decide to encode that in the name by renaming the variable at that point. Variable renaming is one of the safest and most ubiquituous refactorings around so I have no qualms about deferring such identity expansion to such a time as it is needed.

Why?

But what about the other side of the argument? Is there any advantage to using a short, even single character, variable name in the first place?

This is often cast as a matter of optimising for typing speed - in world where we typically read these names many many times more than we write them.

While introducing, even small, speed bumps to writing code might discourage spending more time than necessary writing code (which in turn may discourage certain refactorings) it's not really about typing performance at all - It's about readability! Consider again the linguistic definition of anaphora: substituting an, unambiguous, subsequent reference to an entity with a shorter form (e.g. a pronoun) that means the same thing. We do this all the time in natural speech and the written word. Why? Because it would sound unnatural and cumbersome to fully qualify every entity we talk about all the time!

The same applies in programming. Where it is perfectly clear from the immediate context what an identifier refers to then using greater verbosity actually increases the cognitive friction! The more unnecessary and redundant noise and ceremony we can strip away from our code the easier it will be to read, in a shorter period of time. That fact that anaphora is so common in natural language should give us a clue as to our ability to code with it's use in a natural and efficient way.

Now I've only mentally organised my thoughts around this as a result of ruminating on those two talks - and some of the offshoot discussions - but I realise this is essentially how I had interpreted The Scope Rule. Now I've worked it through when I go back and compare it with what Mr Martin actually said his version sounds like a poor proxy for the anaphoric interpretation.

So naming - good naming - is still hard. We've only just discussed one narrow aspect here. But perhaps this has made some of it that little bit easier.