TDD - is it worth it?

There are many articles on the subject of what TDD is, why and when it is worth it, and which attempt to counter common objections.

This is not one of those.
Well. Maybe a bit.

This is more specifically a response to Marco Arment's comments in his podcast, Build & Analyze, episodes 107 and 108. Episode 108 was the last episode so there is an air of finality to the subject matter. Many Mac and iOS developers (as well as developers for other platforms) listen to the show and, while you'd hope they can all think for themselves and reach their own conclusions, it's undeniable that opinions, if not already well formed, may easily be swayed by what a respected figure says in a high profile, and well polished, medium. This can be unfortunate. I'm sure Marco didn't intend to do any damage. I've listened to every episode of Build & Analyze for over a year and enjoyed it. This is certainly not a flame against Marco or the show. However I'm going to walk through Marco's comments as a proxy for many who make similar statements. In doing so I quote him liberally, rearranging to fit my narrative. I'm including the time markers so you can easily listen to it in the original context

Episode 107: 6:50 The comments in question started in response to a listener question about testing and logging. "Test first development" was mentioned in passing, but Marco groups these all together as "a whole lot of formalism" which "matters a lot in an enterprise environment", but he, "[doesn't] really do any of this stuff".

These are quite different things but let's set that aside for the moment. So far he's just told us he doesn't use any of it and it works ok for him. Ok.

But he carries on, "All of this structure and all of this overhead - I wouldn't be able to release this software as one person and have it stay competitive and have it released frequently - there wouldn't be enough time for that".

Now he's making a claim.

Some more choice phrases:

"You've gotta keep moving so fast"
"It isn't worth it in those environments"
"I don't care about all that stuff"

(The last seems to contradict an earlier statement, "Some of this stuff I do regret not knowing what it is", but we'll let that pass)

The first two continue on the theme of needing to move fast and "all those formalisms" "slowing you down".

Ok. Let's stop a moment. I want to be clear again: I'm not trying to attack Marco here. It is not my intention to pick apart the details of his extemporaneous words. He honestly appears to be speaking on the assumption that all the things the listener had asked about are excessive, "formal things", that "matter a lot in […] medical […] or banking" enterprises, or for a "space shuttle" - but are just unnecessary overhead for a one-man independent developer writing, "for apps on phones that don't do anything important".

If you accept the faulty premise the rest of his reasoning makes a lot of sense.
So after I listened to episode 107 I mailed in to explain the difference between low-level, developer oriented, disciplines - TDD in particular - and the higher level, quality-driven, testing approaches more commonly associated with large enterprises or critical industries. I attempted to briefly summarise the benefits that even a one-man developer shop might get from adopting such a discipline.

Unsurprisingly I was not the only one.

Episode 108, from 28:58: Marco brings up the feedback he received, amused to note that everyone pointed out that TDD was, "not only about testing, but it's about writing, 'self-contained code that makes it easier to refactor…'". He recognises that this (and other principles of software design) are "good programming practice". He references The Pragmatic Programmer as being a good source of such principles - and it is. However he appears to believe that proponents of TDD think this is "exclusive to TDD" and, unfortunately, this takes him off-track again. (These days I would probably recommend Growing Objected Oriented Software Guided By Tests and Clean Code as more up-to-date works that cover similar ground.)

Interestingly, The Pragmatic Programmer thoroughly recommends various forms of testing - including TDD (although not by that name - I'm not sure that term had been coined when the book was written). Pertinent to Marco's objections it has this to say:

"A good project may well have more test code than production code. The time it takes to produce this test code is worth the effort. It ends up being much cheaper in the long run" (emphasis mine).

The book also has a section on Refactoring. The topic was new (by that name) at the time - it even mentions that the "first major book on refactoring" was being published around the same time. Nor had the central role of refactoring in TDD's, "Red, Green, Refactor", cycle been clearly established at the time. Nonetheless it makes these two points that cut to the heart of it:

1. "Don't try to refactor and add functionality at the same time."
2. "Make sure you have good tests before you begin refactoring. Run the tests as often as possible. That way you will know quickly if your changes have broken anything. […] [Martin] Fowler's point of maintaining good regression tests is the key to refactoring with confidence."

As to the other principles: can they be followed without using a TDD approach? Of course they can. So what was all Marco's feedback about? Simply this: Driving your code from tests forces you to make the code testable. This naturally leads to code that is less coupled, has less responsibilities per unit, with higher cohesion and is written from the start with the idea that it is easy to change. As a by-product you have a great set of automated tests that give you the confidence to refactor to improve all these - and other - principles.

On their own that would all require a lot of discipline - and can be hard to measure (to know how well you are doing). In TDD it's usually easier to follow them from the start - and if you don't the tests will "tell" you by becoming harder to write. TDD is, itself, a discipline - and does require some experience and a prior understanding of the design principles that make it go smoothly. But in my experience it is much easier - and more gratifying - to follow the simple discipline of TDD than to remember to apply all the other individual principles with nothing to guide you.

The result is that your code will tend to be lean and supple and can respond to changes in requirements quickly and easily. You'll spend less time on finding and fixing bugs, and less time on writing code you didn't really need. The time spent writing the tests usually pays for itself almost straight away - often several times over. Saving time and being able to respond to change quickly - are these not the very qualities Marco values so much?

Back in Episode 107, from around 25:57 - right after his first comments on testing - he talks about a situation he got into with The Magazine - a brand new code base for him. "The code was just getting messier and messier". He had a bug but he "couldn't figure out what the heck was going on" and "had all these other things that were problems with that system and it was really clunky". He goes on to say, "it was starting to get to a point where I was fearing adding anything to it. When code gets to a point where it's like a giant pile of messy spaghetti and it feels very fragile and you feel like, 'Oh my God. I need to add an attribute to this - is that going to break anything?'"

It's a shame that he appears to have just dismissed what is probably the best tool we have come up with to date for avoiding this ever happening in the first place.

Thanks to all those who I coerced into reviewing this post for me: Seb Rose, Jez Higgins, Paul Grenyer, Claudius Link, Pal Balog, Hubert Matthews, Peter Pilgrim, Martin Moene, Giovanni Asproni, Yechiel Kimchi, Chris O'Dell, Peter Sommerlad and Graham Lee

Upcoming speaker engagements

I seem to have gotten myself committed to some speaker engagements over the next couple of months:

Next week, on 26th April, I'm giving the thursday keynote at the ACCU 2012 conference in Oxford. My topic there is "The Congruent Programmer" and is about aligning what we do with our motivations and gaining clarity on why we do things.

Then, on 29th June, I'll be over at Mobile East talking about how to TDD your iOS apps. There seems to be growing interest in this area. I've just been engaged with a team at the BBC doing just that. And a few days ago Graham Lee's new book on "Test Driven iOS Development" was released. Graham's book mentions my C++/ Objective-C test framework, CATCH, so it must be good!

Finesse

I've been playing piano since I was about 11.
Not continuously, of course - my fingers would have fallen off a long time ago! In fact I've barely touched one for a decade.

I'm really more of a synth player - and I've never been a great performer - my interest lay more in composition anyway (some old pieces of mine over on soundcloud.com/phil_nash). But I appreciate a good piano action on a synth keyboard. I chose my Ensoniq TS-12 synth as it had one of the better piano actions (and piano sounds) when I bought it in the mid 90s.

But something happened in 2002 that changed the way I thought about it. I was trying to get back into playing again after a few dry years. I'd just bought a MOTU 828 (effectively a very low-latency external sound card) and a copy of Steinberg's "The Grand". The Grand was a VST instrument that was one of the first to use high definition, full decay, samples of every key on the piano at multiple velocities. That was amazing enough. But then it could perform addition processing - to apply the sympathetic resonance of the open piano strings when the sustain pedal is down, for example, or add in the sounds of the felt and hammers themselves. The result was a breathtaking leap forward in authenticity in digital piano sound.

The was only one problem. At the time the computer processing power, as well as disk IO, was limited enough that it didn't take much layering to push the boundaries. This resulted in note-stealing (where notes deemed least audible are culled, freeing up processing power for those more to the fore), freezes or even crashes. One option to counter this was to reduce the complexity of the instruments. Turning off features such as open string resonance - or using a simpler version of the instrument (e.g. my keyboard's built-in piano sample).

In theory that was an acceptable trade-off as it only really affected live playback and recording. The finished mix could be rendered in non-real-time, including all those CPU-intensive features in the final recording.

That's when I realised something quite surprising. When I played the full-featured version of The Grand I found I played differently to when I was playing the TS-12's on-board piano sound.
Even more surprising was that even playing a simplified Grand was noticeably different to playing the fully-enabled version!
And when I say I played differently the difference really was stark! The more authentic the piano sound, the more my fingers flowed across the keys. I was more accurate, more musical, and felt more connected with the music. Remember this was using the same physical keyboard with, essentially, the same instrument.

Audibly, the difference between a no-holds-barred Grand and one with the extra processing disabled, was very subtle - especially during normal playing. If you played a chord and let it ring you could hear the harmonics "shimmer" with the processing enabled. But I wouldn't consciously notice that while playing in general.

And yet I was quite clearly picking up on it and behaving differently as a result of it. Why?
Obviously all of that extra disk IO and processing was there to make the sound more authentic. To more closely mimic the nuances of the real world instrument. That's all intended to trick the listener's brain into thinking it is the real world instrument. But the player is a listener too. And the player, even one as unaccomplished as I, has a different interaction experience with a real instrument than an artificial one.

This has been quite a long anecdote to make one point: that small, barely perceptible, differences may have a huge impact on our experience - although not necessarily in ways we are consciously aware of. This seems especially true when applied to the way we interact with digital interfaces - whether that be a synth keyboard pretending to be a piano, or a touch screen pretending to be, say, a piece of paper. The details matter. We are participating in a fragile sensual suspension of disbelief. The tiniest crack that betrays the deception brings the whole thing down.

And we're only just at the beginning of a revolution in interaction metaphor.

Could the Internet please stop changing while I finish this blog post?

Whenever I write a blog entry I iterate it a few times - minor corrections here, typographic fixes there - or often rewriting (or deleting) large chunks of it.

Sometimes I'll have a half-finished entry in draft and come back to it months later - only to make substantial changes to it.

Normally I wouldn't mention that in the final version (with this previous exception). But on this occasion the changes reflect the mercurial nature of the subject matter so nicely that I hope you'll forgive a brief aside.

I'd started an entry about a year ago with the title, "Welcome to the semi-connected age". It was meant to be a summary of the current state of the art in "connected" apps, including the stand-off between web apps and native apps - all leading up to my take on it and what I'm working on in that area.

I'd already written a lot.

But when I picked it up a couple of week ago to see if I could finish it I realised that just about everything I touched on in it had changed! I had referenced Silverlight as a way forward with much potential (it has since been sidelined as a desktop web technology), Mono (in connection with Moonlight, in particular) - just had it's staff laid off, and the inadequacy of Javascript and HTML (Javascript is a belatedly rising star and HTML5+CSS2/3 has been thrown into the mainstream despite not having settled into a standard yet). I even touched on Dropbox as being the poster child for cloud storage (they have since been embroiled in security concerns).

So I started again from scratch, with a piece called, "That syncing feeling". But even before the end of that day things were changing again! Microsoft started showing off Windows 8 - which pushes HTML5+CSS3+JS even further into the limelight on the desktop - much to the dismay of Silverlight developers. And Apple confirmed that they would be launching iCloud at the, then, following week's WWDC. That's what prompted the tweet that I took the title of this post from.

I thought it might be better to wait until things had settled down a bit.

And I'm glad I did. The WWDC Keynote has really stirred things up. I'm not sure if most people really "get" why, yet. But what I find reassuring is that Apple seem to be moving in exactly the direction that my original post was trying to promote.

So that brings us full circle. I can now put my points across, but this time with Apple to back me up.

To The Cloud

'Cloud Computing' by Combined Media

Even after writing that intro I've abandoned the rest of this post and restarted from scratch a couple of times. There's plenty on the cutting room floor for follow-up posts.

I'm going to use this post to cover why I see Apple's iCloud services as doing it right where some people see shortcomings.

What are these "shortcomings"?

iCloud is about transferring and syncing data. It's about being the canonical store of that data. I've seen a number of complaints that this is not really "the cloud", and that Apple are giving us a half-baked solution. What is the other half? The True Cloud, they say, hosts the apps themselves - not just the data (we'll ignore the MobileMe apps for the moment).

I couldn't disagree more! Why?

Dire RIA

First. Even web apps run on your local machine. They might be hosted on a server but they are effectively deployed to your desktop/ device every time you use them! Caching may play a role here, but that's really just a deployment performance tweak.

So a web app is just a Javascript (or some RIA language) app that is continuously deployed then interpreted on your desktop. It has some cross-platform advantages, due to being browser hosted - although it does trade these for cross-browser issues instead.

Second. Writing a good, responsive, sophisticated web app is hard. Harder than the equivalent native app. But getting sync right between distributed clients is harder. Much harder. It could be argued that no-one has got it quite right yet. You could make a case, and this is my position, that hosting data for distributed native apps is The True Cloud. Web apps, in a way, are the half-baked solution.

So, what are the pros and cons of each?

Web apps:

Pros: Continuously deployed - always up-to-date. Minimal data integrity issues (always working off canonical version).
Cons: Requires constant connectivity. Slower. poorer UX.

Native apps:

Pros: Can work disconnected. Can be much faster. Matches look-and-feel of your chosen platform. Integration with other apps.
Cons: Installation/ Updates can be more onerous or require user action and take time. Must deal with sync issues.

In my earlier drafts I went into much more detail on these points - especially connectivity (e.g. RIA technologies that allow disconnected working). But this time I'm just going to jump straight into how last week's WWDC announcements change the score:

But let's add a third category

iCloud enabled, Mac OS-X Lion or iOS 5 app:

Pros: Can work disconnected. Can be much faster. Matches look-and-feel of your chosen platform. Integration with other apps. One click install, automatic pushed updates using delta patches (fast!). Sync issues taken care of.

What happened to the Cons field? Well you might still have some reasons to prefer web apps - such as the cross-platform promise. But for me, at least, now there are no cons! Especially if you combine native apps with web-hosted versions. That makes senses for PIM apps, like contacts, email and calenders. Maybe it makes sense for productivity apps too, like word processors, spreadsheets and slide presentation apps.

And guess what, Apple has cloud hosted versions of all those too - which work seamlessly with their native counterparts. At time of writing the future of these is uncertain, but I think it highly likely that they will continue to exist.

Best of all worlds?

Maybe. It does severely lock you into Apple's products, of course. I'm a big fan of Apple hardware and software in general - but this is something that must transcend a single company. They're not doing anything new at the small scale but, at the moment, it's only really Apple that have everything necessary to be able to pull this off end-to-end. I hope that in doing so they pave the way for the community to piece together a more coherent alternative picture. We have all the components out there. Many of them better than Apple is offering.

That syncing feeling

There are those who have been claiming that iCloud does not sync, but merely pushes content that it holds down to devices. It's true that Jobs didn't use the word, "sync" in his WWDC Keynote coverage. In fact he seemed to be specifically avoiding the word. Does that mean there really is no syncing capability in iCloud?

Well remember that, whether iCloud assimilates the MobileMe services that sync contacts, mail, calendars, etc. But even for the new services sync is fundamental to how they work. You add a song on one device, the other devices get it (which may involve the song being uploaded). You take a photo on one device, it gets synced to other devices.

However these new services seem to be designed in such a way as to avoid, or at least minimise, the possibility of conflicts. If it was just a case of holding songs and photos in a file system and then syncing the file system then all those thorny conflict resolution challenges that are traditionally associated with sync arise.

A lot of us have been working for 10 years to get rid of the file system

But Apple have been very careful to keep away from those issues by managing the content at a higher level. Jobs seemed particularly proud when he said, "A lot of us have been working for 10 years to get rid of the file system". This is not just about simplification - it's about the file system being the wrong tool for the cloud - and I say this as someone who has worked for a file-based Cloud Storage company.

It's this "post-file-system era" that is central to what I'm going to cover in more detail in a future post.

Unit Testing in C++ and Objective-C just got ridiculously easier still

'Spider Web in Morning Sun' by Rob van Hilten

In my previous post I introduced Catch - my unit testing framework for C++ and Objective-C.

The response was overwhelming. Thanks to all who commented, offered support - and even contributed to the code with fixes and features.

It certainly gave me the motivation to continue active development and a lot has changed since that post. I'm going to cover some highlights, but first I want to focus on what has been one of the most distinguishing features of Catch that has attracted so much attention - and how I have not rested but made that even better!

How easy is easy enough?

Back in April I gave a five minute lightning talk on Catch at the ACCU conference in Oxford (I highly recommend the conference). With just five minutes to talk about what makes Catch special what was I going to cover? The natural operator-based comparison syntax? The use of Sections instead of class-based fixtures? Data generators?

Well I did touch on the first point. But I decided to use the short amount of time to drive home just how quickly and easily you can get up and running with Catch. So after a 30 second intro I went to the GitHub page for Catch (now aliased as catch-test.net), downloaded the zip of the source (over a 3G connection), unzipped and copied to a central location, fired up XCode, started a fresh C++ project, added the path to Catch's headers, #include'd "catch_with_main.hpp", wrote an anonymous test case, compiled and ran it, demonstrated how it caught a bug, fixed the bug and finally recompiled and re-ran to see the bug go away.

Phew! Not bad for five minutes, I thought. And from the feedback I got afterwards it really did drive the point home.

Compare that with my first experience of using Google Test. It took me over an hour to get it downloaded and building in XCode (the XCode projects don't seem to have been maintained recently - so perhaps that is a little unfair). There are other frameworks that I've tried where I have just run out of patience and never got them going.

Of course I'm biased. But I have had several people tell me that they tried Catch and found it to be the easiest C++ Unit Test framework they have used.

But still I wasn't completely satisfied with the initial experience and ease of incorporating Catch into your own projects.

In particular, if you maintain your own open source project and want to bundle it with a set of unit tests (and why wouldn't you?) then it starts to get fiddly. Do you list Catch as an external dependency that the user must install on their own? (no matter how easy they are to install external dependencies are one or my least favourite things). Do you include all the source to Catch directly in your project tree? That can get awkward to maintain and makes it look like your project is much bigger than it is. If you host your project on GitHub too (or some other Git based repository) you could include Catch as a submodule. That's still not ideal, has some of the problems of the first two options, and is not possible for everyone.

There can be only one

Since Catch, as a library, is fully header-only I decided provided a single header version that is ideal for direction inclusion in third-party projects.

How did I do this?

Go on guess.

Did you guess that I wrote a simple Python script to partially preprocess the headers so that the #includes within the library are expanded out (just once, of course), leaving the rest untouched?

If you did you're not far off. Fortunately some of the conventions I have used within the source meant I could drastically simplify the script. It doesn't need to be a full C preprocessor. It only needs to understand #include and #ifndef/#endif for include guards. Even those are simplified. The whole script is just 42 lines of code. 42 always seems to be the answer.

The result is https://github.com/philsquared/Catch/blob/master/single_include/catch.hpp

I see no reason why this should not be the default way to use Catch - unless you are developing Catch itself. So I'm now providing this file as a separate download from within GitHub. Think of it as the "compiled" header. The lib file of the header-only world.

Licence To Catch

But Open Source is a quagmire of licensing issues, isn't it?

Well it certainly can be. Those familiar with GPL and similar open source licences may be very wary of embedding one open source library (Catch) within another (their own).

IANAL but my understanding is that, contrary to what might seem intuitive, source code with no license at all can be more dangerous, legally speaking, than if it does have one (and if you thought that sentence was difficult to parse you should try reading a software license).

So Catch is licensed. I've used the Boost license. For a number of reasons:

• It is very permissive. In particular it is not viral. It explicitly allows the case of including the source of Catch along with the distribution of your own source code with no requirements on your own code
• It's been around for a while now - long enough, I think, that most people are comfortable with it. I work with banks, who can be very nervous about software licensing issues - especially open source. But every one I have worked at has already got Boost through it's compliance process. I'm hoping that will ease any barriers to adoption.
• I'm familiar with Boost, know many of it's contributors personally, and generally trust the spirit of the licence. Boost itself is a very well known and highly respected set of libraries - with very widespread adoption. A large part of Boost is in header-only libraries and people are already comfortable including them in their own projects.

So what's the Catch? The catch is that I retain the right to keep using that joke - well beyond it's humorous lifetime.

The important bit:

In short: any open source author who wants to use Catch to write unit tests for their own projects should feel very free to do so and to include the single-header (or full) version of the library in their own repository and along with their distribution.

That fully applies to commercial projects too, of course.

What else?

Here's a quick run down of some of the other changes and features that have gone in:

• Single evaluation of test expressions. The original implementation evaluated the expression being tested twice - once to get the result, and then again to get the component values. There were some obstacles to getting this to work whilst only evaluating the expression once. But we got there in the end. This is critical if you want to write test expressions that have side-effects.
• Anonymous test cases. A little thing, but I find them really handy when starting a new project or component and I'm just exploring the space. The idea is that you don't need to think of a name and description for your test - you can just dive straight in and write code. If you end up with something more like a test case it's trivial to go back and name it.
• Generators. These are in but not fully tested yet. Consider them experimental - but they are very cool and very powerful.
• Custom exception handlers. (C++) Supply handlers for your own exception types - even those that don't derive from std::exception, so you can report as much detail as you like when an exception is caught within Catch. I'm especially pleased this went in - given the name of the library!
• Low build time overhead. I've been aggressive at keeping the compile-time footprint to a minimum. This is one of the concerns when using header only libraries - especially those with a lot of C++ templates. Catch uses a fair bit of templates, but nothing too deeply recursive. I've also organised the code so that as much as the implementation as possible is included in only one translation unit (the one with main() or the test runner). I think you'll be pushed to notice any build-time overhead due to Catch.
• Many fixes, refactorings and minor improvements. What project doesn't have them? This is where a lot of the effort - possibly the majority - has gone, though. I've wanted to keep the code clean, well factored, and the overhead low. I've also wanted it to be possible to compile at high warning levels without any noise from Catch. This has been challenging at times - especially after the Single Evaluation work. If you see any Catch-related warnings please let me know.

Are we there yet?

As well as my own projects I've been using Catch on a large scale project for a bank. I believe it is already more than just a viable alternative to other frameworks.

Of course it will continue to be refined. There are still bugs being found and fixed.

But there are also more features to be added! I need to finish the work on generators. I'd like to add the tagging system I've mentioned before. I need to look at Matchers. Whether Catch provides its own, or whether I just provide the hooks for a third-party library to be integrated, I think Matchers are an important aspect to unit testing.

I also have a stub project for an iPhone test runner - for testing code on an iOS device. Several people have expressed an interest in this so that is also on my list.

And, yes, I will fill out the documentation!