About

This is the M Cubed Software weblog. To find out more about us head to our about page.

Search

Feed

Thoughts on Unit Testing

Posted on 01/02/2009 at 02:27 AM in

Unit testing is a topic that brings up a lot of very strong opinions, especially in the Mac community. But even with all these opinions, the fact is that a lot of developers don't do it. I've started using unit tests in Lighthouse Keeper 1.1 so I thought I would give an overview of how and why I've used them.

One of the most annoying parts of software is testing. It takes forever and is boring and repetitive. And worse of all, you often miss stuff, often stuff that is hard to track down. One way to make part of this easier is to use unit tests.

There are two extremes to the argument over whether you should write unit tests:

- "You should write tests for absolutely every single method and/or function in your code"

- "Unit tests are boring, a waste of coding time and don't test everything anyway so don't write any"

As with anything, the truth is often found in the middle ground. Some of the 2nd quote is true: unit tests ARE boring (at least for most devs) and they don't test everything. However, they may not be a waste of coding time if used properly as they can save you a huge amount of testing and debugging time.

The 1st quote is also completely true in some cases. If you are writing software for planes, nuclear power stations, rockets etc I would expect you to be writing unit tests for everything (as well as various other sorts of tests). If something goes wrong in these situations due to a program error it potentially lead to the loss of human life.

Unit Tests in Lighthouse Keeper

But let's assume you aren't writing software that could potentially kill someone. Do you need to cover every part of your code with unit tests? No. In Lighthouse Keeper I just have tests for the backend that works with the Lighthouse API. There are two reasons for this:

1. In an MVC app, the model layer is the easiest layer to tests as it should work completely in isolation from the rest of the application.

2. This is where most of the complex code that is.

Your time is precious so you want to get the biggest benefit from the time you spend writing tests. As such you should target the parts of your code that are most likely to gain bugs, especially ones that may not be immediately noticeable in normal use.

In the backend of Lighthouse Keeper I have a lot of code that parses data, or filters content based on certain logic. These can easily be broken by accident while I'm working on something else. These are ideal candidates for writing unit tests. But the objects that simply store data don't have any tests related to them at all as they only have simple accessor methods. These methods are unlikely to gain bugs and if they do they'll be pretty easy to detect and fix.

Future Unit Testing

While this is where my unit testing is now, I'm wanting to expand it further. While there is a lot of complex code in my model classes, there is also some easily breakable code in other layers of my code. Unfortunately this is much harder to test.

One of the big problems is designing for testing. Some parts of this are sensible and result in better, cleaner code. But I've found some parts actually lead to more code as you're building to make things easy to tests, not to make things simple for you. Model classes are very often designed well for testing by their nature, with controllers you have to put more effort in.

One thing I won't be heading towards is Test Driven Development. This is where you write your tests first and then code later. To me I find that a far too rigid form of development. It requires a lot of thought and planning before you start coding which goes against the way that I work, which involves a lot of chopping and changing. It leads to a more haphazard way of developing but I feel it helps me find the best solutions to problems. Ironically, while this form of development isn't ideal for doing Test Driven Development, it is ideal for automated tests such as unit tests as they help prevent me breaking stuff I know works.

OCRunner

One of the big problems with unit testing in Cocoa is that the tools are very weak. Xcode uses the same UI for unit tests failures as for compiler errors and warnings. It also requires you to go into the raw build log to see individual test details, multi-line errors, timings and other information.

To solve this I spent a few days coding up a replacement UI and this resulted in OCRunner. It's far from complete but it's already very useful for getting more information about your unit tests than Xcode provides in its UI. And best of all, it's released under the MIT licence so you can play around with it yourself. The source is available on Launchpad.

Unfortunately other commitments (aka my degree) has prevented me from working on it for a while, but when I next have some time I'm hoping to add some more features and polish and get a built version online for people to download. If you have any feature requests of find any bugs, then be sure to file them in the bug tracker.

(7) Comments

Comments

I think you misunderstand the purpose and process of TDD. It isn’t to code the tests first, and then begin coding your application. It’s to code *one* test first, run the test to ensure it fails, and then code the part of your app that gets that test to pass (and no more than necessary to get it to pass that one test). If you tried to develop it the way you explained it, you’d indeed have a heavy process.

Try it the one test, fail, code to pass, pass, way for a week and see if TDD might not actually be a better fit for your chop-chop coding style.

Posted by Alfonso Guerra on 01/02/2009 at 03:46 AM

I agree with Alfonso. I don’t think you’re really grasping TDD.

http://en.wikipedia.org/wiki/Test-driven_development

Posted by Milo Hyson on 01/02/2009 at 11:03 AM

Oh, I understand TDD isn’t writing all tests first, but generally you write a test for something before you write code. This leads to you simply coding to pass tests which is bad in two ways:

1. You may end up writing needless tests. You don’t need to tests accessor methods

2. It adds friction to making major changes to your code.

I know the idea is that you update your tests before major changes so that you can test that the changes are right, but that assumes that you know EXACTLY what the changes will entail.

TDD requires thinking of things at a code level rather than from a user level. If you work that way then it’s fine, but many people work in more abstract top down ways. If I want to change something major I say: “I’m going to change X”. Then I get to work on changing it, but until I’ve finished I have no clue what the final code will look like, plus it’s often quicker to test it is working while developing by simply running it.

For most people, the best use of unit tests is to make sure existing code in other parts of your app doesn’t break when you change some other code. When developing, actually running the code to test it makes the process much more fluid and agile.

By the time someone doing TDD has updated all their tests, started changing their code, realised they have to do something differently, updated their tests again, changed their code, repeated this several times and finished, I’ve updated the app, written a few tests to make sure things don’t break again, shipped and gone down the pub for a pint while the money rolls in.

Will my software have more bugs this way? Maybe. They’ll likely be unforeseen bugs which unit tests wouldn’t catch anyway. But I can develop software much faster as I’m writing much less code.

Posted by Martin Pilkington on 01/02/2009 at 08:42 PM

“...it’s often quicker to test it is working while developing by simply running it.”

This is a materials problem (tools, language, etc.) not really an effect of TDD. I don’t doubt that much of what you say is true when developing for the mac w/ objective-c, Cocoa and Xcode but when I consider the smalltalk roots of TDD, some of your assertions don’t ring true to me.

Is it possible that your language is shaping your thinking ala Sapir–Whorf?

Posted by Robert Evans on 01/02/2009 at 09:27 PM

Of the 268 million games sold in the U.S. last year, 132 million (49%) were for Nintendo platforms, compared to 86 million (32%) for Sony consoles, and 50 million (19%) for Microsoft. Furthermore, the Wii software tie-in ratio isn’t looking too shabby: an even six games per system, an identical figure to that of the PlayStation 3.
Software Computers - unique catalogue of new useful computers software products with detailed description.

Posted by Computers Software on 06/02/2009 at 07:49 PM

You’re doing it wrong.

Do you start writing code without thinking of what you’re trying to accomplish? Really? You just type without an inkling of what the code you’re typing is expected to do, what result it will produce for those invoking it?

If you’ve got some result in mind for the code you’re about to add, then you write a test for that result. By thinking how you want the result to be determined by your test, by thinking how to make the result easiest to be tested, you’re more likely to write efficient & clean code than just writing code by the seat of your pants.

We have anglophiles here, those who are ga-ga over all things English. I think you have the opposite problem. It’s best not to emulate the cowboy-mentality our hackers glorify. Hang those spurs up, pardner! You’re coding for money now. Your money, and your time.

Posted by Alfonso Guerra on 06/02/2009 at 08:06 PM

Rethinking our conversations, I guess you really don’t know what the result is going to be when you’re coding. Or better stated, you know what it should do, but you’re unsure how it should do it.

You’re pseudocoding.

Stop it.

There are two solutions for you.

First, because you’re pseudocoding, or fleshing out everything you want the methods to do before you really get a grasp of how it will work, you’re putting too much behavior into them. If you use the TDD process to code, you’ll do a similar thing, but in a much slower, more deliberate fashion. Write a test for a method returning a result. Create a method that returns 1 result. Write a test that the method doesn’t return some result. Add code to the method that checks when it’s supposed to return the result and returns a different result when the check fails. Write another test that the method returns a correct result under a different set of circumstances. Add code to the method that handles that set of circumstances.

Writing code in this manner ensures you’re slowly bulding up the functionality you’re looking for, that you accurately internalize how your classes interact, and that your code executes properly each step of the way.

What it doesn’t offer is the ability to view the big picture of how it all works out before you decide it’s done. It isn’t done, and likely will never be all done. That’s the nature of code, and the sooner you realize it is, the sooner you’ll be able to write code that takes this into account. Make your code easy to change, and you make your code more useful.

The second technique is to write out the steps your method(s) need to perform on Post-Its, a single step for each sheet. Then lay out the sheets in the order you think they should be executed. This will help you to visualize the method in its entirety before actually wasting time coding it. Later you’ll progress to using colored sheets for loops, conditionals, exceptions, etc.