Welcome to My Weblog

I’ve been meaning to start blogging for quite a while now but just haven’t gotten “a round tuit”. I have lots of excuses including preparing for quite a few conferences as part of my launch as an independent consultant. For those of you who know me as Chief Scientist of ClearStream Consulting in Calgary, Alberta. ClearStream is no longer. Unfortunately, the founder Denis Clelland died in April 2006 and the company hasn’t been the same since. This spring his widow Carol decided to cease operations. All the employees of the company were laid off and have now gone independent.

Being layed off is both a blessing and a curse. The curse is that one doesn’t know where one’s next pay cheque will come from. The blessing is that it opens doors to a range of new opportunities. I’m hoping that this new blog will help open those doors. To that end I’ll be posting my most recent thoughts on the whole area of automated testing as we apply it on agile projects. This will include both traditional unit test automation as part of Test-Driven Development and automated acceptance/customer/functional testing. I also expect to blog a fair bit about test automation strategy including how the unit testing, acceptance testing and other forms of automated testing complement each other.

Test-Driven Development for Embedded Software

Recently I had the pleasure of presenting my two day course on developer unit testing at Sioux Systems in Holland. Sioux works with a lot of clients who build software embedded in hardware devices (like microscopes and in-car GPS units.) I also presented to a larger audience (190 signed up but there was only room for about 120) on the topic “Test Patterns: The road to effective test automation: Hot or Not?”. Several of the participants in the course and the Hot or Not presentation asked “How practical is it to do TDD on embedded software?” and expressed doubts about the practicality of it. The reservations seemed to be related to a few key areas:

1. The fact that the hardware interfaces on which the software depends make it hard to test the software outside the hardware.
2. The lack of space within the hardware to run a unit test framework.
3. Issues around timing and multi-threading.

While these are all good points, none of them is actually a contra-indication for automated unit testing and TDD. We have well-established techniques for decoupling our logic from its dependencies using Test Doubles (stubs, mocks, spies, etc.) The lack of space on the hardware is exactly why we should be considering debugging the software outsid of the hardware. And issues around timing and multi-threading are not unique to embedded software. In fact, these should encourage us to do more unit testing in which we can properly control the timing.

The general principle could best be summarized as “Avoiding DOH!” (I would love to claim that I made up this great term but I first saw it on James Grenning’s blog DOH stands for Debugging On the Hardware except on The Simpsons, where it is what Homer Simpson exclaims after doing something stupid. Now this is a particularly fitting cultural reference because that is usually exactly how we feel after we have spent many minutes or hours debugging some stupid little mistake, especially when we are doing it in a difficult to use environment (like “On the Hardware”!) The true value of TDD is to avoid these DOH moments by Debugging In Simulated Environment (DISE). Once we have eliminated all the stupid little programming errors we always make, then we can go on to testing the software On Hardware.

In a future post I’ll go into more detail about these thoughts.

Automatic Test Code Generation

Recently, I was asked by a visitor to my website http://xunitpatterns.com:

What are your thoughts on using automatic test code generators for new projects? My team is just starting with TDD. We are currently debating the value of automatically generated tests by JTest (which we would enhance) versus home grown tests using EasyMock & Eclipse.

I’ve seen tools like JTest demonstrated at various Agile conferences and they can certainly provide value in helping get a mass of legacy software under control. (Note that many people in the agile community consider any software whose maintainers are not protected by automated unit tests as “legacy software”.) But this question specifically relates to the use of test code generators for new projects doing Test-Driven Development (TDD.) This got me thinking about my position on the whole topic of test generation.

But first, for those of you are not familiar with these test generation tools, here’s a quick, over-simplified summary of their value proposition: You fire up the tool, point it at a chunk of existing code such as a JAR or a DLL and it goes to work. It examines the public interfaces of the classes, does static code analysis on the structure of each method and generates tests that exercise the code. You can learn more by going to the vendor’s web sites (e.g. http://www.parasoft.com/jsp/products/home.jsp?product=Jtest)

So let’s examine these two cases separately:

1. Legacy code base needing some enhancements
2. Building new code using TDD

I can definitely see some potential benefit for teams that have a legacy code base and are afraid to make changes to it. Generating a suite of regression tests for the code as it currently exists will allow them to do experiments such as “What happens if I change the return value of this method? How far and wide does that impact the code base?” The regression tests help “lock down” the functionality already in the code. But how will that help us actually refactor the code? If we go ahead and change the existing code, we should expect some existing tests to break and we’ll have to decide what to do with them. We could just regenerate the test code after we are done but then we really haven’t taken much advantage of them while we are refactoring. As a minimum, we should be looking at the tests broken in each step of the refactoring and/or enhancement to see it there is anything that should alarm us. How easy this will be to do will depend very much on how easy it is to understand the tests, and that depends on how “intent revealing” is the generated test code.

Disclaimer, I haven’t looked at any of the generated test code beyond the trivial examples shown in demos so I’ll leave this decision to you.

One strategy might be to determine whether the broken tests are about the internals or the interface contract (as in Bertrand Meyer’s “Design by Contract”) of the software we are changing. Ideally, we won’t break any of the interface contract tests during our refactoring; if we do then we’ll certainly have to think about the implications on the clients.

Now let’s look at the 2nd scenario, building new software using TDD. Test generation software works by examining the existing software. How will that work when we haven’t written the software yet. Or from a development perspective, TDD is about expressing our intentions of what the software should do when it’s done. This definition of “what done looks like” is what allows us to avoid writing a lot of unnecessary software by only writing just enough code to pass the tests we have already written. To work properly in TDD, the test generator would have to be able to read my mind to determine my intent for the software I’m about to write. I might be able to use the test generator to generate the templates for the Test Methods
once I’ve defined the interface but even then, will it be able to determine all the test conditions for which it needs to generate Test Methods? I’m skeptical on this point but I would certainly like to hear from anyone who has tried these tools in conjunction with TDD.

One final comment I’d like to make is that there is no substitute for hard work and discipline. I’m a great fan of trying to work more effectively and I encourage all the teams I work with to hold regular retrospectives. Retrospectives are just one disciplined way in which teams can reflect on how they develop software and plan experiments that may yield improvements. Trying code generation tools might be just such an initiative. It is important, however, to properly frame the experiment before diving in so that we don’t get into a revolving tool-of-the-month situation that detracts from productivity in the long term as we flit from tool to tool; that certainly wouldn’t help improve the predictability of team’s output as measured by our team velocity.