Integration Tests are a Scam

On my first day at the Agile 2009 conference, and after my presentation there on “The Amazing Team Race”, I attended a session by J.B. Rainsberger titled, “Integration Tests are a Scam.” He wasn’t suggesting that integration tests aren’t needed, but rather the way they handled today by the majority of organizations is a lie; a scam. The title, an assertion, was a bit shocking for me but I attended mainly because the teams I currently manage face several challenges with regards to testing. What follows is what I learned from Rainsberger.

One main problem with integration tests have to do with excessive test setups. This causes:

• Messy test structures.
• Assertions to be repeated throughout the structure.
• Programmers/testers to check all the different places to determine where the test precisely failed and why.

Another problem is that integration tests take a long time to execute. Is there a way to address this? Well, one problem Rainsberger asked is to determine how many tests are actually needed to consider your testing truly complete? Assume we have a large-scale application to test, and after considering all the combinations for complete test coverage we come up with 4 million tests. That’s 4 million tests we will never ever write. Therefore if we won’t ever write that many tests, what tests should we even bother to write? Some suggest writing a small percentage of the total possible tests that are geared to test 80-90% of the application’s functionality. Rainsberger throws out 2,000 tests as the number of tests that a team may actually consider writing for this large application. 2,000 tests out of 4 million only cover less than 1 percent of the total possible number of tests. Even by selecting the “right” 2,000 “focused” tests to write, your chances for good coverage run somewhere in the range of 20-80%. And how do you know you’ve covered 20-80% of the application’s functionality in your integration tests? You DON’T!

Here’s where J.B. introduces the concept of “Basic Correctness.” What is “Basic Correctness” of an application? His definition is “…given the myth of perfect technology, do we compute the right answer?” Integration tests should NOT be needed to prove basic correctness of an application. His solution is, “…write only focused object tests for programming tests… this deals with 98% of the problems, leaving you to use the majority of efforts to address the 2% of tests that fail that aren’t address by your focused object tests.”

Steps to writing good focused object tests are:

• Don’t test the platform. Trust the equipment and libraries that are being used.
• When writing a single object to test with four other collaborating objects/points, use four interfaces for those four other points. Interfaces are not the actual collaborating object you are testing, but merely something that behaves like the actual collaborating object. J.B. also notes to only use interfaces for services.
• Leverage the interfaces so you don’t need to actually test the other objects/points.
• Test the single object to speak to itself, aka State Tests.
• Create your focused Collaboration and Contract Tests.

Above is a diagram for what J.B. describes in setting up focused object tests.

What are Collaboration and Contract tests? Well to start, Collaboration tests are what J.B. refers to what others traditionally call integration tests. To have your collaboration tests be focused, you only need to ask two questions:

• Do I ask the collaborators the right questions?
• Can I handle the collaborators’ responses?

For example, perhaps your function needs to determine how many upcoming customer birthdays are there. Your focused test needs to ask of its collaboration point(s) to find the number of customers with a birthday in the next seven days. Your focused tests needs to handle the following responses: zero, one, many, or lots (for capacity and performance reasons perhaps).

When writing your focused collaboration tests, it’s important to handle the responses from the interface and not the actual object/point. Don’t do the backend work for the test. Leverage the interface.

Contract tests are a test for the interface you are leveraging. It’s also a systematic way for setting up tests reliably. They complete the other side of the test and in essence fulfill a contract. The two questions to ask to create a focused Contract test are:

• Can the interface accept the question being asked of it?
• Can the interface supply the responses expected?

In the end you have focused object tests that are succinct and complete in the verification of both what is being asked of its collaborators as well as ensuring that your collaborators can complete their side of the contract by supplying what is needed.

As shown in the diagram above, Rainsberger tells us that the number of tests needed is directly correlated to the number collaboration and contract tests per interface for each related collaborator object to the focused test object. This also eliminates the worry of the numerous combinations of code paths to tests. Therefore when you find a test failure, you only need to check in two places. So when referring back to the example J.B. gave us, as I mentioned in the beginning of this post, about an application with the need to have 4 million tests in order to have complete test coverage:

• The need to create 4 million tests for complete test coverage has been brought down to 10,000.
• Performance time in executing the tests is now shorter.
• Actual effective test coverage can be achieved.
• Pushes engineering towards a good modular design.

Your integration tests are sound and complete when you have:

• State tests
• Collaborator tests
• Contract tests

Now J.B. admits that there may an issue in automating contract tests, but he suggests that there may be some workarounds.

Finally the benefits, which he says he has not been proven but that some PhD potentially could, are:

• When integration tests fail, engineering will only need to check in two places for where and why the failure occurred.
• Integration tests only need to be written into Collaboration tests and Contract tests.
• There is no need to write integration tests during design.

J.B. Rainsberger is heavily involved with the Agile community and has spoken at many conferences. To learn more or follow J.B., be sure to check out his blog at: jbrains.ca.

To see a sample of what J.B. means by Contract tests click here.