In dev, a missing test always passes

What if a you lose an automated test? What if that was testing a critical functional area? This article discusses this and for unit tests implements a Java @RequiresTest annotation

How to lose a test

Can’t lose a test? Sure you can. Test reports just give you counts and changes? Who looks at that? No one. In a long time span, test maintenance can develop warts and tests are silently deleted since they are failing and its too hard to fix or no time available. The original developers of a component may have gone on to other things and that tender loving care is nowhere to be found.

Coverage reports don’t help much with this. Unless there are drastic changes in test results or coverage levels, no one looks at them, they become just another management spreadsheet number, or developer hipster feel-good schtick.

Who loses tests

Sure, for tools, utilities and highly focused systems, especially FOSS, this is not likely. The rapid change and larger development teams ensure full use of test tools. In these projects there is more likely to be a level of “test-infected” developers.

For other kinds of systems, like IT projects, testing will be forgotten when the scat hits the fan, or when the test evangelist moves on or gives up. Testing will just rely on manually repeated testing of a local facsimile of the target system and waterfail test department testing.

Quoting Fowler here “Imperfect tests, run frequently, are much better than perfect tests that are never written at all.”, I would add or those that are never run.

Does it matter

For real-world large applications that quickly become legacy, any missing tests can prove disastrous. A missing test would make any potential defect show up much later. Later is too late and costs more to fix.

Ironically, the best example of the lost of tests are legacy systems that have no automated tests, a de-testable system. In such a system, defects are found in late stage waterfall phases, or worse in production.

What should be tested

Ideally everything would have a valid unit/functional/integration test. In reality this is not cost effective and some would argue that some things should not be tested. For example, it is claimed that getter/setters do not need tests. (Clearly in a language with true properties, this is true. Java, not.)

So if some things should not be tested, what should be? And if those things that should be tested are not tested?

Options

If missing tests are a concern, what can be done? As in many system decisions, it depends: What kind of tests, when are the tests run, who manages the tests, what kind of test monitoring, and so forth.
The following are just a few options that could be considered.

Monitoring of missing tests

The Continuous Integration system or the build tools it invokes present and track missing tests. Missing tests are considered a failure and must be acted on: confirming or removing from consideration.
There is no need to create this missing test list. The build system adds to this list as each build is performed.

Required tests database

For critical systems or subsystems a required test database could be used. This would be more of a management and tool issue since an ongoing project may change many tests during its duration.
Required tests specification is not a new concept. Hardware systems have always had such a concept and even go further by having Built-In Self Tests.
Note that one argument against recent standards and by extension a ‘required test database’ is that this is not congruent with modern agile processes.

Requires test annotation

For “devtests”, using xUnit frameworks, it is much easier to indicate what should be tested. This can be easily made part of the build configuration system and run as part of the test phase. To make more resilient to bit rot, the source code itself should store this information. In the Java devcology this can be done for unit tests using annotations.

Example

In listing 1, a developer has decided that two methods should have tests. So the methods are annotated with @RequiresTest.
Listing 1, an application class with annotations
[sourcecode language=”Java”]
package com.anywhere.app;

import com.octodecillion.test.RequiresTest;

/**
*/
public class Foo1 {
@RequiresTest
public void fum1(){
// stuff
}

@RequiresTest
public void fum12(){
// stuff
}
}
[/sourcecode]

Below in listing 2, a Java implementation of a RequiresTest annotation is shown. It uses the same approach I used in Search Java classpath for JUnit tests using Spring, except now the filter checks for a different annotation. These two could be combined into one implementation that searches for tests or requires annotation.

Funny, I did not annotate the RequiresTestAnnotationScanner with @RequiresTest.

Listing 2. A requires test annotation
[gist id=”98756136cc30d15c1b67″]

T. C. Mits 108

Blog