Instrument source code or byte code

This is the 3rd post of the “Test Coverage” series.

Test coverage works by instrumenting the code under test to insert probes at the appropriate places (I’ll discuss what is an “appropriate place” in a coming post). A probe is basically a statement that records the fact that the current location has been executed. Instrumenting code will in any case require a slight change in the build process, but at what stage? In fact, either the source code or the object code may be instrumented.

Many, if not most, test coverage tools instrument byte code or object code, probably for lack of a general method for source code instrumentation.

SD’s test coverage tool instead relies on a generic language infrastructure to instrument source code for various languages in a regular way. We believe that source code instrumentation is better in the following fundamental way: test coverage sees the code as it is, not as the compiler has fiddled/filtered it.

Precise location

Source code instrumentation provides precise location information for a covered block (file/line/column start and end). Byte code has at best line-number traceabililty back to source code, and that only if the line numbers aren’t stripped. There are circumstances where multiple blocks occur in a single line, and current and future tools need that precision to pick out the part of the line that is relevant.

Example:

if(...) { a(); } else { b(); }

Byte code instrumenters will tell this line is covered whenever the if statement is reached, even though one of the branches a() or b() is not exercised. This means that depending on coding style, you might think you got 100% coverage when you did not, and that’s a critically wrong conclusion.

This is the same code as above, with different test coverage results when using byte-code instrumentation:

if(...) {
    a();
} else {
    b();
}

Precise location is also required to display branches with enough precision.

This matters even more with advanced test coverage methods such as MC/DC, where you
want to track fragments of the conditional itself.

Assert statements

Source code instrumentation is not confused by “assert” statements, which translate into something like this in Java:

assert(cond) ==> if (~cond) then throw assert_exception;

Byte code instrumentation sees this as an “if” and computes coverage of the then clause,
which is zero in working programs. This causes people with lots of asserts to get low
test coverage, which seems like a pretty big loss. Some of the byte code instrumenter
tools look for this idiom and suppress coverage.

Annotations

Source code instrumentatoin can pick up annotations in the form of structured comments in the original code to guide the instrumentation process (used in SD’s COBOL test coverage tool). This allows the addition of test coverage statements (such as “dump to file”) into the instrumented code when such comments are encountered by the instrumenter.
This makes it easier to add permanent, custom code to your application to support
test coverage, that has no impact on your production code (they’re just comments).

Further reading

This paper describes how to use program transformations to install test coverage probes into source code, with code samples:

Branch Coverage for Arbitrary Languages Made Easy, Ira. D. Baxter [PDF file].

[Thanks to Ira Baxter for the raw material of this post]

This entry was posted in Test Coverage. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>