Instrument source code or byte code

This is the 3rd post of the “Test Coverage” series.

Test coverage works by instrumenting the code under test to insert probes at the appropriate places (I’ll discuss what is an “appropriate place” in a coming post). A probe is basically a statement that records the fact that the current location has been executed. Instrumenting code will in any case require a slight change in the build process, but at what stage? In fact, either the source code or the object code may be instrumented.

Many, if not most, test coverage tools instrument byte code or object code, probably for lack of a general method for source code instrumentation.

SD’s test coverage tool instead relies on a generic language infrastructure to instrument source code for various languages in a regular way. We believe that source code instrumentation is better in the following fundamental way: test coverage sees the code as it is, not as the compiler has fiddled/filtered it.

Precise location

Source code instrumentation provides precise location information for a covered block (file/line/column start and end). Byte code has at best line-number traceabililty back to source code, and that only if the line numbers aren’t stripped. There are circumstances where multiple blocks occur in a single line, and current and future tools need that precision to pick out the part of the line that is relevant.

Example:

if(...) { a(); } else { b(); }

Byte code instrumenters will tell this line is covered whenever the if statement is reached, even though one of the branches a() or b() is not exercised. This means that depending on coding style, you might think you got 100% coverage when you did not, and that’s a critically wrong conclusion.

This is the same code as above, with different test coverage results when using byte-code instrumentation:

if(...) {
    a();
} else {
    b();
}

Precise location is also required to display branches with enough precision.

This matters even more with advanced test coverage methods such as MC/DC, where you
want to track fragments of the conditional itself.

Assert statements

Source code instrumentation is not confused by “assert” statements, which translate into something like this in Java:

assert(cond) ==> if (~cond) then throw assert_exception;

Byte code instrumentation sees this as an “if” and computes coverage of the then clause,
which is zero in working programs. This causes people with lots of asserts to get low
test coverage, which seems like a pretty big loss. Some of the byte code instrumenter
tools look for this idiom and suppress coverage.

Annotations

Source code instrumentatoin can pick up annotations in the form of structured comments in the original code to guide the instrumentation process (used in SD’s COBOL test coverage tool). This allows the addition of test coverage statements (such as “dump to file”) into the instrumented code when such comments are encountered by the instrumenter.
This makes it easier to add permanent, custom code to your application to support
test coverage, that has no impact on your production code (they’re just comments).

Further reading

This paper describes how to use program transformations to install test coverage probes into source code, with code samples:

Branch Coverage for Arbitrary Languages Made Easy, Ira. D. Baxter [PDF file].

[Thanks to Ira Baxter for the raw material of this post]

Posted in Test Coverage | Leave a comment

The Anatomy of Test Coverage

This is the 2nd post of the “Test Coverage” series.

The test coverage process with Semantic Designs’ Test Coverage Tool consists of 3 phases:

  1. Instrument: add test coverage probes to the code.
  2. Compile the code, run the tests and collect test coverage data – just as you would do if not implementing test coverage.
  3. Analyze test coverage data, produce reports and views.

Anatomy of test coverage

Instrument

Instrumentation generates two kinds of files: instrumented source files, and probe reference files (.PRF).

Instrumented source files contain probes at the right places. The notion of “right place” can be quite tricky and depends on the precise notion of test coverage we are using (it’s basically a trade-off between precision and performance, I’ll discuss this in a future post). Usually, we will end up with a probe at the entry point and all possible exit points of each branch. In a language such as Java that has exceptions, every method call can potentially be an exit point for its surrounding block by throwing an exception, so there will be a lot of probes.

A test coverage probe is simply a statement recording that execution proceeded through the current position. Each probe has a unique number, and firing a probe simply amounts to setting the corresponding bit in a array TCV.visited (TCV stand for Test Coverage Vector). 

Instrumented code

Instrumentation also generates one or more probe reference files (.PRF). They contain, in a compressed form, a mapping of each probe number to the corresponding file and location. PRF files are used by the reporting tool to locate test coverage information in the original source code. They are also used for incremental test coverage (I’ll cover this in a future post).

Compile and run

The only change required in the build chain is obviously to compile the instrumented code rather than the original code. Any strategy will do, but typically you would generate all instrumented code under a new folder structure and point the compiler to this new folder.

Running the tests proceeds as usual. Whether your tests are automated or manual, there is no change in the way you run them. The only difference is that the instrumented code will collect test coverage data in TCV (Test Coverage Vector) files.

A TCV file is created for each call to the TCV.dump() method. A call to TCV.dump() is inserted automatically at the end of the main method when there is one, but additional calls can be inserted manually. For example, if using JUnit, you may want to place a call to dump() at the end of each test suite. Another example is a long running application, where you may want to dump() a TCV file every once in a while.

A test campaign will typically result in many TCV files. Note that TCV files may be produced on remote machines (in the case of a distributed or client/server application) and from different source languages (in the case of a multi-language application), as long as you have an instrumenter for each language. Semantic Designs’ test coverage tool can instrument several dozen of mainstream languages, from C++ to COBOL.

A TCV file contains a very strongly compressed version of the TCV bit vector updated by the probes. The probe numbers in a TCV file always refer to a given probe reference file (PRF). Magic numbers in TCV and PRF guarantee that the reference is correct.

Analyze

At the end of a test campaign, you collect all the TCV files dumped by the instrumented code and feed them to a reporting tool.

At a high level, test coverage reports provide concise but critical objective information to managers about the quality of the tests, and thus of the code. The test coverage index can be global, or detailed by module, team or test campaign. Incremental test coverage additionally provides comparisons with earlier test campaigns, making it possible to measure the progress (and the non regression) of the test coverage index over time.

At a detailed level, tests coverage views exhibit the fragments of application source code that have never been tested, and help software developers improving their test suite. An uncovered fragment of code may also be the sign of dead code that should better be removed. There are also legitimate cases where you don’t want the code to be covered by tests, such as the body of assertions.

Posted in Test Coverage | Leave a comment

Test your Tests

“Most organizations today use unit tests in varying degrees.”

- Editorial of the Sept. 2012 issue of DrDobbs journal.

It took a few decades to have the software development community reach a consensus on the utility of units tests and test-based development, and this is a big leap forwards. My bet is that the next big leap will be about testing tests.

Tests are an indication of the quality of the code. You want your unit tests to be green (passing) as often as possible – a red test is the clear sign of a problem in the tested code. Fine, but what the un-tested code? Tests tell nothing about it.

Obviously you want your test to cover most of your code. Leaving a large part untested make the testing process meaningless. This is the role of test coverage tools: they identify which parts of the code have been tested or not, and usually provide some high-level metrics such as percentage of coverage.

Test-Driven Development

Test-driven software development, or at least making sure you have decent tests for your code, has become a standard part of modern software engineering. Tests can even be developed for their own sake, independently of any software development effort. As an example, the Jacks test suite is an independent effort to provide thousands of units tests for assessing the conformance of Java compilers to the official standard.

Unit tests are typically written together with the code and apply to small program entities such as individual methods. They are best run automatically after each build or with the click of a button after a change in the code, using tools such as JUnit. Integration tests help ensure that whole applications will behave as expected in their real environment, and may involve manual steps.

Tests must be Tested

Tests are useful in that they will tell you, to some extent, whether your code behaves correctly. An essential element here is “to some extent”. One can think of many criteria regarding measuring the usefulness of tests. The most obvious one is that all parts of the code have been tested. Whatever the number of tests you run, if parts of the application are never tested, testing will tell you nothing about this part. This is where “Test Coverage” or “Code Coverage” tools become useful.

My bet is that test coverage will soon become as ubiquitous as testing. The test coverage report has already become a standard morning reading for many CTOs.

Test Coverage or Code Coverage?

Checking whether your tests cover your code, or whether your code is covered by your tests, are really the same thing. You will find both terms “Test Coverage” and “Code Coverage” used in the literature.

Index to the test coverage series of postings

In coming posts I will look at various aspects of the test coverage process:

  • The anatomy of test coverage
  • instrumenting byte code or source code ?
  • the many notions of coverage
  • scalability
  • comparing test campaigns
  • distributed and manual test runs
  • testing across language boundaries
  • how to price test coverage tools ?
Posted in Test Coverage | Leave a comment

Index to the Test Coverage series

Published:

  1. Test your Tests
  2. The Anatomy of Test Coverage
  3. Instrument Source Code or Byte Code

In preparation:

  • the many notions of coverage
  • scalability
  • comparing test campaigns
  • distributed and manual test runs
  • testing across language boundaries
  • how to price test coverage tools ?
Posted in Test Coverage | Leave a comment