What are the different code coverage types? Test coverage explained

In certain GitHub repositories, e.g. Apache ShardingSphere, you can find so-called code coverage badges provided by services like codecov.io. You might have an intuitive understanding of what code coverage means, but did you know that there are different coverage types and that the expressiveness of the reached coverage depends on which type you choose? In this blog post, we’ll take a look at what code coverage is, what types of coverage exist, and what they tell you about your software quality.

What is code coverage?

In short, code coverage (test coverage) is a metric for how well your code is tested. It specifies how much of your code base is covered by executing tests. The higher the percentage the better, right? In principle yes, but it ain’t that simple, though! There are different types of code coverages, and some of them are better suited for measuring your software quality than others. It is therefore important to choose the right metric for your project.

This post gives you an overview of the most common coverage types to make an informed decision about the best coverage type to track the code quality for your project.

We briefly explain and discuss line, branch, condition and MC/DC coverage by having a look at the example below.

We also show you that even one of the most sophisticated coverage types out there (MC/DC coverage) does not guarantee flawless programs, and how Symflower can help you to be more confident about the quality of your code base.

boolean isRainbowPossible(boolean hasRained, boolean isSunShining) {
	boolean isPossible = false;
	if (hasRained && isSunShining) {
		isPossible = true;
	}
	return isPossible;
}

You might have noticed that the above example could be simplified to just returning hasRained && isSunShining. We deliberately chose this version to illustrate the different coverage types with a simple running example.

What is line coverage?

Line coverage is the most common code coverage type. It tracks which lines within your code base were executed while running your automated tests. The percentage you get in the end tells you how many lines of your code are covered by your tests. To get full line coverage for our example, a single test case suffices: isRainbowPossible(true, true) == true.

Yey, full coverage! Great, right? Nope! What if we would have the following slightly modified piece of code:

void showRainbow(boolean hasRained, boolean isSunShining) {
	Rainbow r;
	if (hasRained && isSunShining) {
		r = new Rainbow();
	}
	return r.show();
}

We would again get full coverage with showRainbow(true, true), but what if we would call showRainbow(false, false)? That’s right, we would miss an obvious NullPointerException even though we have full line coverage. So we just saw that line coverage is conceptually simple but the simplicity comes at the cost of conveying a false sense of confidence regarding the quality of the code.

What is branch coverage?

The problem we had with line coverage in the showRainbow example is that it does not force us to write tests such that every possible branch is visited. This is where branch coverage, also known as decision coverage, comes in. In order to get full branch coverage for our example we have to provide test cases such that we DO and DO NOT visit the body of the if. In other words, we have to make sure that every outcome of the boolean expression hasRained && isSunShining is considered.

The tests isRainbowPossible(true, true) == true and isRainbowPossible(false, false) == false would therefore give us full branch coverage. Hence, full branch coverage allows us to detect the NullPointerException in our modified example. That’s great! We will however see that there are more complicated code constructs where even branch coverage is not enough.

Coming back to codecov.io, you can check here that they are tracking branch coverage.

What is condition coverage?

Condition coverage seems to be similar to branch coverage but they are not comparable as we will see. Here we are required to provide tests such that every predicate in the condition of a control structure, such as an if, evaluates at least once to true and once to false.

Our example contains a single control structure: if (hasRained && isSunShining). This means that we have to find tests such that after executing the whole test set, hasRained (and isSunShining) was at least once true and once false. We can therefore achieve full condition coverage with the same tests as for branch coverage:

isRainbowPossible(false, false) == false;
isRainbowPossible(true, true)   == true;

However, condition coverage does not imply branch coverage. We reach full condition coverage with the following two tests, but not full branch coverage.

isRainbowPossible(true, false) == false;
isRainbowPossible(false, true) == false;

The other way around does not hold either, that is, condition coverage does not imply branch coverage. Try to come up with your own counter-example!

What is modified condition / decision coverage?

Modified condition / decision (MC/DC) coverage can be seen as combining branch and condition coverage, and is therefore a stronger notion than both of them. It is used in safety-critical projects such as avionics software, and is recommended or required by international standards like ISO 26262, EN 50128, DO-178C and IEC 61508.

In order to achieve full MC/DC coverage we have to make sure that

• every boolean expression in a control statement takes every possible outcome,
• every predicate within a boolean expression takes every possible value, and
• it is shown that each predicate independently affects the outcome of the boolean expression.

In contrast to line, branch and condition coverage, MC/DC coverage introduces semantic requirements for tests. This means for our example, that the tests have to reflect the fact that the absence of either rain or sunshine is enough to keep us from seeing a rainbow.

Let us look again at the two tests in Branch coverage:

isRainbowPossible(false, false) == false;
isRainbowPossible(true, true)   == true;

We can see that these tests satisfy the first two bullet points of our MC/DC criteria as the boolean expression hasRained && isSunShining takes every possible outcome, and every predicate within that expression is at least once true and once false. The third requirement, however, is not fulfilled as the tests do not capture that either hasRained = false or isSunShining = false would suffice to make the boolean expression false.

Recall the second set of tests in Condition coverage:

isRainbowPossible(true, false) == false;
isRainbowPossible(false, true) == false;

These satisfy the second and third requirements of the MC/DC criteria. However, in both tests the expression hasRained && isSunShining does not hold which means the first bullet point is not satisfied.

As you might have guessed, we need a combination of both to achieve full MC/DC coverage:

isRainbowPossible(true, false) == false;
isRainbowPossible(false, true) == false;
isRainbowPossible(true, true)  == true;

With this test set we cover each possibility of how the parameters influence the outcome of the condition in the if-statement, and therefore the outcome of the function. This is the distinguishing feature of MC/DC coverage.

Symflower problem coverage - MC/DC on FIRE

As already mentioned, MC/DC is the standard in safety-critical projects, and it supersedes the other coverage types when it comes to software quality. But is that good enough? Are you confident that your software does not have bugs if you have full MC/DC coverage? Let’s take a look why that’s not the case and what you can do about that!

Let’s have a look at the following cake-piece calculator. Assume we have a cake with dimension cakeSideA × cakeSideB, and you want to cut the cake into pieces of dimension pieceSideA × pieceSideB, how many pieces do we get out of the cake?

public class Cake {
	public static int numberOfPieces(int cakeSideA, int cakeSideB, int pieceSideA, int pieceSideB) {
		if (cakeSideA <= 0 || cakeSideB <= 0 || pieceSideA <= 0 || pieceSideB <= 0) {
			return -1;
		}

		return (cakeSideA * cakeSideB) / (pieceSideA * pieceSideB);
	}
}

We would get full MC/DC coverage for Cake.numberOfPieces(...) with the following test cases:

Cake.numberOfPieces(0, 1, 1, 1) == -1;
Cake.numberOfPieces(1, 0, 1, 1) == -1;
Cake.numberOfPieces(1, 1, 0, 1) == -1;
Cake.numberOfPieces(1, 1, 1, 0) == -1;
Cake.numberOfPieces(1, 1, 1, 1) == 1;

Full MC/DC coverage, pretty great?! But what if we have a gigantic cake with an area of around 2 147 483 646 which is the maximum value an integer can take in Java referred to as Integer.MAX_VALUE?

If, for instance, cakeSideA == 2 and cakeSideB == (Integer.MAX_VALUE / 2) + 1, we would get a so-called addition overflow as cakeSideA × cakeSideB would then be Integer.MIN_VALUE. If that’s the case we would definitely return a wrong number of cake pieces, but Java would not tell you in any way. Maybe even worse: if pieceSideA == 4 and pieceSideB == (Integer.MAX_VALUE / 2) + 1, then pieceSideA × pieceSideB would give us 0 resulting in a division by zero. In that case we would get an ArithmeticException and the cake application would crash.

Now, our cake program was just a toy example, but imagine you are developing a safety critical application, you definitely want to know about such flaws. Even if your application is not safety critical, you don’t want it to crash due to unhandled exceptions.

This is where symflower excels. In addition to autonomously generating tests which give you full MC/DC coverage, symflower also generates tests for corner cases which lead to problematic behavior such as buffer overflows, null pointer exceptions and arithmetic exceptions. This means that symflower tracks down potential bugs and generates reproducers for you.

For example, symflower would (among others) generate the following test which results in an ArithmeticException due to a multiplication overflow.

@Test // (expected = ArithmeticException.class)
public void numberOfPieces() {
	int cakeSideA = 1;
	int cakeSideB = 1;
	int pieceSideA = 4;
	int pieceSideB = 1073741824; // (Integer.MAX_VALUE / 2) + 1
	int actual = Cake.numberOfPieces(cakeSideA, cakeSideB, pieceSideA, pieceSideB);
}

Conclusion: coverage types in programming

We saw that having a high coverage is a good starting point to detect bugs in your code base. However, we also saw that it highly depends on the coverage type how confident you can be about your software quality if you just look at code coverage. Moreover, even with the most sophisticated coverage metric (MC/DC coverage), having 100% coverage does not guarantee a flawless program.

symflower brings you another step forward towards bug free code. It explores all possible paths through your code and generates a test case for each relevant path. The distinguishing feature is that it also includes error paths, that is, symflower tracks down errors caused by problems such as null pointer exceptions, arithmetic exceptions and overflows.

The next post in our blog series about Software Testing will be about unit tests where we answer questions like What makes a good unit test?. We also have a new blog series in the pipeline that introduces symflower in detail and shows you how it advances the state-of-the-art in software testing. Make sure you do not miss out on any of our upcoming posts by signing up for our newsletter.