JUnit testing tips and tricks 3/3: Changing test execution order in JUnit 4 & JUnit 5

Using JUnit for your daily unit testing? Part 3 of our series on JUnit tips and tricks covers specifying a sequence for your tests to be executed in.

In part 1 of this series, we showed how to ignore test cases in JUnit 4 and 5. Part 2 covered how to tag and filter test cases in both JUnit 4 and JUnit 5.

In this post, we’ll finish up our series on JUnit testing tips and tricks with a tutorial on changing the order of test execution in both JUnit 4 and 5.

How to change the order of test execution in JUnit 4?

First off, a quick piece of advice: in general, you shouldn’t need to change the order of test execution at all. That’s because your unit tests should be able to be run independently of each other. If one unit test depends on the execution of another, that typically leads to a test maintenance nightmare that is best avoided. Debugging/fixing issues may also get very difficult if you don’t know about the requirement to run tests in a specific order.

If, however, you do find yourself in a situation where you need to order your tests (for instance, when you prefer to execute faster test cases first), this guide will help you do that.

JUnit 4.11 and up provide the @FixedMethodOrder annotation and the MethodSorters.class to facilitate setting an order for test execution. Make sure you import the package org.junit.runners.* which contains the class MethodSorters. MethodSorters offers three enum values with the following use cases:

Using MethodSorters.DEFAULT in JUnit 4

The default option executes tests in a deterministic but not predictable order:

import org.junit.FixMethodOrder;
import org.junit.Test;
import org.junit.runners.MethodSorters;

@FixMethodOrder(MethodSorters.DEFAULT)
public class TestingOrderWithMethodSortersDefault {
    @Test
    public void TestB() {
        System.out.println("Executing TestB");
    }
    @Test
    public void TestA() {
        System.out.println("Executing TestA");
    }
    @Test
    public void TestC() {
        System.out.println("Executing TestC");
    }
}

Executing the above test suite yields:

Executing TestA
Executing TestB
Executing TestC

It is a coincidence that the tests execute in alphanumerical order. Again, remember that the order of execution may stay the same since all that JUnit states is that tests are executed in a deterministic but not predictable sequence.

Using MethodSorters.JVM to execute tests in a random order in JUnit 4

Going with the JVM option has tests execute in a random order every time the test suite is run.

import org.junit.FixMethodOrder;
import org.junit.Test;
import org.junit.runners.MethodSorters;

@FixMethodOrder(MethodSorters.JVM)
public class TestingOrderWithMethodSortersJVM {
    @Test
    public void TestB() {
        System.out.println("Executing TestB");
    }
    @Test
    public void TestA() {
        System.out.println("Executing TestA");
    }
    @Test
    public void TestC() {
        System.out.println("Executing TestC");
    }
}

Our first run for instance yielded:

Executing TestB
Executing TestA
Executing TestC

Process finished with exit code 0

Using MethodSorters.NAME_ASCENDING to sort tests in a lexicographical order in JUnit 4

For more predictability and control over the sequence, you can sort tests by name in ascending order, and name your tests as you see fit:

import org.junit.FixMethodOrder;
import org.junit.Test;
import org.junit.runners.MethodSorters;

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class TestingOrderWithMethodSortersNameAscending {
    @Test
    public void TestB() {
        System.out.println("Executing TestB");
    }
    @Test
    public void TestA() {
        System.out.println("Executing TestA");
    }
    @Test
    public void TestC() {
        System.out.println("Executing TestC");
    }
}

Tests will be executed in ascending order of method name, so the sequence will not vary with each test run. Executing this test case yields the expected output of:

Executing TestA
Executing TestB
Executing TestC

Process finished with exit code 0

How to change the test execution order in JUnit 5?

JUnit 5 offers built-in order classes to help you order test cases. In JUnit 5, you can use the MethodOrderer interface to order test methods in a given class. You can configure a MethodOrderer globally (e.g. for the entire test suite) using the junit.jupiter.testmethod.order.default configuration parameter.

For more granularity, use the @TestMethodOrder annotation locally (e.g. for the selected test class only). See more on this below. The @Order annotation helps further refine test method ordering by enforcing a defined sequence for test execution.

You have three built-in options when using MethodOrderer:

Using MethodOrderer.MethodName to sort test cases alphanumerically by name in JUnit 5

Ordering test cases by name is pretty straightforward:

import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;

@TestMethodOrder(MethodOrderer.MethodName.class)
public class TestingOrderWithMethodOrderMethodName {
    @Test
    public void TestB() {
        System.out.println("Executing TestB");
    }
    @Test
    public void TestA() {
        System.out.println("Executing TestA");
    }
    @Test
    public void TestC() {
        System.out.println("Executing TestC");
    }
}

Executing the above test suite yields the following output:

Executing TestA
Executing TestB
Executing TestC

Process finished with exit code 0

Using the @Order annotation with MethodOrderer.OrderAnnotation in JUnit 5

Here’s how you can order test methods numerically using OrderAnnotation based on the values in @Order:

import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;

@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class TestingOrderWithTestMethodOrder {
    @Test
    @Order(2)
    public void TestB() {
        System.out.println("Executing TestB");
    }
    @Test
    @Order(3)
    public void TestA() {
        System.out.println("Executing TestA");
    }
    @Test
    @Order(1)
    public void TestC() {
        System.out.println("Executing TestC");
    }
}

Running the above test yields:

Executing TestC
Executing TestB
Executing TestA

Process finished with exit code 0

Using MethodOrderer.Random to execute tests in a pseudo-random order in JUnit 5

Want to execute JUnit 5 tests in a random order? Here’s how you can do it:

import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;

@TestMethodOrder(MethodOrderer.Random.class)
public class TestingOrderWithTestMethodOrderRandom {
    @Test
    public void TestB() {
        System.out.println("Executing TestB");
    }
    @Test
    public void TestA() {
        System.out.println("Executing TestA");
    }
    @Test
    public void TestC() {
        System.out.println("Executing TestC");
    }
}

Executing the above test suite yields:

Executing TestC
Executing TestA
Executing TestB

Process finished with exit code 0

How to create a custom order for test execution in JUnit 5?

If you’re using JUnit 5 version 5.4 and up, you can use the MethodOrderer interface to implement a custom order for executing tests.

To do this, you’ll create a custom order class which implements the MethodOrderer interface. Provide an implementation for the method public void orderMethods(MethodOrdererContext context). Because this method is a void method, you need to sort the list of method descriptors directly, which are available via context.getMethodDescriptors().

It is best to take a look at the implementations provided by the JUnit 5 framework in the class MethodOrder such as OrderAnnotation or DisplayOrder, to get some examples of how custom orders can be implemented. Find the code for these over here

Let’s take a look at the implementation of DisplayName:

@API(status = STABLE, since = "5.10")
class DisplayName implements MethodOrderer {

	public DisplayName() {
	}

	/**
	 * Sort the methods encapsulated in the supplied
	 * {@link MethodOrdererContext} alphanumerically based on their display
	 * names.
	 */
	@Override
	public void orderMethods(MethodOrdererContext context) {
		context.getMethodDescriptors().sort(comparator);
	}

	private static final Comparator<MethodDescriptor> comparator = Comparator.comparing(
		MethodDescriptor::getDisplayName);
}

It sorts the list context.getMethodDescriptors() by using the custom comparator comparator, which simply relies on the DisplayName of the individual MethodDescriptors.

Summary

So that’s how you can change the order of test execution in JUnit 4 and JUnit 5. Once again, think twice if you need to use this solution: it’s a best practice to keep unit tests isolated i.e. not dependent on each other. If your unit tests can only be run in a set sequence, you may want to reconsider the way you built your unit test suite.

If you’ve missed them, make sure you check out previous parts of this series:

• JUnit testing tips and tricks 1/3: How to ignore test cases in JUnit 4 and 5?
• JUnit testing tips and tricks 2/3: Tagging and filtering test cases in JUnit 4 & JUnit 5

Need some help creating unit tests in the first place? Check out Symflower!

Make sure you never miss any of our upcoming content by signing up for our newsletter and by following us on Twitter, LinkedIn or Facebook!