There are some widely used unit testing frameworks and useful tools that can greatly support your unit testing efforts. Here’s a top list of the ones we find best!
Most developers today are familiar with unit testing: it has become a fundamental, and increasingly widely used technique to check logically isolated, small bits of software code (e.g. a specific behavior) at a time to see if everything works as expected. Since unit testing focuses on, well, units, you’ll need a way to scale up your unit testing when you start applying unit tests in the case of complex applications, or if you take a TDD approach. That’s where unit testing frameworks can be very useful.
What is a unit testing framework?
Unit tests help you test the objects of your application in isolation, making it a lot easier to locate the cause of failing test cases. Since you’re testing smaller bits of code at a time with unit tests, there’s a good chance that you’ll have to create, manage, and run lots of unit tests when building more complex applications. Unit testing frameworks help you write and manage unit tests at scale, also offering functionality for test execution, analysis, and reporting.
While you can certainly create a homegrown unit testing framework for yourself or your team, most developers rely on one of the popular (mostly open-source) frameworks available. Despite developers relying on a wide variety of tools for all sorts of testing purposes, the unit testing framework landscape is actually quite easy to navigate. This may surprise you as much as it surprised us when working on this article, but it turns out that 85% of developers use JUnit! The most popular viable alternative is TestNG.
When applying unit testing, you’re also likely to need to use other tools such as mocking frameworks, test coverage tools, and mutation testing tools. In this article, we’ll be providing a short comparison of JUnit and TestNG, the two most widely used unit testing frameworks for Java software. We’ll also provide an overview of popular unit test mocking frameworks, test coverage tools, and mutation testing tools available.
Comparing JUnit vs TestNG: similarities and differences
Common features of JUnit and TestNG
JUnit and TestNG have a lot in common besides serving a similar purpose. Both are open-source tools that are support by all major Java IDEs (IntelliJ, Eclipse, NetBeans), and both work well with Maven (which is by far the most popular build system, used by 72% of developers). They both use the well-known system of annotations and assertions, but TestNG offers two assertion mechanisms (see below).
TestNG and JUnit both let you use test suites, and they both support parameterized testing. Each tool provides the option to use timeouts for tests (see the documentation of TestNG and JUnit 5), let you ignore tests (TestNG, JUnit 5), and they both enable you to specify an order for test execution (TestNG, JUnit 5). Both tools offer reporting features: JUnit provides XML reports and needs to be integrated with for instance Maven for HTML reports, while TestNG has built-in HTML reports.
JUnit vs TestNG: the differences
There are also some key differences between the two frameworks. One of these is the use of assumptions to skip certain tests based on specified conditions (assumptions), which is available in JUnit, but is not supported by TestNG. For conditional test running in TestNG, you’ll use the SkipException type. If you are looking for a complete example on how to use skip exceptions, this is a good resource to check out.
As mentioned above, TestNG provides two assertion mechanisms: Soft Assertions e.g. when failing assertions are recorded but the execution continues when the assertion fails without an exception being thrown, and Hard Assertions, e.g. when an assertion fails, the whole test run fails.
For a long while, JUnit has not offered native support for parallel test execution – in this regard, TestNG had a clear edge. With JUnit 5.3, that changed: in that version, parallel test execution is an experimental feature. The annotations used by JUnit and TestNG are slightly different but are used similarly. Learn about the annotations in TestNG and those used in JUnit 5. For a deep dive into more detailed information about each framework, check out the documentation of JUnit and TestNG.
Besides the actual unit testing frameworks, you’ll very likely be using a range of different tools to support your testing efforts. Below, we’ll be introducing the most commonly used mocking frameworks, test coverage tools, and mutation testing tools – plus a pro tip to accelerate your unit testing!
Popular mocking frameworks for Java
Objects being tested often have dependencies on other (in some cases, more complex) objects. To make sure you adequately test these objects, you’ll want to check their core logic in isolation – that is, without your test being influenced by the behavior of the objects that your tested method has dependencies on. That’s precisely what mocking helps you achieve.
Mocking is used to isolate a unit of code/functionality and lets you simulate the behavior of real objects that the functionality being tested depends on. With mocking, you can do this in a controlled way and can use different implementations of dependencies to test different behaviors of your software. A common example for mocking is to mock out database queries, e.g. you can let a database query successfully return a list of items in one test to test your happy path, and in another test return an exception to test your code to handle errors.
There are various mocking frameworks out there. Here are the most common ones, with small code examples to help you get started:
Mockito
Mockito is by far the most popular mocking framework, with about 45% of Java developers using it. Whether you need fakes, stubs, mocks, or spies, Mockito lets you easily create test doubles and define the behavior you expect from these mock objects. To show you how Mockito works, let’s take a look at this example from softwaretestinghelp.com, in which we’re building an application that calculates the total marks (in all subjects) for a student:
class StudentScores {
public void calculateSumAndStore(String studentId, int[] scores, ScoringDatabase databaseImpl)
{
int total = 0;
for (int score: scores) {
total = total + score;
} // write total to DB
databaseImpl.updateScores(studentId, total);
}
}
interface ScoringDatabase{
void updateScores(String studentId, int total);
}
Source: https://www.softwaretestinghelp.com/mockito-tutorial/
Suppose we want to unit test calculateSumAndStore, in which case we’ll need a database to store the total value. If we have no database, we can’t perform the unit test – unless, that is, if we use Mockito for instance as follows:
@Test
public void calculateSumAndStore_withValidInput_shouldCalculateAndUpdateResultInDb()
{
// Arrange
studentScores = new StudentScoreUpdates(mockDatabase);
int[] scores = { 60, 70, 90 };
Mockito.doNothing().when(mockDatabase).updateScores("student1", 220);
// Act
studentScores.calculateSumAndStore("student1", scores);
// Assert
Mockito.verify(mockDatabase, Mockito.times(1)).updateScores("student1", 220);
}
Source: https://www.softwaretestinghelp.com/mockito-tutorial/
Like that, we’ve provided a mockDatabase object for the method we’re aiming to test, and we’ve also defined what that mocked object is supposed to do: (Mockito.doNothing().when(mockDatabase).updateScores(“student1”, 220);).
PowerMock
PowerMock can be thought of as an extension to mocking frameworks like Mockito, adding powerful features to vanilla mocking frameworks. It uses the PowerMockito API to support Mockito. Used by 9% of developers, it is a popular tool that enables the mocking of static methods, constructors, final classes and methods, private methods, as well as the removal of static initializers. Overall, PowerMock lives up to its name: it is more powerful than Mockito.
One drawback of PowerMock is that it had its last major release 4 years ago and is since then mainly just maintained. As a result, it is not compatible with JUnit 5. If you want to add PowerMock for Mockito to your maven project you need to add the following dependencies: powermock-module-junit4 and powermock-api-mockito.
Also, consider that PowerMock uses a custom class loader. When your software makes it into production, that custom class loader shouldn’t be there at runtime – then again, since you’re only using PowerMock for unit or integration tests, this shouldn’t cause any issues (your system tests should never rely on mocking).
Let’s go through a quick example from knoldus.com to see how PowerMock can be used to test a static method. We start off with the DemoUtil class which includes a static method called demoStaticMethod:
public class DemoUtil {
public static int demoStaticMethod(int argument){
return argument;
}
}
Source: https://blog.knoldus.com/mocking-static-methods-with-powermock/
The DemoImpl class uses the static method from the Demo class:
public class DemoImpl {
public String useDemoStaticMethod(){
DemoUtil.demoStaticMethod(1);
return "function executed successfully";
}
}
Source: https://blog.knoldus.com/mocking-static-methods-with-powermock/
Our test cases for this static method will look as follows:
@RunWith(PowerMockRunner.class)
@PrepareForTest(DemoUtil.class)
public class DemoImplTest {
@Test
public void testUseDemoStaticMethod throws Exception{
PowerMockito.mockStatic(DemoUtil.class);//For mocking static functions
Mockito.when(DemoUtil.demoStaticMethod(1).thenReturn(1);
DemoImpl demoImpl= new DemoImpl();
assertEquals(demoImpl.useDemoStaticMethod(),"function executed successfully");
}
}
Source: https://blog.knoldus.com/mocking-static-methods-with-powermock/
Test coverage tools for Java
Next up, test coverage tools! Most of us have an intuitive understanding of what code coverage means. In general, code coverage is a metric that measures the extent to which your code is covered by executing tests. Diving deeper though, we find that there are actually different types of coverage:
- Line coverage tracks if specific lines in your code base are executed while running your automated tests.
- Branch (/ decision) coverage determines whether branches/decisions of control statements, e.g. “if” statements, are executed. Specifically, having 100% branch means that we provided tests for both outcomes (true and false) of every branch.
- Condition coverage is used for conditions, e.g. the conditions of “if” statements. Specifically, having 100% condition coverage means that we provide tests for all possible outcomes of all conditional expressions in the code.
- Modified condition / decision (MC/DC) coverage is, essentially, the combination of branch and condition coverage.
What are the different code coverage types?
Test coverage tools help you gauge the extent to which your code is covered with test cases by recording specifically which portions of code were executed when running your tests. Therefore, they make it easy to identify untested parts of your code, and extend coverage to them. Since high test coverage leads to a reduced chance of unidentified errors slipping under the radar, code coverage tools can help deliver robust and high-quality applications. But they can also improve code base efficiency, as unused (dead) code that isn’t covered by unit tests may possibly be just unnecessary code that you can remove from your code base.
Some IDEs like IntelliJ IDEA actually have built-in coverage tools for Java. If you’re looking for more functionality or are not relying on your IDE for another reason, here’s a list of some of the most well-known and useful test coverage tools for Java.
JaCoCo
The name, which stands for Java Code Coverage, says it all: JaCoCo is an open-source toolkit for measuring and compiling reports on code coverage in a Java environment. It is able to analyze coverage for instructions, branches, lines, methods, types, and can also provide insights into cyclomatic complexity.
JaCoCo provides a visual report highlighting sections of code that are run and an analysis of various types of code coverage:
- Instruction coverage focuses on Java byte code instructions, providing key information about the extent to which code has been executed.
- Branch coverage: For all the if and switch statements in your code, JaCoCo can determine how many of all the possible branches have been executed.
- Cyclomatic complexity provides heuristics into how many test cases you’ll need to fully cover your application by analyzing the minimum number of paths that will generate all possible paths through the method being tested.
- Line coverage: JaCoCo also calculates coverage for individual lines of your code. In case at least one instruction assigned to the line in question has been executed by a test, JaCoCo considers the line as executed.
- Method coverage: Similarly, methods are considered executed if at least one instruction in the method has been executed.
- Class coverage: When at least one method in a class has been executed, JaCoCo considers that class as executed, adding to overall coverage.
It’s also important to note that while most coverage tools require pre-instrumentation to be able to record the execution of code by test runs, JaCoCo can also track coverage without pre-instrumentation by using a Java Agent. To find out how to configure offline instrumentation, see the relevant section of JaCoCo’s documentation.
Overall, JaCoCo is quite easy to configure and has native or 3rd party integrations with a number of tools including Gradle, Jenkins, IntelliJ IDEA, NetBeans, TeamCity, and more. If you’re working with Maven, take a look at the documentation to find POM files that configure your project in a way to track code coverage when executing unit tests.
JaCoCo generates readable and easy-to-digest visual reports on code coverage, and it can actually generate these reports in a number of formats including HTML, CSV, and XML.
Cobertura
Cobertura is another open-source coverage tool for Java. Note that while it’s still maintained, it is no longer actively developed. The tool is based on JCoverage, and like JaCoCo, it provides cyclomatic code complexity analysis. It works for Maven, Gradle, Ant, and the command line. Cobertura’s developers claim that it produces the “prettiest” (e.g. most easy to read) output, with its (HTML or XML) reports being easily filtered and ordered as needed. When compared with JaCoCo, however, resources available online claim that Cobertura tends to be a little slower due to its memory use and also supports fewer coverage metrics (branch and line coverage only).
Mutation testing tools for Java
Mutation testing is a core technique in fault-based testing, a testing principle focused on anticipating errors in the system and designing test cases to catch those errors.
The goal of mutation testing is to check the robustness of your test suite: if code is mutated, test cases should fail. If they don’t, that means there may be a problem with your test suite. The process of mutation testing is quite simple. It involves making small modifications to the program (creating mutants). Your tests are meant to detect this difference in behavior between the original vs the mutant, and reject the change by failing (the mutant is killed). If the mutant lives (e.g. the change is not detected), it’s time to check your test suite.
We use mutation testing because traditional test coverage only provides information on what code is executed by your test suite – but it doesn’t check whether your tests can detect errors in the executed code. Think of mutation testing as “testing your tests”: it helps check that each statement in your code is meaningfully tested, rather than just executed.
Since mutants are created by making minimal changes to your code, you may need to create lots of mutations. Doing that manually would not be feasible, but thankfully, there are mutation testing tools out there that can do this for you. Even so, creating mutants may take a while, so don’t expect instant results with mutation testing!
µJava
µJava is an early mutation testing system built by two universities in Korea and the United States and released in 2003 (as JMutation). The developers of µJava don’t guarantee support, bug fixes, or making improvements to the tool – they offer it on an “as-is” basis (the last round of bug fixing on the tool was done in 2016).
That said, it’s still a useful tool for generating mutants for traditional and class-level mutation testing. µJava can test individual classes and packages of multiple classes, and lets you automate the creation and execution of mutants. The tool also provides an analysis of mutation coverage.
[PIT] (pitest) (http://pitest.org/)
PIT (also known as pitest) is a mutation testing framework for Java that works with Ant, Maven, Gradle, and other build tools. Its developers claim that PIT is fast and easy to use – in any case, one clear advantage it has over µJava is that pitest is still actively developed and support is provided. PIT combines line coverage and mutation coverage data in easy-to-read reports, with coverage information color-coded for individual lines of code.
+1: Unit test generator tool for Java, Spring, and Spring Boot
Unit testing & mocking frameworks are essential when it comes to unit testing Java code. However, you’ll still need to write your unit tests manually. Not only is that a lengthy and effort-intensive process, it also leaves some room for error.
A JUnit test generator like Symflower can help you overcome those limitations by automatically generating smart unit test templates that provide you with the boilerplate code you need – just fill in the right values and your unit tests are ready to run. Symflower may also be able to automatically create complete unit tests with meaningful values – all without your intervention. Overall, Symflower is an automatic JUnit test generator plugin for IntelliJ IDEs, VS Code, and CLI that developers use for boosting their Java, Spring, and Spring Boot testing activities.
Find out how to get started with Symflower to automate the creation of unit test templates and to complement your test suite with auto-generated JUnit test cases:
Make sure you stay updated on future content about software development and software testing from us! Sign up for Symflower’s newsletter, and follow us on Twitter, LinkedIn or Facebook!