Automatically generating values for unit testing can be a huge time-saver. This post features a comparison of two alternative strategies for generating test values: fuzzing and symbolic execution.
Unit testing can be an effort-intensive process: not only do you have to write the tests manually, but you also have to figure out the right values for your test scenario. Automated test value generation can help to drastically accelerate this process while saving you a ton of manual effort. However, finding the most accurate method for figuring out the right test values is actually a huge technical challenge.
Fuzzing vs symbolic execution in software testing
In a software testing scenario, tests are created with specific test values that trigger certain code paths in the application to verify their behavior. Ideally, values would be chosen to test all possible execution paths. But how do you pick the right values to do that in a way that balances the required time & effort with sufficient accuracy to yield efficient test coverage? While there are other methods for automated test value generation out there, in this blog post, we’ll focus on comparing the widely used fuzzing (random value generation) technique with the rarely used and cutting-edge technology of symbolic execution.
Read our post: Methods for automated test value generation
Both software fuzzing and symbolic execution are general techniques to find values to execute software programs. Note that while fuzz testing has a broader scope and was originally used to test the whole application, in this article, we’re exclusively looking at the application of generation fuzzing to figure out values for unit tests.
What is fuzzing?
Fuzzing (aka fuzz testing) involves generating and injecting random data inputs for a program. It is considered a type of black-box testing. That is, traditional fuzzing doesn’t involve accessing the internal structures of the system under test (SUT). With fuzzing, the goal is to introduce “wrong" (unexpected, invalid, or malformed) data inputs into a system, and repeatedly execute the function under test with these values. The system is then monitored for software errors or crashes that would signal quality issues with the program.
A good fuzzer creates random inputs that are “almost" valid, that is, valid enough so that the parser doesn’t instantly reject them, allowing them to cause unexpected behaviors downstream. These values help uncover corner cases that the developer needs to deal with. That’s why fuzzing is so useful for exposing bugs.
The main benefit of this technique is that it provides a good way to check the robustness and security risks of your system under test. It is a resource-efficient testing method as the fuzzer can just run in the background automatically, saving time and effort costs for the developer.
Fuzzing can help identify bugs that conventional testing may have missed. On the downside, a fuzz testing system can take a while to set up and can generate lots of data. The random value choice that fuzzing involves may make it more difficult for developers to read and interpret the test suite, adding to the burden of debugging.
Looking for specific examples of fuzzing? For Go developers, we have previously reviewed Go version 1.18 which contained built-in support for fuzzing. Those working in Java should check out our post about Jazzer, the most popular Java fuzzing solution.
What is symbolic execution?
The technique of symbolic execution offers an alternative to fuzzing. Rather than assigning random values, symbolic execution involves a computational way of analyzing the program to identify the specific inputs that trigger the execution of each and every software path.
In the symbolic execution process, source code is executed with symbolic rather than actual values, with conditions being collected along the way. Then, a constraint solver is used to analyze if the path constraints can be satisfied and with which values. Testing values are computed that fulfill all these conditions and execute all possible paths to explore all behaviors in your software. Contrary to fuzzing, symbolic execution is considered a type of white-box testing since it involves analyzing the code. As it’s based on mathematical models to compute test values, symbolic execution ensures that all possible paths in the analyzed code base have been explored and covered with test cases.
A good example of a symbolic execution tool is Symflower. Learn about the inner workings of symbolic execution with Symflower in one of our previous posts.
Comparing symbolic execution vs fuzzing: differences & advantages of SE
Symbolic execution has a range of benefits when compared to fuzzing. Here are the key advantages:
Handling complexity
Symbolic execution is theoretically unlimited in terms of both depth and complexity. It is a mathematically precise approach, so using it results in a test suite that aims to explore all possible program outcomes. Fuzzing, on the other hand, applies random values and therefore there’s some chance of edge cases being missed.
Slim test suites and test values
An added benefit of computing rather than randomizing test values is that symbolic execution generates exactly one test case per execution path through the code. With symbolic execution, you won’t see redundant test cases: each test case explores a new path. That’s why using symbolic execution tests results in slim test suites with only necessary test cases. Compare that with the high-volume, random-value test cases of fuzzing and you’ll see why symbolic execution is considered a more efficient and economical means of testing. A fuzzer may generate large volumes of input data, even with each of them executing the same execution path.
Symbolically executing a program with Symflower will result in using minimal test values, which makes it easier to debug a problem once it’s found. For instance, rather than using a large array (for instance, of the size 1024), it will try to generate the smallest possible array to trigger a certain problem.
Deterministic, human-readable values
A huge advantage of using symbolic execution with Symflower is that it is deterministic. That is, every time you run Symflower’s symbolic execution engine over the same piece of code, you’ll always end up with the same values. (Note that not all symbolic execution is deterministic, but Symflower’s engine works that way). Naturally, that’s different with a fuzzer tool that uses random values every time. Human-readable values are another big difference: symbolic execution can generate such values (e.g. “abc” rather than “Ж 𠃯𠀀”), making the test cases easier to understand.
Since fuzzing uses random values, how much test coverage you end up with is purely a matter of luck. In case a specific value is needed to enter a certain path in the code and that value is never randomly chosen, it’s entirely possible that full coverage will never be reached. That’s also why fuzz testing can take a long time to run.
Optimal test coverage
Compare that with symbolic execution, where the right values to execute all paths are calculated. This means that symbolic execution can be a resource-intensive process – but it always aims to provide optimal coverage of your code. Tools like Symflower allow you to run symbolic execution in the background so that creating high-coverage test suites doesn’t block you as you code. One final advantage of symbolic execution-based tools like Symflower: not only does Symflower generate human-readable tests, but it also automatically maintains the generated tests! This lets you use your test suite to check new behavior and avoid regressions when making code changes.
Check out this video to see Symflower in action:
Curious about how much faster testing can be with Symflower? Try it in your IDE for free!
Make sure you never miss any of our upcoming content by signing up for our newsletter and by following us on Twitter, LinkedIn or Facebook!