Go 1.18 - native Fuzzing and Dinosaurs

Last month, the Go language team released the Go 1.18. It contains the much awaited generic support, which we’re very excited to try out. We also noticed some other interesting changes in the beta announcement: the Go language now also has built-in support for fuzzing-based testing. Let’s take a look at this exciting new release, and while we’re there, compare it with the testing capabilities of Symflower.

The running example

Here we have a simple example where we check if a particular animal is allowed to board Noah’s Ark. The function makes sure that heavy dinosaurs are excluded for safety purposes. Save the following source code in a file called valid.go in a directory called ark and change your terminal to this directory.

package ark

type Animal struct {
    Kind string
    Weight int
}

func IsValidAnimalForArk(animal Animal) bool {
    switch animal.Kind {
    case "Dinosaur":
        return animal.Weight < 5000 // They might be too heavy for the ark.
    default:
        return true // All others should be fine.
    }
}

Let’s now explore how to apply both Go’s new fuzzing and Symflower to this example. We’re especially interested in the case where a weighty dino is present.

Testing with fuzzing

Fuzzing is an automated testing technique aiming at uncovering bugs that existing unit tests may have missed. It frequently runs the same code with random input data to uncover corner cases and vulnerabilities. Go’s Fuzzing is augmented with coverage tracking, meaning that input which produced new coverage is preferred during input generation.

Preparations for fuzzing

As Go 1.18 is now officially released, the following tutorial works out of the box as long as you have already upgraded to Go 1.18.

We need to prepare the fuzzing tests by adding a test file called valid_test.go and writing a specialized fuzzing test function. The source code of that file should look like this:

package ark

import "testing"

func FuzzIsValidAnimalForArk(f *testing.F) {
    f.Add("Cat", 4)

    f.Fuzz(func(t *testing.T, d string, w int) {
        if !IsValidAnimalForArk(Animal{Kind: d, Weight: w}) {
            t.Fail()
        }
    })
}

Note that we need to construct the Animal type ourselves, as the Go fuzzer doesn’t work with user-defined types. We also provide a kitty with 4 as her weight as a valid example seed. Finally, we tell the fuzzer to stop once an invalid animal is found.

Execution and obtained tests

To run the fuzzer on our example, we call Go with the new -fuzz flag:

go test -fuzz Fuzz

The fuzzer should now be running, which gives us something like the following output. If you want to stop the fuzzer, just hit CTRL+C.

fuzz: elapsed: 0s, gathering baseline coverage: 0/1 completed
fuzz: elapsed: 0s, gathering baseline coverage: 1/1 completed, now fuzzing with 8 workers
fuzz: elapsed: 3s, execs: 521894 (173379/sec), new interesting: 0 (total: 0)
…
fuzz: elapsed: 20m0s, execs: 117780001 (91660/sec), new interesting: 0 (total: 0)
[Ctrl-C]

We can observe that the fuzzer started generating random inputs with 8 workers, but it wasn’t able to find our special heavy dinosaur within 20 minutes, so we stopped the execution. The reason why this is so hard for the fuzzer is because it’s executing the code with random strings for the Animal.Kind. Even if we limit the number of random strings to the correct length of 8 letters then “Dinosaur” is one of 26^8 = 208.827.064.576 possible strings. The fuzzer was able to try only a fraction of this – 117.780.001 different inputs in 20 minutes. This underlines how important it is for fuzzing to have a good seed corpus (also called a testing corpus). Without it, might one never reach certain areas in the source code. After all, fuzzing is testing randomly and random has no guarantee.

Testing with Symflower

Symflower, similar to fuzzing, aims at finding bugs and corner cases missed during manual unit testing but it employs the method symbolic execution to do so, i.e. it computes inputs instead of guessing them randomly. Additionally, it not just generates input values but complete unit tests that you can integrate in your test suite and debug right away. If you’re interested in how Symflower works under the hood, check out our previous blog post on the core techniques of Symflower.

Preparations for testing with Symflower

We prepare for our Symflower-based approach by simply installing the free Symflower CLI tool. Once the tool is downloaded and installed it is ready to go. You do not need to prepare any test file nor special test function but can start generating tests right away.

Executing and obtained tests

We can simply generate complete unit tests for our example by executing the following command in our current directory:

symflower valid.go

After a few milliseconds, Symflower generated unit tests for our program in a separate file valid_symflower_test.go with the following source code:

func TestSymflowerIsValidAnimalForArk1(t *testing.T) {
    animal := Animal{}
    expected := true

    actual := isValidAnimalForArk(animal)

    assert.Equal(t, expected, actual)
}

func TestSymflowerIsValidAnimalForArk2(t *testing.T) {
    animal := Animal{
        Kind: "Dinosaur",
    }
    expected := true

    actual := isValidAnimalForArk(animal)

    assert.Equal(t, expected, actual)
}

func TestSymflowerIsValidAnimalForArk3(t *testing.T) {
    animal := Animal{
        Kind: "Dinosaur",
        Weight: 5001,
    }
    expected := false

    actual := isValidAnimalForArk(animal)

    assert.Equal(t, expected, actual)
}

Symflower took complete care of the custom type, found the simplest possible case of a heavy dinosaur – one just above the weight limit, and even found two other cases that help us test our implementation.

Roundup

The new native fuzzing support in the Go 1.18 release is a powerful extension that makes finding corner cases and bugs easier. We’ve seen, however, that there are examples where fuzzing has a hard time finding such missed undesired behavior due to the random nature of the technique. Plus it still requires writing manual code and doesn’t support custom types out of the box. Symflower analyzes the code and, with the help of a symbolic execution, automatically generates unit tests that can reflect corner cases perfectly.

The more one tinkers around with fuzzing the more one sees that fuzzing generates weird looking input values. These values are due to the random nature of fuzzing which are fine for reproducing failures, but come with a heavy price as they are not minimal and hence harder to follow for humans. In the upcoming blog post we are looking at how values of fuzzing compare to values of Symflower, and how productive one can be in testing implementations with them. Follow our social media accounts on Twitter, LinkedIn, and Facebook, or sign up for our newsletter.

Thanks for joining us on this small journey about the most recent release of the built-in fuzzing support for Go. If you’re interested in the capabilities of Symflower, please check out the free Symflower CLI for yourself. You can also read the full release post of the new Go 1.18 Beta 1 and check out the other amazing features such as workspace support. If you liked this blog post, you can read on and browse through our recent technical blog posts.