Incremental development with short iterations for more productivity and less frustration

Implementing new features and thinking up the best way to get everything just right is part of what makes software development exciting. In addition, being able to apply our problem solving skills when hunting down bugs and finding neat fixes is quite rewarding. But why is software development sometimes so frustrating at the same time?

You definitely know the feeling of a rising headache or mounting anger because your pull request is almost done but the continuous integration pipeline just won’t run through, or because that “last” bug holds up the whole branch from being merged. And in the end, maybe the commit or pull request you are working on doesn’t even get into production. Your change might not be needed anymore, causing days or even weeks of work being lost. Sound familiar? All of these problems are avoidable and we learned the hard way how. In this blog series, we share how we develop software at Symflower. We created a process that is more efficient during development, feels more rewarding and, on top of that, decreases frustration.

Development process at Symflower

At Symflower, we always strive for less painful ways to achieve the current goals, through automation and through constant improvement of our workflows. In our opinion, both automation and the ability to change are the base for a productive development process. Over the years we constantly changed how we work, which conventions we follow and added automation to decrease our workload and mistakes. If we did not change how we work, we would be stuck with an increasingly blocking process. This does not mean that everything we present with this series is applicable to your needs, but we hope that you find our learnings useful for your project, your team and yourself. Drop us a note at hello@symflower.com with your thoughts! 😃

This blog post considers the following Symflower-related example while going through the development process: Symflower provides a developer tool that analyzes your source code and autonomously generates unit tests so that you only need to check the behavior and fix emerging problems. Let’s assume for the sake of simplicity that all components of Symflower cannot handle “if” statements and fail if they meet one. The goal of the example is to support the analysis of “if” statements for Go source code and generate unit tests for functions that contain “if” statements. Our user story is therefore to implement "if" statements for Go source code to generate unit tests.

The below flowchart depicts how we are tackling this story. It shows the process from issue planning, to committing and merging, the basic phases of software development you deal with every day. In the remainder, we will fill in the details on advantages, how to apply this process and what to pay attention to.

Plan issues with short iterations

In the classic project management literature you will find that an iteration is a slice of the whole functionality that touches all components of the user story. At Symflower, we learned that this definition of an increment is far too big and redefined this learning into the definition of a short iteration:

"An iteration of a user story is a smaller portion of the initial goal, a sub-goal, that already provides a gain. The union of all iterations is the complete user story we intend to implement."

With this definition in mind we can break down our user stories into individual gains aka iterations. Every gain, can be planned, worked on, reviewed and merged individually. Hence, we can make constant progress on a feature and bring constant gains to our users, even though we are not implementing the full user story for every component at once. Also, this makes it possible to implement multiple iterations of the same story in parallel by multiple persons.

When you plan to implement a user story, decide early on how to split the story into multiple iterations from a higher-level aka behavioral perspective. Each iteration should work independently from the next iterations, but can depend on the previous ones. This guidance allows to merge complete iterations individually and brings one important convention to light:

"Never implement multiple user stories at once."

To better understand this convention, think about how many developers are implementing features: diving right in, without a plan. At Symflower we learned that having no plan is one of the biggest mistakes in software development. Sounds logical, right? However, often we developers like to dive right into the code, because we “already know what we need to do”. This is a common mistake. Nevertheless, the goal of planning iterations should not be to list every individual task and change but to think about the essential behavior changes that actually need to happen. Doing so gives you a list of gains, because every behavior change, is a gain for the user, but makes also two things possible:

• a. move non-essential changes to later iterations that you could (and maybe should) even cut from the story and
• b. identify if the user story is possible to implement right now.

The later is especially important, since it can happen that you are not just implementing one user story, but implement a rabbit hole of multiple stories. Better to break such dependencies into their own issues and work on them first. This leads to gains sooner and completing changes sooner reduces everyone’s frustrations.

"Always define each iteration with controllable goals."

At this point you might be wondering how you can identify an individual iteration? At Symflower we learned that the easiest distinction for an iteration is a behavioral test case as its goal. This test case might be at first for an individual component but, with later iterations, tests can grow to an integration and finally to a system test. The level of such test cases simply depends on the size of the story itself and how many components you need to change. Most importantly, keep it small and shallow, because as a result, you have a smaller context of your current tasks as well as an already usable part of the feature in the software.

"Never plan out every iteration at once."

Since every iteration of our user story is now listed with a controllable goal, we can think a single iteration completely through while remembering an important convention: Never plan out every iteration at once. The goal of planning a single iteration is to list every high-level change, especially API changes, that are needed for the current single iteration. If you are including changes for one of the next iterations, you are including changes that are strictly not necessary to complete your current goal. Hence, you can end up with a bunch of unfinished, non-working features and need a long period of time to get them working. Keeping the context small is key for a productive development process. Moreover, if you are implementing changes over other iterations, you are breaking YAGNI, which we learned multiple times, is the most important programming convention ever.

To come back to our example: An if statement can have different appearances. For each of them, we define a test case. Therefore, we define the following ones.

func ifStatement(a int64) int64 {
    if a < 10 {
        return 1
    }

    return 2
}

func ifElseStatement(a int64) int64 {
    if a < 10 {
        return 1
    } else {
        return 2
    }
}

func ifElseIfStatement(a int64) int64 {
    if a < 10 {
        return 1
    } else if a < 20 {
        return 2
    } else {
        return 3
    }
}

func ifStatementWithInit(a int64) int64 {
    if b := a + 5; b == 10 {
        return 1
    }

    return 2
}

As you can see, each test case defines one additional behavior. If we added all test cases at once to our test suite, we would have the implementation of a simple if-then statement (ifStatement) only available after all the test cases worked and our huge pull request was reviewed. Additionally, we might run into problems for test cases that might not even be that important for a first gain for our users. Which makes debugging very frustrating because of the bigger context. Additionally, a big context is a constant waste of time because the scenario that is blocking us might not even be important. On the other hand, when we split them into four iterations, each of them is done fast, more discoverable in reviews, and you get four times the endorphins from accomplishing four different goals.

Moreover, since we are not alone with this user story, we could do the first iteration ifStatement together and then work on ifElseStatement and ifStatementWithInit in parallel. This allows us working as a team, which gives additional endorphins but also creates room to learn from each other. At Symflower we learned to take advantage of such opportunities to onboard team members to new components. Such collaboration greatly reduces the onboarding time of every person.

"Refactor with every iteration."

As we go along with our implementation, we might see better architectural solutions, which were not obvious during the planning or even the first iterations. We learn incrementally about the solution of the overall complex task, which allows us to rethink and refactor our approach. For example, after accomplishing the if-then and the if-then-else test cases and while working on the else-if cascade, you might see the possibility to refactor to have a nice solution for all three options. The great thing about software development is that it is not like architecture of buildings. We do not need to think about everything upfront, but we can improve our solution as we go along.

Additionally, the convention to “never plan out every iteration at once” is especially important for refactoring as it is an optimization problem that can be worked on indefinitely. Only refactor for your current iteration but never for the next iterations. This makes sure that your code quality stays high, while completing the current iteration as soon as possible.

Commit small portions continuously

Now that we planned an iteration, we can finally get down to the implementation. No matter which changes you are doing for your tasks, at some point, you want to commit and maybe even push to the repository’s server. At Symflower, we learned that it matters when you do a commit. Small commits allow for less context during implementation, debugging and reviewing which greatly increases productivity for an iteration. Naturally, one can now immediately ask: What is the best size for a commit? As always, it depends. But here is how we decide the scope of a commit:

"Commit the smallest isolated changes that pass your current test."

Working on just your current test comes naturally when you lean on a TDD (test-driven development) workflow and follow the Beyoncé rule (“If you liked it, then you shoulda put a test on it."). However, the convention of committing only the smallest isolated changes that pass your current test underlines an important advantage during development: When you have a program that passes all tests, and you start to change it and then a set of tests start to fail, it is highly likely that the current changes are the fault for the test failures. Hence, you have a very small context that need to be looked at when you want to fix a new failure and you can even automatically find which commit introduced a specific problem using delta-debugging such as git bisect. Compare such a scenario to the common one where you only commit when all changes are done. The context is then much much bigger. There are even more advantages regarding good commits that we will cover with the next blog post of this blog series.

Coming back to our example we can take a look at the first iteration that needs to be implemented: we have the goal to generate unit tests for the function ifStatement with Symflower. This might include parsing the source code into an abstract syntax tree (AST), converting that AST into a data structure that is understood by our test generation, and supporting “if” statements in the test generation itself. These vague tasks can have various implicit subtasks, each of which should be resolved in its own commit and receive unit tests to fixate and point out the actual behavior.

If you are done with a commit aka subtask, push it to the repository’s server. The CI pipeline should be passing. If not, you know exactly where the error must be: the commit you just pushed. Meanwhile, you can work on the next task locally. In that manner, the unfinished implementation of the next task and its testing process in the pipeline do not interfere with the earlier commit. As a result, the pull request is always in a mergeable state. At Symflower, we learned that pushing changes early and often not just allows for a fast feedback loop, we can also integrate changes earlier since we can often split out early mergers that simply take some commits that are already done of a pull request and merge them. This way, we have even earlier gains, even before completing an iteration. The next section will take a closer look at how to handle such pull requests.

With this workflow, not only are the implementation and debugging processes more structured and, hence, less frustrating, but first and foremost, every time you finish a goal, endorphins are released. Hence, completing many smaller tasks instead of a single big one that spans a whole day of work gives you more moments of accomplishment - and makes you happy. 😊

Pull requests: merge, merge, merge

What to do with a pull request in a mergeable state? Of course, review it. But how can this part made faster?

"The smaller the pull request, the easier it is to implement, review and merge."

First, you already have a small pull request at hand due to splitting the feature into small iterations. And second, no matter how far you come, you can pause and get in what is reviewed up to now. For example if you are not quite done with your inspection at the end of a day, the commits until then can go into an early merger - a new pull request that contains the cherry-picked commits. Or, if some commits still need some discussion, early-merge unrelated changes.

Another use case: Let’s assume there is a bug in the implementation of the first iteration of our example in component B and we need to debug and fix it in order to achieve our goal of generating unit tests for our function ifStatement. Everything up to the implementation of component B can already go in. So, let’s merge the already passing and reviewed commits for component A. The pull request containing the commits for B is then smaller and easier to handle. Not only do the commit numbers shrink, but also the product already contains the implementation of the “if” statement in some components.

In the past, we often had lots of open pull requests at Symflower and wondered how to improve the situation. Unfortunately, sometimes just handling huge requests or working through the feedback is too time consuming for the moment. Something more pressing comes along and the old one is left behind. Hence, it is important to keep pull requests small and break out commits as soon as possible.

Conclusion

The partitioning into small iterations and tasks may look time consuming at first, but splitting a complex task into smaller iterations reduces the context we have to keep in mind greatly. Having a structured workflow eases your development life immensely as well.

Give our suggestions a try, and if you know techniques and workflows that make your day easier, please let us know. We are excited to hear them. Write us a comment or directly to hello@symflower.com .

Since you read this far, you clearly liked this article and are eager for more: Please subscribe to our newsletter today and follow us on Twitter, LinkedIn or Facebook to get notified about the next blog posts or if you are simply into memes.