The Cost of a Test Suite

This is a topic my team has been feeling a lot lately. It is near and dear to my heart. We have been supporting and maintaining a project for over a year, having started it as a greenfield initiative. In our proof-of-concept phase, we opted to focus on the features of the tool, rather than testing them. The justification for such a decision is that we did not have enough funding to build the MVP. With the funding we did have, we had to prove the solution was viable in the market.

As a solution became more obvious and the market signals increased in volume, we shifted into MVP mode. In this phase, an integration test strategy was implemented for our API endpoints. The test suite runs on every commit pushed to a branch, limiting merging into the main branch upon failure. All of this worked well for months, as funding increased and excitement grew, additional teams were added to the project to meet the desired demands of the tool. This is when the test suite became too large and unruly to manage.

Not only did the additional teams add new features to the platform, but they also added integration tests by the dozens. Each test increasing the time to execute the entire test suite little by little. With all base functionality of the tool implemented, the test suite grew to a 19-minute execution time. This is 19 minutes for each commit pushed to a branch and on each merge into the main branch.

This cup spilled over once we had a small 2-minute software change, which took 45 minutes to get to our QA environment for testing. This unruly test suite, which is designed to validate the quality of the changes, has now become a major impediment to software delivery.

I will save you the boring details regarding how we decreased the overall test time and not sacrificing the coverage of the tests. Instead, focus on the takeaways.

Use the correct level of test:

Imagine you have test cases for an API endpoint, testing the negative responses returned from the endpoint. These types of tests are easily added to an existing integration test suite. Or should we create a new unit test suite for these test cases? Adding to the integration test suite will be faster, no doubt. As the integration test suite grows, so does the time, with the additional overhead of data setup. A unit test will increase the overall test execution time, but the rate is much slower.

Test the right things:

Many of the tests we uncovered provided little to no value. They were on versioned API endpoints that were not used, meaning we also failed to clean up after ourselves. Some tests were duplicated, as the folder structure and naming conventions were not standardized. This caused engineering confusion and wasted time. As the volume of tests increased, this problem worsened.

Ensure the scope of the test is correct:

Many tests were configured to cover three or more different types of tests. This often required lots of data setup, which took a non-trivial amount of time. As the test would fail, the goal of the test is not abundantly clear, causing the developer to have to dig deeper to understand the test and fix the issue.

Of course, this is not an exhaustive list of considerations while maintaining a test suite. There are many other smaller-scope issues the team ran into while improving test suite performance. I hope the above is helpful while you are working on your testing strategy.