Improving test coverage with parameterized tests in Swift testing

When you subscribe to the practice of test-driven development or just writing tests in general you'll typically find that you're going to be writing lots and lots of tests for pretty much everything in your codebase.

This includes testing that varying inputs on the same function or on the same object result in expected behavior. For example, if you have a function that takes user input and you want to make sure that you validate that a user has not entered a number greater than 100 or smaller than 0, you're going to want to test this function with values like -10, 0, 15, 90, 100, and 200 for example.

Writing a test for each input by hand will be quite repetitive and you're going to do pretty much the exact same things over and over. You'll have the same setup code, the same assertions, and the same teardown for every function. The difference is that for some inputs you might expect an error to be thrown, and for other inputs you might expect your function to return a value.

The behavior you’re testing is the same every single time.

If you prefer learning through video, this one's for you:

With Swift testing we can avoid repetition by through parameterized tests.

This means that we can run our tests multiple times with any number of predefined arguments. For example, you could pass all the values I just mentioned along with the error (if any) that you expect your function to throw.

This makes it quite easy for you to add more and more tests and in turn improve your test coverage and improve your confidence that the code does exactly what you want it to. This is a really good way to make sure that you're not accidentally adding bad code to your app because your unit tests simply weren’t extensive enough.

A plain test in Swift testing looks a little bit like this:

@Test("Verify that 5 is valid input")
func testCorrectValue() throws {
  #expect(try Validator.validate(input: 5), "Expected 5 to be valid")
}

The code above shows a very simple test, it passes the number 5 to a function and we expect that function to return true because 5 is a valid value.

In the code below we've added a second test that makes sure that entering -10 will throw an error.

@Test("Verify that -10 is invalid input")
func testTooSmall() throws {
  #expect(throws: ValidationError.valueTooSmall) {
    try Validator.validate(input: -10)
  }
}

As you can see the code is very repetitive and looks pretty much the same.

The only two differences are the input value and the error that is being thrown; no error versus a valueTooSmall error.

Here's how we can parameterize this test:

@Test(
  "Verify input validator rejects values smaller than 0 and larger than 100",
  arguments: [
    (input: -10, expectedError: ValidationError.valueTooSmall),
    (input: 0, expectedError: nil),
    (input: 15, expectedError: nil),
    (input: 90, expectedError: nil),
    (input: 100, expectedError: nil),
    (input: 200, expectedError: ValidationError.valueTooLarge),
  ]
)
func testRejectsOutOfBoundsValues(input: Int, expectedError: ValidationError?) throws {
  if let expectedError {
    #expect(throws: expectedError) {
      try Validator.validate(input: input)
    }
  } else {
    #expect(try Validator.validate(input: input), "Expected \(input) to be valid")
  }
}

We now have a list of values added to our test macro’s arguments. These values are passed to our test as function arguments which means that we can quite easily verify that all of these inputs yield the correct output.

Notice that my list of inputs is a list of tuples. The tuples contain both the input value as well as the expected error (or nil if I don’t expect an error to be thrown). Each value in my tuple becomes an argument to my test function. So if my tuples contain two values, my test should have two arguments.

Inside of the test itself I can write logic to have a slightly different expectation depending on my expected results.

This approach is really powerful because it allows me to easily determine that everything works as expected. I can add loads of input values without changing my test code, and this means that I have no excuse to not have an extensive test suite for my validator.

If any of the input values result in a failing test, Swift Testing will show me exactly which values resulted in a test failure which means that I’ll know exactly where to look for my bug.

In Summary

I think that parameterized tests are probably the feature of Swift testing that I am most excited about.

A lot of the syntax changes around Swift testing are very nice but they don't really give me that much new power. Parameterized testing on the other hand are a superpower.

Writing repetitive tests is a frustration that I've had with XCTest for a long time, and I've usually managed to work around it, but having proper support for it in the testing framework is truly invaluable.