- What is mutation testing?
- How accurate is mutation testing?
- Comparing mutation coverage with code coverage. What should mutation coverage be?
- How to speed up mutation testing?
I am looking for ways to improve my tests and code quality. On this search, I came across Test Driven Development. With TDD, our test and code quality are too damn high. But still, while we do TDD, there are times when we still can’t predict real-world scenarios.
For example: If we have a String coming from the external system, it can be null.
We can have a repository of known issues and remember them. A form of threat-modelling/chaos engineering for logic. It’s also helpful to offload such a task to an automated test. That test is mutation testing.
What is mutation testing?
Assuming you wrote a test for a piece of code passes. If there are changes to the code, the test should fail.
That is what mutation testing does. Mutation testing mutates (modifies) the code and checks if tests fail. If the test passes, then we know the test isn’t accurate.
Tests not being accurate can mean two things:
- Either the scenario isn’t covered.
- Or assertion isn’t accurate enough.
In both cases, once we have uncovered a mutant, we can update our test and code to handle the situation.
Mutation testing improves both code and tests.
How accurate is mutation testing?
Sometimes mutation testing produces garbage responses. Like when a mutant removes a log line.
Similarly, for specific autogenerated codes, we don’t need mutations.
Sometimes it’s just garbage. But sometimes, it does uncover accurate details.
The accuracy of mutation testing depends on the application, selected mutants, and current test quality.
If the application is simple like CRUD, mutation tests might not uncover anything.
If the types of mutants selected don’t fit your particular application, then mutations might not uncover anything.
Mutation tests may not discover anything useful if your test quality is already high. But that’s the point. Right? A fallback safety might discover things early in the SDLC process.
Comparing mutation coverage with code coverage. What should mutation coverage be?
During a test, a piece of code can be executed. But unless a specific assertion is applied, it isn’t tested.
Code coverage can be high even without any assertion. Mutation coverage cannot be high if there are no assertions.
Mutation coverage depends on code coverage. If a code isn’t reachable, mutating that code will not affect the test.
So aim for high code coverage.
For mutation coverage, my current answer is that sometimes mutation results aren’t good enough.
How to speed up mutation testing?
- Make sure your tests without mutation testing is already fast.
- Exclude POJO classes.
- Select the right mutators. Read “Less is More”
- Use history so that tests don’t repeat.
- Multiple threads