Mutation testing is a process in which faults (known as mutants) are intentionally injected into your codebase, tests are run and any faults which are not detected by your tests are known as survivors (or surviving mutants). Survivors can often indicate that your test assertions are not focused on expected behavior or might be missing altogether. Whereas code coverage can tell you which code was exercised during a test run, mutation testing can provide some insight regarding the strength of the assertions and whether or not certain classes of bugs would be caught.

With increasingly more code and tests being authored by coding agents, it seemed like a good time to build a tool like this. Mutation testing is not new, but many of the tools are either single-language specific or haven’t been designed with an agentic workflow in mind, so those are itches we wanted to scratch. Models continue to become more powerful and effective, but at some point there is still a “who watches the watchmen” scenario and we hope Flawd can provide one check towards that balance.

We recently ran Flawd against 10 established open source projects and you can find the full writeup here: https://fixture.dev/writing/we-mutation-tested-10-projects

Also, you can see what a sample report looks like here: https://fixture.dev/flawd/sample-report . Flawd can also generate machine readable reports that can be handed off to coding agents for test strengthening tasks.

Feel free to ask my anything about Flawd and thanks for having a look!