mix mutagen
Synopsis
MIX_ENV=test mix mutagen [--scope <target>] [--tests <target|auto>] [--since REF] [--site ID] [--summary PATH] [--summary-max N] [--escalate] [--escalate-max-files N] [--escalate-budget-ms N] [--timeout-ms N] [--no-fail-fast] [--seed N] [--json PATH] [--stream] [--no-progress] [--unsafe-json-outside-project] [--max-sites N] [--budget-ms N] [--max-concurrency N] [--no-host-app] [--no-test-alias] [--boot-timeout-ms N] [--no-source] [--include-host-output]Run mutation testing against one or more scope targets, judged by a chosen
set of tests. The final JSON document goes to stdout or --json <path>.
It lists every mutation and its outcome together with run metadata.
Reading the summary
MIX_ENV=test mix mutagen --since origin/main --summary mutagen.md --json mutagen.jsonThe JSON and Markdown reports give advice for one run. Check first marks functions with only uncovered sites or with at least three evaluated high-confidence mutants and no kills. Worth a look lists remaining high-confidence survivors. Low-confidence survivors stay in the detailed results. Timeouts are counted as unevaluated.
Copy a printed --site <id> command to rerun one site with wider
automatic test discovery. Use the same checkout.
Audit mode
MIX_ENV=test mix mutagen --scope MyApp.Billing --tests auto --json billing.json --summary billing.md
jq '.summary.by_function[]' billing.json
find lib/my_app -name '*.ex' | while IFS= read -r file; do
MIX_ENV=test mix mutagen --scope "$file" --tests auto --json "audit/$file.json" --summary "audit/$file.md"
done
find audit -name '*.json' -exec cat {} + | jq -rs 'sort_by(.summary.counts.kill_rate // 2)[] | [.summary.counts.kill_rate, ([.scope[].module] | join(", "))] | @tsv'Running in CI
Check out full history (fetch-depth: 0). A shallow checkout aborts.
Set MUTAGEN_BASE to the pull request's base commit.
MIX_ENV=test mix mutagen --since "$MUTAGEN_BASE" --json mutagen.json --summary mutagen.mdAppend the Markdown to $GITHUB_STEP_SUMMARY when present and upload
the JSON. Use continue-on-error: true for aborts. See the README for a GitHub Actions
example. The unmapped list shows changed lines outside runtime mutation
sites, including attributes and macros. Each run is advisory.
Host test setup
Use MIX_ENV=test or explicitly select a custom test environment.
The task compiles test support, loads runtime config, and runs the host
test alias entries before the test task. It then starts the host OTP
application and requires test_helper.exs once from each configured
:test_paths entry with ExUnit autorun disabled.
Coverage, baseline, and each mutant use the helper's ExUnit options.
Formatters run on each pass. The report is written before after-suite
callbacks run. Each callback gets its own boot timeout. Cleanup errors
and timeouts go to stderr. A call to System.halt/1 in the helper
prevents reporting.
The default report contains source excerpts and structured facts. Host diagnostic bodies go to stderr. Abort documents include a fixed hint and the step, exception and location when available. Source excerpts may contain secrets committed in the repository. Pattern redaction of opted-in host diagnostics is best-effort.
--json PATH is the guaranteed report channel. The file contains exactly
one JSON document, or NDJSON with --stream, regardless of host output.
Without --json, Mutagen writes the report to stdout and routes the
:user device and known console logger handlers to stderr. The VM can
write directly to file descriptor 1, including when an OTP logger
handler crashes. In-process I/O routing cannot intercept those writes.
Use --json PATH and --summary PATH in CI.
Mix can also write to stdout before the task starts. Do not publish stderr.
Flags
--scope <target>selects a file ending in.ex, a module, orModule.Name.function/arity. Repeat it to add targets. Required unless you use--sinceor--site.--tests <target>selects a test file, afile:linereference, ortag:<name>. Repeat it to add targets, or useautoon its own. Required unless you use--sinceor--site.--summary <path>writes Markdown to a file alongside the JSON report. The files must have distinct paths. As with--json, the path must stay within the project root unless--unsafe-json-outside-projectis set. Use a filename instead of-.--summary-max <int>caps the Worth a look list. Default 5, minimum 0.--site <id>selects one report site. It defaults to automatic tests.--since <ref>plans the runtime mutations touched since the merge base with the given ref.--scopeintersects the diff when supplied. Test selection defaults toauto.--tests autoselects conventional test files and, with--since, test files with code changes and new test files.--escalatesearches for more tests for surviving or uncovered sites.--no-escalateis accepted with no effect for one release.--escalate-max-files <int>caps files per function in each pass. Default 20.--escalate-budget-ms <int>limits time spent on the extra passes. Default- Zero removes the limit for runs without
--site. A single-site run caps this budget at the larger of 30 seconds and ten times the narrow baseline.
- Zero removes the limit for runs without
--boot-timeout-ms <int>sets the deadline for host setup after compilation and the deadline for each cleanup callback. Each coverage and baseline pass gets the greater of this timeout and ten times--timeout-ms. Default300000.--no-sourceomits site source excerpts, file and function from aggregate and stream results and compile errors. Scope, test and coverage paths, module names and source-derived IDs remain.--include-host-outputincludes captured compiler and test stderr and abort messages after best-effort redaction and truncation.--no-test-aliasskips the host test alias pre-steps. Mutagen ignores the arguments on thetestentry and the return values of alias functions. It warns if the alias has entries and notestentry.--timeout-ms <int>overrides the per-mutant deadline in milliseconds. By default the deadline is three times the baseline duration, with a 5000 ms floor. Mutagen warns if a requested timeout is shorter than the baseline.--no-fail-fastruns the full cited suite for each mutant. By default, the mutation phase stops executing tests on the first failure. A failure recorded before the deadline kills the mutant even if the full suite would time out. Baseline and coverage have no failure cap. The JSON fieldmeta.fail_fastrecords the mode and defaults totrueon argument errors. Survivor and kill equivalence assumes the tests run independently. Tests that set up or clean up state for other tests depend on ExUnit's ordering when the seed changes. Skipping one can change later mutants' verdicts, warnings, and taint, even when a later mutant runs every test. Use--no-fail-fastfor these suites.After cited files load, Mutagen sets the seed, serial execution, failure cap and resolved filters. It keeps host formatters, adds its own when collecting failures and restores the host settings after each phase. Other host options stay active.
--seed <int>— ExUnit seed, propagated to every test-running phase. Default0. See Constraints.--json <path>— write the final JSON document to<path>instead of stdout. The document always ends with a single newline. The path is canonicalised before any mutation runs:..segments and NUL bytes are refused at parse time, and the resolved path must stay inside the project root unless--unsafe-json-outside-projectis also passed. Symlinks whose target escapes the project root are refused.--unsafe-json-outside-project— opt-in to writing--jsonoutput outside the project root. CI integrations targeting an artifacts directory above the project root pass this flag; everyday use should leave it off. When set, a one-shot warning naming the resolved target lands on stderr at run start.--streamemits per-site NDJSON lines withstartandendenvelopes on the JSON sink. The last line is the aggregate document without akindfield.--no-progresssuppresses the human-readable progress feed on stderr. By default, the feed appears when stderr is a TTY.--max-sites <int>— upper bound on enumerated mutation sites for one run. Default10000. The enumerator aborts withabort_reason: "too_many_sites"before the runner starts when the cap is exceeded; narrow--scope(or raise the cap) to proceed.--budget-ms <int>— aggregate wall-clock budget for the mutation phase, in milliseconds. Optional (default: unbounded; the per-site--timeout-msis still enforced). When the budget elapses the runner stops dispatching new sites and emits atruncated: truepartial report.--max-concurrency <int>— per-site mutation dispatch cap. Default1(fully serial, v1.0-equivalent). Values greater than1are experimental and add warnings to the report and stderr because real ExUnit/:cover backends can produce incorrect kill/survive classification and corrupted coverage under parallel dispatch. The report records the limit asmeta.max_concurrency.--no-host-app— skip starting the host OTP application in the runtime preamble. Reserved for self-test sandboxes and libraries that intentionally need a minimal boot; the default is to start the host app so Repo/Sandbox, PubSub, Mox, and supervised test support are available. Runtime config and the helper still load when the app is skipped.
Caps
--scopeand--testseach accept at most 100 occurrences. Excess is refused at parse time withabort_reason: "too_many_targets".--max-sitescaps enumerated mutation sites (default 10_000).--budget-ms(optional) caps aggregate wall-clock for the mutation phase. The per-site--timeout-msstill applies.
Examples
# Single file scope, single test file
MIX_ENV=test mix mutagen --scope lib/foo.ex --tests test/foo_test.exs
# Multiple scopes, tag-based tests, custom timeout
MIX_ENV=test mix mutagen --scope MyApp.Foo --scope lib/bar.ex \
--tests tag:fast --timeout-ms 10000
# Module-and-function scope, deterministic seed, redirected output
MIX_ENV=test mix mutagen --scope MyApp.Foo.bar/1 --tests test/foo_test.exs \
--seed 42 --json out/mutagen.jsonConstraints
- Tests run serially (
max_cases: 1) in every phase. There is no--parallelflag in v1 (seemutagen.decision.serial_execution_and_seed). --seedcontrols ExUnit test ordering only. Mutation enumeration order is independent of the seed.mutagen_exruns in-process (same BEAM as your suite); see Caveats for the consequences (state drift, no self-mutation).
Requires Elixir 1.18.0 or later and Erlang/OTP 27 or later.
Exit Codes
0means Mutagen wrote a report, including incomplete runs and runs with surviving mutants.2means Mutagen aborted and wrote an error document to stdout or--json <path>. The string fieldabort_reasonidentifies the error.
A host callback calling System.halt/1 decides the exit code. During
after-suite cleanup it can leave a completed report with aborted: false
and a non-zero exit.
JSON Schema Pointer
Each aggregate result and NDJSON "result" line carries confidence.
The rating applies to the whole mutator. with_swap, pipeline and
guard_drop rate low. The rest rate high. Both levels count toward
aggregate totals and the kill rate.
The aggregate's meta.timings_ms records elapsed time for boot, coverage,
baseline, mutation and the full run. When comparing otherwise identical
reports, exclude this field.
See the README for examples from the CLI and
.spec/specs/json_schema.spec.md for the schema contract.
A survivor means “the tests you selected miss it”. First rerun with a
wider --tests selection before writing a test.
Known Caveats
- State drift on
use SomeModule. Modules using compile-time DSLs can drift between baseline and mutation runs; the JSONwarningsarray names the affected modules. - Macro mutation slowdown. Mutations inside macro-heavy modules recompile dependents; expect longer per-site runs.
- Equivalent mutants. Some mutations are semantically equivalent to the original; they survive every test by definition. This is a known limit of mutation testing, not a tool bug.
- Mutation IDs use scheme 2. Formatting and unrelated edits preserve
IDs when the file path, enclosing definition, normalized expression,
and occurrence among identical sites stay the same. Both JSON and
NDJSON use report schema version
"2". IDs and reports from earlier versions cannot be compared with version 2. Compare the schema, ID scheme, and Elixir and OTP versions before comparing IDs. Formatter migrations (mix format --migrate) can rewrite the syntax and change IDs. When an identical expression is added earlier in a function, later IDs can shift. Persist and comparefunctionandbefore_sourcealongside IDs. --no-jsonis not supported in v1. Pretty terminal output is deferred to v1.1 (seemutagen.decision.no_pretty_output_v1); usejqfor now.--seedcontrols ExUnit ordering only. It does not seed mutation enumeration. Given the same source, enumeration order is stable.--scopecolon syntax is unsupported.file.ex:Moduleis rejected withabort_reason: "colon_syntax_unsupported"(seemutagen.decision.scope_syntax_simplified).- Self-mutation is refused.
--scope MutagenEx.*orMix.Tasks.Mutagenexits withabort_reason: "self_mutation_refused"(seemutagen.decision.self_mutation_refused). --jsonpaths are canonicalised inside the project root. Paths with..segments or NUL bytes exit withabort_reason: "unsafe_json_path"at parse time. Symlinks whose target escapes the project root exit the same way at canonicalisation time, before any mutation runs. CI workflows writing artifacts outside the project root must pass--unsafe-json-outside-projectexplicitly; a one-shot stderr warning then names the resolved target.--max-concurrency > 1is experimental. Parallel dispatch shares ExUnit.Server, the Code.Server, and:coveracross per-site tasks. It can produce incorrect kill/survive classification and corrupted coverage on real ExUnit/:cover backends; the safe path is the default,--max-concurrency 1.tag:NAMEcharset is bounded.--tests tag:NAMEmust match~r/\A[a-z][a-z_0-9]{0,63}\z/— lowercase ASCII, digits, or_, up to 64 chars, with a lowercase first character. Targets outside this charset are refused withabort_reason: "invalid_tag_name"before any test resolution runs. This is the atom-table-DOS bound (seemutagen.cli.r11): CI loops likemix mutagen --tests tag:$(uuidgen)cannot grow the BEAM atom table.- Caps on input and output volume.
--scopeand--testseach accept at most 100 occurrences (abort_reason: "too_many_targets").--max-sitescaps enumerated mutation sites (default 10_000;abort_reason: "too_many_sites").--budget-ms(optional) caps the aggregate mutation wall-clock and yields atruncated: truepartial report when exhausted.
Summary
Types
Pluggable collaborators for the mix task's state machine. Each entry is a module atom the task uses instead of a hard-coded call, so every error path AND the happy path can be unit-tested by injecting a stub module that captures its arguments and returns canned shapes.
Functions
Test seam: run the task with a custom dispatch table.
Types
@type dispatch() :: %{ optional(:cli) => module(), optional(:otp_release) => String.t(), optional(:elixir_version) => String.t(), optional(:project_root) => Path.t(), optional(:clock) => (atom() -> integer()), optional(:scope) => module(), optional(:tests) => module(), optional(:ast_cache) => module(), optional(:coverage) => module(), optional(:enumerator) => module(), optional(:baseline) => module(), optional(:mutation) => module(), optional(:reporter_ok) => module(), optional(:reporter_error) => module(), optional(:io) => module(), optional(:reporter) => {module(), atom()}, optional(:pipeline) => {module(), atom()} }
Pluggable collaborators for the mix task's state machine. Each entry is a module atom the task uses instead of a hard-coded call, so every error path AND the happy path can be unit-tested by injecting a stub module that captures its arguments and returns canned shapes.
Per bw mutagen-wrd.33 (closes F9), the 11 production phase keys carry
module atoms (not {module, function} tuples). The Mix task calls
mod.callback(args) directly; the callback for each slot is defined
by a dedicated behaviour under MutagenEx.Pipeline.*Facade.
The keys correspond to the orchestration stages of
mutagen.mutation_pipeline:
:cli—mod.parse(argv)(MutagenEx.Pipeline.CliFacade, defaultMutagenEx.CLI).:scope—mod.resolve(target, opts)(MutagenEx.Pipeline.ScopeFacade, defaultMutagenEx.ScopeResolver).:tests—mod.resolve(tests, opts)(MutagenEx.Pipeline.TestsFacade, defaultMutagenEx.TestSelector).:ast_cache—mod.load(files, opts)(MutagenEx.Pipeline.AstCacheFacade, defaultMutagenEx.AstCache).:coverage—mod.run(input)(MutagenEx.Pipeline.CoverageFacade, defaultMutagenEx.CoverageRunner).:enumerator—mod.enumerate(cache, scope, covered, opts)(MutagenEx.Pipeline.EnumeratorFacade, defaultMutagenEx.MutationEnumerator).:baseline—mod.run(input)(MutagenEx.Pipeline.BaselineFacade, defaultMutagenEx.Baseline).:mutation—mod.run(input)(MutagenEx.Pipeline.MutationFacade, defaultMutagenEx.MutationRunner).:reporter_ok—mod.emit_report(report)(MutagenEx.Pipeline.ReporterOkFacade, defaultMutagenEx.JsonReporter).:reporter_error—mod.emit_error(report, reason)(MutagenEx.Pipeline.ReporterErrorFacade, defaultMutagenEx.JsonReporter).:io—mod.emit(iodata, exit_code, Config.t())sink for the final document (MutagenEx.Pipeline.IoFacade, defaultMutagenEx.Pipeline.DefaultIo— writes to stdout orConfig.json_pathand halts the VM withexit_code).
S1 shipped a two-key legacy shape (:reporter, :pipeline) for
early CLI testing. Those keys carry {module, function} tuples and
route through the legacy code path (run_legacy/2), which uses
apply/3 to dispatch — that is the explicit back-compat fallback
carve-out of mutagen-wrd.33's "no apply/3 in the Mix task"
contract.
Functions
@spec run([String.t()], dispatch()) :: :ok | {:aborted, atom(), MutagenEx.JsonReporter.Report.t()} | {:error, MutagenEx.CLI.reason(), map()}
Test seam: run the task with a custom dispatch table.
Production code calls run/1, which threads through default_dispatch/0.
Tests pass a partial dispatch via run/2 — any key absent from the
passed map falls back to the default.
The return value mirrors what run/1 does observably:
:okon a successful parse and pipeline run that produced a success-shape JSON document.{:aborted, reason, %Report{}}on any abort path. The reporter and io collaborators have already been invoked with the iodata + exit code; the return value is for tests that want to assert which abort reason fired without parsing the emitted JSON.
run/1 itself does not return one of these — it calls System.halt/1
via the default io collaborator on every path. run/2 lets tests observe
without halting the test VM by overriding the :io collaborator.