mix mutagen (mutagen_ex v0.2.0)

Copy Markdown View Source

mix mutagen

Synopsis

MIX_ENV=test mix mutagen [--scope <target>] [--tests <target|auto>] [--since REF] [--site ID] [--summary PATH] [--summary-max N] [--escalate] [--escalate-max-files N] [--escalate-budget-ms N] [--timeout-ms N] [--no-fail-fast] [--seed N] [--json PATH] [--stream] [--no-progress] [--unsafe-json-outside-project] [--max-sites N] [--budget-ms N] [--max-concurrency N] [--no-host-app] [--no-test-alias] [--boot-timeout-ms N] [--no-source] [--include-host-output]

Run mutation testing against one or more scope targets, judged by a chosen set of tests. The final JSON document goes to stdout or --json <path>. It lists every mutation and its outcome together with run metadata.

Reading the summary

MIX_ENV=test mix mutagen --since origin/main --summary mutagen.md --json mutagen.json

The JSON and Markdown reports give advice for one run. Check first marks functions with only uncovered sites or with at least three evaluated high-confidence mutants and no kills. Worth a look lists remaining high-confidence survivors. Low-confidence survivors stay in the detailed results. Timeouts are counted as unevaluated.

Copy a printed --site <id> command to rerun one site with wider automatic test discovery. Use the same checkout.

Audit mode

MIX_ENV=test mix mutagen --scope MyApp.Billing --tests auto --json billing.json --summary billing.md
jq '.summary.by_function[]' billing.json
find lib/my_app -name '*.ex' | while IFS= read -r file; do
  MIX_ENV=test mix mutagen --scope "$file" --tests auto --json "audit/$file.json" --summary "audit/$file.md"
done
find audit -name '*.json' -exec cat {} + | jq -rs 'sort_by(.summary.counts.kill_rate // 2)[] | [.summary.counts.kill_rate, ([.scope[].module] | join(", "))] | @tsv'

Running in CI

Check out full history (fetch-depth: 0). A shallow checkout aborts. Set MUTAGEN_BASE to the pull request's base commit.

MIX_ENV=test mix mutagen --since "$MUTAGEN_BASE" --json mutagen.json --summary mutagen.md

Append the Markdown to $GITHUB_STEP_SUMMARY when present and upload the JSON. Use continue-on-error: true for aborts. See the README for a GitHub Actions example. The unmapped list shows changed lines outside runtime mutation sites, including attributes and macros. Each run is advisory.

Host test setup

Use MIX_ENV=test or explicitly select a custom test environment. The task compiles test support, loads runtime config, and runs the host test alias entries before the test task. It then starts the host OTP application and requires test_helper.exs once from each configured :test_paths entry with ExUnit autorun disabled.

Coverage, baseline, and each mutant use the helper's ExUnit options. Formatters run on each pass. The report is written before after-suite callbacks run. Each callback gets its own boot timeout. Cleanup errors and timeouts go to stderr. A call to System.halt/1 in the helper prevents reporting.

The default report contains source excerpts and structured facts. Host diagnostic bodies go to stderr. Abort documents include a fixed hint and the step, exception and location when available. Source excerpts may contain secrets committed in the repository. Pattern redaction of opted-in host diagnostics is best-effort.

--json PATH is the guaranteed report channel. The file contains exactly one JSON document, or NDJSON with --stream, regardless of host output.

Without --json, Mutagen writes the report to stdout and routes the :user device and known console logger handlers to stderr. The VM can write directly to file descriptor 1, including when an OTP logger handler crashes. In-process I/O routing cannot intercept those writes. Use --json PATH and --summary PATH in CI.

Mix can also write to stdout before the task starts. Do not publish stderr.

Flags

  • --scope <target> selects a file ending in .ex, a module, or Module.Name.function/arity. Repeat it to add targets. Required unless you use --since or --site.

  • --tests <target> selects a test file, a file:line reference, or tag:<name>. Repeat it to add targets, or use auto on its own. Required unless you use --since or --site.

  • --summary <path> writes Markdown to a file alongside the JSON report. The files must have distinct paths. As with --json, the path must stay within the project root unless --unsafe-json-outside-project is set. Use a filename instead of -.

  • --summary-max <int> caps the Worth a look list. Default 5, minimum 0.

  • --site <id> selects one report site. It defaults to automatic tests.

  • --since <ref> plans the runtime mutations touched since the merge base with the given ref. --scope intersects the diff when supplied. Test selection defaults to auto.

  • --tests auto selects conventional test files and, with --since, test files with code changes and new test files.

  • --escalate searches for more tests for surviving or uncovered sites. --no-escalate is accepted with no effect for one release.

  • --escalate-max-files <int> caps files per function in each pass. Default 20.

  • --escalate-budget-ms <int> limits time spent on the extra passes. Default

    1. Zero removes the limit for runs without --site. A single-site run caps this budget at the larger of 30 seconds and ten times the narrow baseline.
  • --boot-timeout-ms <int> sets the deadline for host setup after compilation and the deadline for each cleanup callback. Each coverage and baseline pass gets the greater of this timeout and ten times --timeout-ms. Default 300000.

  • --no-source omits site source excerpts, file and function from aggregate and stream results and compile errors. Scope, test and coverage paths, module names and source-derived IDs remain.

  • --include-host-output includes captured compiler and test stderr and abort messages after best-effort redaction and truncation.

  • --no-test-alias skips the host test alias pre-steps. Mutagen ignores the arguments on the test entry and the return values of alias functions. It warns if the alias has entries and no test entry.

  • --timeout-ms <int> overrides the per-mutant deadline in milliseconds. By default the deadline is three times the baseline duration, with a 5000 ms floor. Mutagen warns if a requested timeout is shorter than the baseline.

  • --no-fail-fast runs the full cited suite for each mutant. By default, the mutation phase stops executing tests on the first failure. A failure recorded before the deadline kills the mutant even if the full suite would time out. Baseline and coverage have no failure cap. The JSON field meta.fail_fast records the mode and defaults to true on argument errors. Survivor and kill equivalence assumes the tests run independently. Tests that set up or clean up state for other tests depend on ExUnit's ordering when the seed changes. Skipping one can change later mutants' verdicts, warnings, and taint, even when a later mutant runs every test. Use --no-fail-fast for these suites.

    After cited files load, Mutagen sets the seed, serial execution, failure cap and resolved filters. It keeps host formatters, adds its own when collecting failures and restores the host settings after each phase. Other host options stay active.

  • --seed <int> — ExUnit seed, propagated to every test-running phase. Default 0. See Constraints.

  • --json <path> — write the final JSON document to <path> instead of stdout. The document always ends with a single newline. The path is canonicalised before any mutation runs: .. segments and NUL bytes are refused at parse time, and the resolved path must stay inside the project root unless --unsafe-json-outside-project is also passed. Symlinks whose target escapes the project root are refused.

  • --unsafe-json-outside-project — opt-in to writing --json output outside the project root. CI integrations targeting an artifacts directory above the project root pass this flag; everyday use should leave it off. When set, a one-shot warning naming the resolved target lands on stderr at run start.

  • --stream emits per-site NDJSON lines with start and end envelopes on the JSON sink. The last line is the aggregate document without a kind field.

  • --no-progress suppresses the human-readable progress feed on stderr. By default, the feed appears when stderr is a TTY.

  • --max-sites <int> — upper bound on enumerated mutation sites for one run. Default 10000. The enumerator aborts with abort_reason: "too_many_sites" before the runner starts when the cap is exceeded; narrow --scope (or raise the cap) to proceed.

  • --budget-ms <int> — aggregate wall-clock budget for the mutation phase, in milliseconds. Optional (default: unbounded; the per-site --timeout-ms is still enforced). When the budget elapses the runner stops dispatching new sites and emits a truncated: true partial report.

  • --max-concurrency <int> — per-site mutation dispatch cap. Default 1 (fully serial, v1.0-equivalent). Values greater than 1 are experimental and add warnings to the report and stderr because real ExUnit/:cover backends can produce incorrect kill/survive classification and corrupted coverage under parallel dispatch. The report records the limit as meta.max_concurrency.

  • --no-host-app — skip starting the host OTP application in the runtime preamble. Reserved for self-test sandboxes and libraries that intentionally need a minimal boot; the default is to start the host app so Repo/Sandbox, PubSub, Mox, and supervised test support are available. Runtime config and the helper still load when the app is skipped.

Caps

  • --scope and --tests each accept at most 100 occurrences. Excess is refused at parse time with abort_reason: "too_many_targets".
  • --max-sites caps enumerated mutation sites (default 10_000).
  • --budget-ms (optional) caps aggregate wall-clock for the mutation phase. The per-site --timeout-ms still applies.

Examples

# Single file scope, single test file
MIX_ENV=test mix mutagen --scope lib/foo.ex --tests test/foo_test.exs

# Multiple scopes, tag-based tests, custom timeout
MIX_ENV=test mix mutagen --scope MyApp.Foo --scope lib/bar.ex \
            --tests tag:fast --timeout-ms 10000

# Module-and-function scope, deterministic seed, redirected output
MIX_ENV=test mix mutagen --scope MyApp.Foo.bar/1 --tests test/foo_test.exs \
            --seed 42 --json out/mutagen.json

Constraints

  • Tests run serially (max_cases: 1) in every phase. There is no --parallel flag in v1 (see mutagen.decision.serial_execution_and_seed).
  • --seed controls ExUnit test ordering only. Mutation enumeration order is independent of the seed.
  • mutagen_ex runs in-process (same BEAM as your suite); see Caveats for the consequences (state drift, no self-mutation).

Requires Elixir 1.18.0 or later and Erlang/OTP 27 or later.

Exit Codes

  • 0 means Mutagen wrote a report, including incomplete runs and runs with surviving mutants.
  • 2 means Mutagen aborted and wrote an error document to stdout or --json <path>. The string field abort_reason identifies the error.

A host callback calling System.halt/1 decides the exit code. During after-suite cleanup it can leave a completed report with aborted: false and a non-zero exit.

JSON Schema Pointer

Each aggregate result and NDJSON "result" line carries confidence. The rating applies to the whole mutator. with_swap, pipeline and guard_drop rate low. The rest rate high. Both levels count toward aggregate totals and the kill rate.

The aggregate's meta.timings_ms records elapsed time for boot, coverage, baseline, mutation and the full run. When comparing otherwise identical reports, exclude this field.

See the README for examples from the CLI and .spec/specs/json_schema.spec.md for the schema contract.

A survivor means “the tests you selected miss it”. First rerun with a wider --tests selection before writing a test.

Known Caveats

  • State drift on use SomeModule. Modules using compile-time DSLs can drift between baseline and mutation runs; the JSON warnings array names the affected modules.
  • Macro mutation slowdown. Mutations inside macro-heavy modules recompile dependents; expect longer per-site runs.
  • Equivalent mutants. Some mutations are semantically equivalent to the original; they survive every test by definition. This is a known limit of mutation testing, not a tool bug.
  • Mutation IDs use scheme 2. Formatting and unrelated edits preserve IDs when the file path, enclosing definition, normalized expression, and occurrence among identical sites stay the same. Both JSON and NDJSON use report schema version "2". IDs and reports from earlier versions cannot be compared with version 2. Compare the schema, ID scheme, and Elixir and OTP versions before comparing IDs. Formatter migrations (mix format --migrate) can rewrite the syntax and change IDs. When an identical expression is added earlier in a function, later IDs can shift. Persist and compare function and before_source alongside IDs.
  • --no-json is not supported in v1. Pretty terminal output is deferred to v1.1 (see mutagen.decision.no_pretty_output_v1); use jq for now.
  • --seed controls ExUnit ordering only. It does not seed mutation enumeration. Given the same source, enumeration order is stable.
  • --scope colon syntax is unsupported. file.ex:Module is rejected with abort_reason: "colon_syntax_unsupported" (see mutagen.decision.scope_syntax_simplified).
  • Self-mutation is refused. --scope MutagenEx.* or Mix.Tasks.Mutagen exits with abort_reason: "self_mutation_refused" (see mutagen.decision.self_mutation_refused).
  • --json paths are canonicalised inside the project root. Paths with .. segments or NUL bytes exit with abort_reason: "unsafe_json_path" at parse time. Symlinks whose target escapes the project root exit the same way at canonicalisation time, before any mutation runs. CI workflows writing artifacts outside the project root must pass --unsafe-json-outside-project explicitly; a one-shot stderr warning then names the resolved target.
  • --max-concurrency > 1 is experimental. Parallel dispatch shares ExUnit.Server, the Code.Server, and :cover across per-site tasks. It can produce incorrect kill/survive classification and corrupted coverage on real ExUnit/:cover backends; the safe path is the default, --max-concurrency 1.
  • tag:NAME charset is bounded. --tests tag:NAME must match ~r/\A[a-z][a-z_0-9]{0,63}\z/ — lowercase ASCII, digits, or _, up to 64 chars, with a lowercase first character. Targets outside this charset are refused with abort_reason: "invalid_tag_name" before any test resolution runs. This is the atom-table-DOS bound (see mutagen.cli.r11): CI loops like mix mutagen --tests tag:$(uuidgen) cannot grow the BEAM atom table.
  • Caps on input and output volume. --scope and --tests each accept at most 100 occurrences (abort_reason: "too_many_targets"). --max-sites caps enumerated mutation sites (default 10_000; abort_reason: "too_many_sites"). --budget-ms (optional) caps the aggregate mutation wall-clock and yields a truncated: true partial report when exhausted.

Summary

Types

Pluggable collaborators for the mix task's state machine. Each entry is a module atom the task uses instead of a hard-coded call, so every error path AND the happy path can be unit-tested by injecting a stub module that captures its arguments and returns canned shapes.

Functions

Test seam: run the task with a custom dispatch table.

Types

dispatch()

@type dispatch() :: %{
  optional(:cli) => module(),
  optional(:otp_release) => String.t(),
  optional(:elixir_version) => String.t(),
  optional(:project_root) => Path.t(),
  optional(:clock) => (atom() -> integer()),
  optional(:scope) => module(),
  optional(:tests) => module(),
  optional(:ast_cache) => module(),
  optional(:coverage) => module(),
  optional(:enumerator) => module(),
  optional(:baseline) => module(),
  optional(:mutation) => module(),
  optional(:reporter_ok) => module(),
  optional(:reporter_error) => module(),
  optional(:io) => module(),
  optional(:reporter) => {module(), atom()},
  optional(:pipeline) => {module(), atom()}
}

Pluggable collaborators for the mix task's state machine. Each entry is a module atom the task uses instead of a hard-coded call, so every error path AND the happy path can be unit-tested by injecting a stub module that captures its arguments and returns canned shapes.

Per bw mutagen-wrd.33 (closes F9), the 11 production phase keys carry module atoms (not {module, function} tuples). The Mix task calls mod.callback(args) directly; the callback for each slot is defined by a dedicated behaviour under MutagenEx.Pipeline.*Facade.

The keys correspond to the orchestration stages of mutagen.mutation_pipeline:

S1 shipped a two-key legacy shape (:reporter, :pipeline) for early CLI testing. Those keys carry {module, function} tuples and route through the legacy code path (run_legacy/2), which uses apply/3 to dispatch — that is the explicit back-compat fallback carve-out of mutagen-wrd.33's "no apply/3 in the Mix task" contract.

Functions

run(argv, dispatch)

@spec run([String.t()], dispatch()) ::
  :ok
  | {:aborted, atom(), MutagenEx.JsonReporter.Report.t()}
  | {:error, MutagenEx.CLI.reason(), map()}

Test seam: run the task with a custom dispatch table.

Production code calls run/1, which threads through default_dispatch/0. Tests pass a partial dispatch via run/2 — any key absent from the passed map falls back to the default.

The return value mirrors what run/1 does observably:

  • :ok on a successful parse and pipeline run that produced a success-shape JSON document.
  • {:aborted, reason, %Report{}} on any abort path. The reporter and io collaborators have already been invoked with the iodata + exit code; the return value is for tests that want to assert which abort reason fired without parsing the emitted JSON.

run/1 itself does not return one of these — it calls System.halt/1 via the default io collaborator on every path. run/2 lets tests observe without halting the test VM by overriding the :io collaborator.