Mutation testing for Elixir. mix mutagen mutates a chosen scope, runs a cited test set against each mutation, and emits a single JSON document classifying every mutation as killed, survived, timeout, error, or compile_error.

The CLI (mix mutagen) is the supported user-facing entry point; MutagenEx.MutationRunner.run/1 is the equivalent library-caller entry point. The CLI ends each run with an aggregate JSON document. --stream writes per-site events to the same destination before it.

Audit mode

Use audit mode to find tests that execute code without asserting its behavior. Run a whole module and read the per-function table. It lists up to 50 evaluated functions, lowest kill rate first, with zero first.

MIX_ENV=test mix mutagen --scope MyApp.Billing --tests auto --json billing.json --summary billing.md
jq '.summary.by_function[]' billing.json

Run each source file in a directory and rank the reported modules by kill rate, with zero first and unevaluated runs last.

find lib/my_app -name '*.ex' | while IFS= read -r file; do
  MIX_ENV=test mix mutagen --scope "$file" --tests auto --json "audit/$file.json" --summary "audit/$file.md"
done
find audit -name '*.json' -exec cat {} + | jq -rs 'sort_by(.summary.counts.kill_rate // 2)[] | [.summary.counts.kill_rate, ([.scope[].module] | join(", "))] | @tsv'

How much to trust a survivor

A blind measurement at a pre-release commit of this series covered ten files from two private Phoenix applications. A seeded script selected the files. They included two contexts, two controllers, LiveViews, and plain modules. A reviewer judged each survivor after seeing only the code change and the test file.

The reviewer judged public functions by the inputs their specs admit. For private functions, controllers, and LiveViews, the reviewer used the inputs their real callers, routes, and forms can deliver.

SurvivorsReal test gapsShare95% interval
High confidence27 of 3284%68% to 93%
High confidence, reachable by current production callers23 of 3272%55% to 84%
Low confidence8 of 2433%18% to 53%
ConfidenceMutatorReal test gaps
Highboolean7 of 7
Highcase_drop12 of 15
Highelse_removal3 of 3
Highresult_tuple2 of 2
Highcompare1 of 1
Highliteral2 of 4
Lowpipeline5 of 19
Lowguard_drop2 of 4
Lowwith_swap1 of 1

Every figure uses one test file per module and measures gaps in the selected tests. A follow-up ran nine of the real-gap survivors against every test file that mentions the function. Other test files in the same suite killed six of them. Three survived.

A survivor means “the tests you selected miss it”. First rerun with a wider --tests selection before writing a test.

The sample is small and per-mutator confidence is provisional. Mutagen gives advice for the current run. See the advisory decision.

This measurement predates two changes that make a whole-file run mutate more code: sites on function-head lines (guards) now run, and sites that line coverage misses are confirmed by an execution probe and then run. On the two files rerun after those changes, every earlier site kept its status and the run gained sites: 30 became 42 on one file and 42 became 69 on the other, with 30 of the 50 new sites surviving. Those added survivors have not been judged, so the shares above describe the earlier set of sites only.

Changed-code measurement before the fixes

A seeded script chose 12 real commits from two private Phoenix applications. A reviewer judged them blind, seeing only the change and the tests that ran.

What the summary showedEntriesReal gapsCovered by a test that did not runFalse claimNoise
Check first (not executed)30120
Worth a look (high confidence)30003
JSON only (low confidence)81106

This measurement predates probe confirmation, the changed-line rule, and opt-in escalation. Those changes address the false execution claims, the off-diff survivors, and the noise from widening the test selection by default.

Third-pass measurement after the fixes

The third pass used the same 12 commits and the same blind protocol after probe confirmation, the changed-line rule, and opt-in escalation.

  • 76 runtime sites in scope, 72 evaluated (59 killed, 12 survived, 1 timed out), 1 could not tell, 3 skipped by the validator.
  • 7 of 12 runs showed nothing. 5 runs showed 12 survivors in all, every one on a changed line.
  • Check first: 0 entries. No “not executed” claim was made, so none was false (the earlier pass made 3, 2 of them false).
  • Worth a look, high confidence: 4 entries, 0 real gaps, 4 noise.
  • Worth a look, low confidence on a changed line: 8 entries, 1 real gap, 7 noise.
  • Noise by kind: 6 reorderings nobody can observe, 2 inputs that cannot arrive through any caller, 2 that only the function's own contract admits, 1 fallback for malformed configuration that another test already rules out.
  • Escalation, rerun on the 5 runs with survivors: 0 outcomes changed, 186 s added.
  • Every compared site (28 across 5 runs) had the same status as one run with all selected tests passed explicitly.

The report still included pipeline in Worth a look during this measurement. Five of the eight low-confidence entries came from pipeline. None was a real gap.

On a commit that came with tests, a changed-code run is usually silent. A listed survivor is more often an unobservable or unreachable change than a missing test. One of the 12 survivors here was a real test gap. Read the line before drawing a conclusion. The whole-file audit figures above support looking for test gaps.

Changed-code runs

Changed-code runs are quiet by design. On a commit that came with tests, “nothing to check” is the normal and correct output. The summary shows functions whose sites the selected tests did not execute, confirmed by probes, functions where at least three high-confidence mutants ran and all evaluated mutants survived, and survivors whose own source spans overlap changed lines. The detailed JSON holds the remaining results.

MIX_ENV=test mix mutagen --since origin/main --json out/mutagen.json --summary out/mutagen.md

Mutagen compares the working tree with the merge base of the ref and HEAD. A shallow checkout always aborts. Fetch full Git history and use fetch-depth: 0 in GitHub Actions. The run plans mutations in named runtime functions on touched lines, then compares their group fingerprints with the base source. Formatting, comments, and moved unchanged functions contribute no runtime sites. Use --scope to narrow the diff further. An empty runtime plan finishes before host setup.

The run includes untracked, unignored files. It excludes test roots and test support from source planning. An incomplete run returns exit 0 with its reasons. A tool abort returns exit 2.

Automatic test selection looks for a/b_test.exs under the configured test roots for a/b.ex under a configured source root that has a planned runtime mutation. When a --since run has runtime sites, it also selects new test files and changed test files whose parsed code differs from the merge base. Formatting and comment edits alone do not add a test file to this selection. An empty runtime plan selects no tests. Add --escalate to search for more tests for surviving or uncovered sites. Mutagen searches uncited test files for the full module name, or for the module's last segment as a whole word in a file that aliases its parent. Brace aliases and aliases declared with as: count in the module pass. It then searches for qualified function calls or captures, or bare calls where the matching module is imported or aliased. It counts text mentions and uses path order to break ties. Each step takes up to 20 files per function by default. Mutagen tests each candidate on the original code. It allows ten times the narrow baseline duration, with a thirty-second floor capped by the host boot and mutant timeouts. Each step reports candidate_deadline_ms. If a batch fails, Mutagen keeps the files that pass together in score order. The module and function passes share a two-minute budget, including candidate probes. Mutagen estimates each pass from the candidate baselines and reports files that exceed the budget as escalation_budget_exhausted. Affected survivors carry selection.escalation_incomplete: true. When escalation takes more than half the run, summary notes give the time spent in each pass. A pass judges mutants with the files admitted in that pass. Only survivors and uncovered sites advance. A timeout stays unevaluated and keeps the step that recorded it.

meta.since records the ref, merge base, and site counts. meta.complete and meta.incomplete_reasons record missing evidence from skipped or unreadable paths and sites that the runner dropped. Every abort sets meta.complete to false and adds its reason to meta.incomplete_reasons. A partial run stopped by --budget-ms records mutation_budget_exhausted. An accounting mismatch between planned sites and the final outcome records also marks the report incomplete. meta.since.unmapped lists changed code without runtime sites and omits blank, comment and structural lines. mutation.pending shows sites with no outcome, including uncovered sites and covered sites left unrun. Coverage includes green tests added during escalation. meta.tests_selected lists why each test file joined, and each result lists the steps it ran. Warnings name rejected files and give a reason. Selected and rejected test paths follow --no-source. Each step counts files dropped after a load attempt in dropped_after_load. Result selection includes after_dropped_file: true for verdicts made in that step or later. Those verdicts can reflect side effects the tool could not undo, such as writes to persistent terms. With --stream, changed-code and automatic runs emit final results and compile errors after all passes with the same fields as the aggregate. Pending sites appear in the final JSON document.

Site IDs are stable within an unchanged function. Editing a function can reassign IDs, so use them to inspect the current report.

Why mutagen_ex (and how it differs from muzak)

Elixir already has a mature mutation tester: muzak / muzak_pro by Devon Estes. If you want a polished, human-facing mutation-testing workflow for everyday TDD, muzak is the established choice and mutagen_ex is not trying to replace it.

mutagen_ex is built for repeatable audits and automated review. It gives people a Markdown summary and gives scripts and agents a stable JSON report with site IDs they can compare across runs.

mutagen_ex focuses on these workflows:

  • JSON output. mix mutagen writes the aggregate document to stdout or --json <path>. Use --stream to receive per-site JSON lines before the aggregate. Read the --json file with jq to inspect it. Output format decision.

  • Stable mutation IDs. Scheme 2 uses {relative_file}:{hash}:{mutator}. The hash includes the enclosing module, function name and declared arity, the parsed expression with source positions and presentation metadata removed, and its occurrence among identical expressions for the same mutator. Bare literal IDs also include the parent expression and child position. Module-body sites count occurrences by module and statement head. Formatting preserves IDs across whitespace and line reflow. Formatter migrations (mix format --migrate) can rewrite syntax and change IDs. Adding an identical expression earlier in a function can shift later IDs and transfer an old ID to a different occurrence. Persist and compare function and before_source alongside IDs. Require version: "2", meta.id_scheme: "2", and matching meta.elixir_version and meta.otp_version before comparing. IDs and reports from earlier versions cannot be compared with version 2. Mutation ID scheme 2.

  • Deterministic, serial-by-default execution. Every phase — baseline, coverage, and each per-mutation run — forces max_cases: 1 and a fixed --seed (default 0). The same {source, --tests, --scope, --seed} always yields the same per-site classification. Reproducibility is the point; a judge comparing two runs must be able to trust that a difference reflects a code change, not test-order flake. See .spec/decisions/serial_execution_and_seed.md.

  • Readable audit reports. Markdown shows findings by function and offers a command to rerun each example. The JSON retains every site for scripts that rank modules or compare outcomes.

  • In-process pipeline (vs muzak's child-BEAM-per-mutation). mutagen_ex swaps bytecode via Code.compile_quoted/2 and restores from a cached binary, all inside the same BEAM VM as the Mix task — roughly 10x faster per mutation than spawning a fresh node each time, which matters when a model is waiting on feedback. The trade-off is honest: shared-VM state drift on use SomeModule macros is possible, so it is surfaced as mutation.state_drift_warning in the JSON rather than hidden. The cost of staying in-process is that mutagen_ex cannot mutate itself (doing so would corrupt the running runner); self-mutation is refused with reason: :self_mutation_refused. See .spec/decisions/in_process_pipeline.md and .spec/decisions/self_mutation_refused.md.

The mutator catalog and its lineage

mutagen_ex ships ten AST mutators (.spec/specs/mutators.spec.md). They fall into two groups: four that are direct Elixir realizations of the classic, language-agnostic mutation operators studied since the Mothra era, and six that are Elixir-idiomatic and have no clean parallel in the C/Java mutation literature.

The first group maps onto the selective mutation operator set — the small family Offutt et al. found sufficient to retain almost all of a full operator set's fault-detection power at a fraction of the mutant count (Offutt, Lee, Rothermel, Untch & Zapf, "An Experimental Determination of Sufficient Mutant Operators," ACM TOSEM 5(2), 1996, doi:10.1145/227607.227610; see also Jia & Harman's survey, IEEE TSE 37(5), 2011, doi:10.1109/TSE.2010.62 for the broader taxonomy these abbreviations come from):

mutagen_ex mutatorStandard operatorWhat it does
arithAOR — Arithmetic Operator Replacementswaps + ↔ -, * ↔ / on numeric binary ops
compareROR — Relational Operator Replacementswaps == ↔ !=, < ↔ >=, > ↔ <=
booleanLCR — Logical Connector Replacementswaps and ↔ or, && ↔ ||, drops not/!
literalCRP — Constant Replacementflips true ↔ false, swaps small integer literals

The remaining six mutate constructs that are idiomatic to Elixir/Erlang and have no language-agnostic equivalent in the standard catalogs — they exist because the standard operators say nothing about tagged tuples, multi-clause guards, with, or the pipe:

mutagen_ex mutatorWhat it doesWhy it has no classic parallel
result_tupleflips {:ok, x} ↔ {:error, x}targets Elixir's tagged-tuple result convention
guard_dropdrops a function-clause when guardguards are an Elixir/Erlang dispatch construct
with_swapswaps the first two leading <- clauses of a withwith is an Elixir control structure
else_removaldrops the else branch of if/withshaped to Elixir's if/with else semantics
case_dropdrops the last case/cond clauseclosest to the classic statement/clause-deletion (SDL) family, but Elixir's exhaustiveness and CaseClauseError behaviour make it its own thing (see Known limitations)
pipelinereorders adjacent |> segmentsthe pipe operator is Elixir-idiomatic

The split is deliberate: the four classic operators give the catalog a well-understood, citable backbone, while the six idiomatic operators cover the fault classes that only show up in Elixir code. Treat the standard-operator names above as the vocabulary a reviewer (human or LLM judge) already knows; treat the idiomatic six as the Elixir-specific extension that vocabulary lacks.

Mutator confidence

Every aggregate mutation.results[] entry and every NDJSON "result" line has a "confidence" field. The ratings apply to whole operators and draw on manual review of survivors from two Phoenix apps. Even a high-rated survivor can be equivalent to the original code.

The ratings are provisional. We will re-measure each mutator as more files are judged.

MutatorConfidenceReason
arithhighThe swapped operator can change a numeric result.
comparehighNegating the comparison can change which branch runs.
booleanhighSwapping a connector or removing negation can change the value or short-circuit behavior.
literalhighA caller can observe the changed value or pattern.
with_swaplowTwo checks can still return the same failure in practice, or their failures may never occur together.
case_drophighInputs handled by the removed final arm can now raise a clause error.
pipelinelowIndependent stages often give the same result in either order, as with LiveView assigns or Map.put/3 calls on different keys.
result_tuplehighThe tag swap changes which consumers treat a result as success or failure.
else_removalhighRemoving the branch changes how a false condition or failed match is handled.
guard_droplowCallers often cannot supply a value that the removed guard would have rejected.

with_swap applies its failure-priority filters only when the two clauses are independent. It treats a pair as dependent when a name bound by either pattern appears anywhere in the other clause, or when either clause uses binding(), dbg() or __ENV__. It records skip reasons in mutation.skipped.

ReasonWhen it applies
bound_var_used_before_bindingA variable-shaped name in the moved-first expression also appears as a binding in the now-later pattern. The check ignores module attributes and binary modifiers.
insufficient_refutable_patternsIn an independent pair, either match uses an unguarded variable, a wildcard, or a literal pattern with an equal literal expression.
indistinguishable_failuresIn an independent pair, both expressions are the same literal or read the same variable.

The RHS name check can skip a valid swap when the name has an outer binding or appears only inside a nested binder or quote. This includes names local to fn, case, for, receive, rescue, nested with, match?, matches and destructure. The compiler resolves dependencies in guards, pins and binary sizes for kept swaps. Unexpanded macros can hide reads and bindings. An else arm does not affect validation. Even if it ignores the failure value, swapping the clauses can run side effects or raise an exception that the original order avoided.

Self-mutation

The repository tests mutation runs against the mutator registry, scope resolver, and JSON reporter. The harness copies those modules under a separate namespace in a temporary Mix project and runs mix mutagen from a fresh shell. Run it with mix test --include self_mutation. Self-mutation test.

Install

Mutagen requires Elixir 1.18 or later and Erlang/OTP 27 or later.

System tool

Install the Mix task for your current Elixir installation. Mix fetches the source from Hex and builds the archive with your local toolchain.

mix archive.install hex mutagen_ex
mix help mutagen

To pin a version in CI, use mix archive.install hex mutagen_ex 0.2.0 --force.

Project dependency

Add this entry to deps/0 in your project's mix.exs:

{:mutagen_ex, "~> 0.2", only: [:dev, :test], runtime: false}
mix deps.get
MIX_ENV=test mix help mutagen

With either install, run the task from the host project:

MIX_ENV=test mix mutagen --scope lib/foo.ex --tests test/foo_test.exs --json report.json

Host test setup

Run MIX_ENV=test mix mutagen with either install method. This loads config/test.exs and compiles test support. Explicit environments such as MIX_ENV=ci are allowed when the host configures them for tests. A host can also set def cli, do: [preferred_envs: [mutagen: :test]] in mix.exs. Mix may print dependency discovery output before the task runs.

After compiling, the task loads runtime config and runs the host test alias entries before the test task. A Phoenix alias can create and migrate the test database there. If the alias lacks those steps or you use --no-test-alias, create and migrate the test database before invoking mutagen. Pass MIX_TEST_PARTITION if your configuration uses it to select a database. Mutagen ignores arguments on the test entry and uses --tests and the helper's ExUnit configuration for selection. Alias functions get []. Mutagen ignores their return values. An alias with entries and no recognizable test entry produces a report warning.

The task starts the host application and requires test_helper.exs once from each configured :test_paths entry, with ExUnit autorun disabled. The default is test/test_helper.exs when test/ exists. Run from the app that owns the tests in an umbrella project. --no-host-app or MUTAGEN_NO_HOST_APP=1 skips application startup and still loads runtime config and the helper.

Coverage, baseline, and every mutant use the helper's ExUnit options. Cases retain their declared async value. With max_cases: 1, ExUnit runs cases serially in its usual async-then-sync order. Whole-file targets honor helper includes and excludes. An explicit tag or line target takes precedence when narrowing the test set. A coverage or baseline pass that executes no tests aborts with no_tests_run. Helper formatters run on every pass. A formatter that writes a file may overwrite it after each mutant. Mutagen writes the final report before running ExUnit after_suite callbacks. Each callback gets its own --boot-timeout-ms deadline. Cleanup errors and timeouts go to stderr. A callback that calls System.halt/1 decides the VM exit code.

--boot-timeout-ms bounds runtime configuration, alias pre-steps, application startup, and helper evaluation after compilation. Coverage and baseline each get the greater of --boot-timeout-ms and ten times --timeout-ms. The mutation budget starts after baseline. An OS process started by a test-alias command can outlive a boot timeout. A helper that calls System.halt/1 ends the BEAM before reporting.

Abort documents include a fixed hint and, where available, the step, exception type, file and line. Host diagnostics go to stderr. Do not publish stderr as a CI artifact. Abort documents repeat this warning in details.note.

--json PATH is the guaranteed report channel. The file contains exactly one JSON document, or NDJSON with --stream, regardless of host output.

Without --json, Mutagen writes the report to stdout and routes the :user device and known console logger handlers to stderr. The VM can write directly to file descriptor 1, including when an OTP logger handler crashes. In-process I/O routing cannot intercept those writes. Mutagen prints a reminder on stderr when it writes to stdout. Use --json PATH and --summary PATH in CI.

Mutagen writes the file with mode 0600. If it cannot safely write the file, it warns on stderr and falls back to stdout.

Entrypoints

mutagen_ex boots MutagenEx.Application whenever the :mutagen_ex application starts. That supervisor is the program entrypoint; everything the tool runs lives under MutagenEx.TaskSup.

User-facing entrypoint surfaces:

  • CLI: mix mutagen (the Mix.Tasks.Mutagen task). This is the documented and supported route.
  • Library: MutagenEx.MutationRunner.run/1 for callers depending on :mutagen_ex directly. Same in-process pipeline, no CLI parsing. Library callers must ensure the :mutagen_ex application is started before invoking this entrypoint.

Only one concurrent MutagenEx mutation cycle per BEAM is supported; concurrent callers are refused with {:error, :cover_already_running, _}. See .spec/decisions/supervision_tree.md.

What mix mutagen does

For the selected scopes, mutagen_ex:

  1. Resolves the target to one or more (module, file) pairs.
  2. Runs the cited tests to record line-level coverage for the in-scope modules.
  3. Enumerates mutation sites from each in-scope file's AST. It uses line coverage first and probes remaining runtime sites before reporting them as unexecuted.
  4. Runs the cited tests on the original code to check the baseline. A red baseline aborts the run.
  5. For each site, applies the mutator, recompiles the module in place, re-runs the cited tests, and classifies the result.
  6. Restores the original module after every site.
  7. Emits the final JSON document.

Tests run serially within each pass. Mutation sites run serially by default in the same BEAM VM. --max-concurrency enables experimental parallel dispatch.

Mutagen applies changes in memory via Code.compile_quoted/2 and restores the original module after every site. The byte-identity contract covers lib/, _build/, cover/, host project config (mix.exs, mix.lock, .formatter.exs), and tmp entries with the mutagen_ex_ prefix — see mutagen.mutation_pipeline.r11 and mutagen.coverage.r7.

Basic usage

# Single file scope, single test file
MIX_ENV=test mix mutagen --scope lib/foo.ex --tests test/foo_test.exs

# Module scope, tag-cited tests, custom per-mutation budget
MIX_ENV=test mix mutagen --scope MyApp.Foo --tests tag:fast --timeout-ms 10000

# Function scope, deterministic seed, redirected output
MIX_ENV=test mix mutagen --scope MyApp.Foo.bar/1 \
            --tests test/foo_test.exs \
            --seed 42 \
            --json out/mutagen.json

A survivor means “the tests you selected miss it”. First rerun with a wider --tests selection before writing a test.

Reading the summary

Add --summary mutagen.md for a Markdown file you can read or paste into a pull request. The JSON always includes the same findings under summary. Both output paths use the project-root boundary by default.

The headline leads with the highest-priority finding. summary.checked reports full or partial along with the total sites, uncovered sites, and sites left unevaluated for other reasons. A partial run states how many sites it could not check. When it has no finding, the headline says “Mutagen found nothing in the sites it could check.”

  • Check first groups findings by function. not_executed means none of its in-scope sites recorded a hit during an execution probe. Mutagen uses line coverage first. For the remaining sites, it installs counters in the original runtime expressions, runs the admitted tests once, and restores the original bytecode. A probe hit adds the site to the mutation queue. Captures, interpolation, bitstring segments, map-update values, keyword arguments, function-head patterns, guards, and cond heads can prevent probe placement. Such sites, compile failures, and failed probe runs yield could_not_tell. An empty test selection also leaves execution unknown and has its own note. Results report their execution evidence in execution, and meta.timings_ms.probe records the probe phase when it runs. nothing_asserted means the tests that ran let every evaluated mutant in the function survive, including at least three high-confidence mutants without a recorded escalation shortfall. If a dropped test file mentions the function and fails alone on the unmutated code, the summary leads with the failed test file. A file that fails in combination appears in the notes.
  • Worth a look in a changed-code run lists survivors whose spans overlap changed lines. Within each confidence group, entries sort by file and line. On-line with_swap and guard_drop survivors follow the high-confidence entries and carry a low-confidence label. Pipeline survivors and survivors away from changed lines stay in the JSON. The notes count them.
  • Classic and audit runs keep the provisional precision order. They list high-confidence survivors outside Check first, followed by those marked not fully escalated. Low-confidence survivors stay in the JSON. Change the default five Worth a look entries with --summary-max N.
  • Timeouts count as unevaluated because the tests did not finish. The summary kill rate uses killed and surviving sites. Notes count results after a timeout or host state change. They list skipped escalation files and explain when a kill followed a file that had already loaded before Mutagen dropped it. When three or more mutants time out and the waits take over half the mutation phase, the summary points to --timeout-ms and --budget-ms.

Copy the printed MIX_ENV=test mix mutagen --site … command from the summary to rerun the one site with --tests auto --escalate and up to 40 test files per function in each pass. The escalation cap is the larger of 30 seconds and ten times the narrow baseline. A smaller configured budget still applies. The summary says when escalation admitted no files. Changed-code reruns keep the merge base from the report. Run the command from the same project checkout. Broader tests can kill a site that previously survived. With --no-source, the summary omits file names, function names and excerpts and prints an opaque reference for the rerun.

Running in CI

Mutagen is advisory. Each run reports the current checkout and uses no repository state from previous runs. A shallow checkout aborts because the diff needs the merge base. Use fetch-depth: 0. The unmapped list names changed lines Mutagen could not map to a runtime site, such as attributes and macros. A separate list shows changed lines outside an explicit --scope. The summary names changed lines that tests missed inside evaluated sites and reports partial coverage. Use --json PATH and --summary PATH in CI. Upload the JSON file and keep stderr out of artifacts.

name: Mutation advice
on: pull_request

jobs:
  mutagen:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
        with:
          fetch-depth: 0
          persist-credentials: false
      - uses: erlef/setup-beam@54075bcc5e249e4758d363f27d099f55d843f124 # v1.24.1
        with:
          version-file: .tool-versions
          version-type: strict
      - name: Collect mutation advice
        continue-on-error: true
        env:
          MUTAGEN_BASE: ${{ github.event.pull_request.base.sha }}
        run: |
          MIX_ENV="test" mix deps.get
          MIX_ENV=test mix mutagen --since "$MUTAGEN_BASE" --json mutagen.json --summary mutagen.md
      - name: Add advice to the job summary
        if: always()
        run: |
          if [ -f mutagen.md ]; then
            cat mutagen.md >> "$GITHUB_STEP_SUMMARY"
          fi
      - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
        if: always()
        with:
          name: mutagen
          path: mutagen.json

A successful tool run exits 0 even when tests miss mutations. The continue-on-error setting also allows the job to proceed after a tool abort. The example assumes the project declares the dev/test dependency. The job summary includes the abort reason and its hint. To post the Markdown on a same-repository pull request, grant pull-requests: write to the job and set GH_TOKEN and PR_NUMBER for an optional step. Run gh pr comment "$PR_NUMBER" --body-file mutagen.md when the file exists.

This repository runs its normal tests and host fixtures in CI. Mutagen refuses a changed-code run on its own runtime modules.

Set --timeout-ms for scopes whose tests finish quickly. A runaway mutant can consume the full timeout. Set --budget-ms to stop dispatching new mutants once the mutation phase has used its budget. Use meta.timings_ms to size both limits on your CI runner. Dividing mutation time by the number of attempted sites gives the average cost per mutant.

Flags

FlagPurpose
--summary <path>Write a Markdown summary to a separate file. As with --json, the path must stay within the project root unless --unsafe-json-outside-project is set.
--summary-max <int>Maximum entries in Worth a look. Default 5. Zero keeps only the omitted count.
--site <id>Rerun one site using the report ID or opaque reference.
--scope <target>What to mutate. A .ex file path, a module name, or Module.fun/arity. Required unless you use --since or --site. Repeatable up to 100 occurrences.
--tests <target>A _test.exs path, a file:line pair, or tag:<name>. Repeatable up to 100 times. Use auto on its own. Required unless you use --since or --site.
--boot-timeout-ms <int>Deadline in milliseconds for startup after compilation. Default 300000.
--no-sourceOmit source excerpts, excerpt lines, file and function fields from mutation results and compile errors, including stream entries.
--include-host-outputInclude captured compiler and test stderr and abort messages, with best-effort redaction.
--no-test-aliasSkip entries in the host's test alias before the test task.
--no-host-appLoad runtime configuration and the helper without starting the host application.
--since <ref>Plan changed runtime sites from the merge base. Defaults to --tests auto.
--escalateWiden automatic test selection for surviving or uncovered sites. Off by default.
--no-escalateAccepted with no effect for one release.
--escalate-max-files <int>Maximum test files per function in each extra pass. Default 20.
--escalate-budget-ms <int>Total time for the extra passes. Default 120000. Zero removes the limit for runs without --site. A single-site run caps the budget at the larger of 30 seconds and ten times the narrow baseline.
--timeout-ms <int>Override the per-mutant deadline in milliseconds. The default uses three times the baseline duration with a 5000 ms floor. Mutagen warns if a requested timeout is shorter than the baseline.
--no-fail-fastRun the full cited suite for each mutant. By default, mutation runs stop executing tests at the first failure.
--seed <int>ExUnit seed, propagated to every test-running phase. Default 0. Controls test ordering only, not mutation enumeration order.
--json <path>Write the final JSON document to <path> instead of stdout. The document always ends with a single newline. Path is canonicalised before any mutation runs: .. segments and NUL bytes are refused at parse time, and the resolved path must stay inside the project root unless --unsafe-json-outside-project is also passed.
--unsafe-json-outside-projectOpt-in to writing --json output outside the project root. CI integrations targeting an artifacts directory above the project root pass this; everyday use should leave it off. Emits a one-shot stderr warning naming the resolved target at run start.
--max-sites <int>Upper bound on enumerated mutation sites for one run. Default 10000. Exceeding the cap aborts with abort_reason: "too_many_sites" BEFORE the mutation runner starts — narrow --scope (or raise the cap) to proceed.
--budget-ms <int>Optional aggregate wall-clock budget for the mutation phase, in milliseconds. Default unbounded (--timeout-ms still bounds each site). When elapsed, the runner stops dispatching new sites and emits a truncated: true partial JSON report.
--max-concurrency <int>Cap on the number of per-site mutation tasks Task.Supervisor.async_stream_nolink/4 runs in parallel. Default 1 (serial execution). Values greater than 1 are EXPERIMENTAL and emit a one-shot stderr warning because real ExUnit/:cover backends can produce incorrect kill/survive classification and corrupted coverage. See Parallel mode below.
--streamEmit one NDJSON line per completed site (and start/end envelope lines) to the same sink the aggregate document goes to. Off by default.
--no-progressSuppress the human-readable per-site progress feed on stderr. Default is auto-on when stderr is a TTY, auto-off otherwise.

See Known limitations for rejected options such as --no-json and --scope file.ex:Module.

Per-mutant fail-fast

By default, a mutant is killed when ExUnit records a test failure before the per-mutant deadline. This includes a mutant whose full suite would time out in a later test. If the deadline arrives before any failure, the mutant is classified as a timeout. Baseline and coverage have no failure cap. JSON reports the mode as meta.fail_fast and reports true if argument parsing fails before a mode is available.

Fail-fast preserves survivors and existing kills when tests run independently. If one test sets up or cleans up state for another, a failed run can skip that test and change the next mutant's verdict, even if the next mutant runs every test. These suites already depend on ExUnit's test ordering when the seed changes. Use --no-fail-fast to run the full suite per mutant. Skipped tests can also change per-mutant warnings and tainted_predecessors. A shorter mutation phase can complete more sites within the same --budget-ms.

Mutagen sets the seed, serial execution, and the failure cap after test files load. Each phase keeps the loaded host formatters and adds its own failure formatter when needed. Other host options, such as test timeouts and log capture, stay active. The resolved test filter supplies include and exclude for each phase. Mutagen restores the host settings after each phase.

Parallel mode, streaming, and progress

Mutagen includes the wiring for parallel per-site dispatch, NDJSON streaming, and a per-site progress feed. The mechanism is in place, but --max-concurrency > 1 is EXPERIMENTAL: the in-process pipeline shares ExUnit's global server, the Code.Server's per-module load locks, and :cover instrumentation across all per-site tasks (see "Known limitations" below). On real ExUnit/:cover backends, parallel dispatch may produce incorrect kill/survive classification and corrupted coverage. Every report records meta.max_concurrency. Values above 1 emit a stderr warning and add "Verdicts are unreliable in experimental parallel mode." to the report's top-level warnings. The default is 1.

Mutagen has no runtime dependencies — it installs cleanly as a dependency-free Mix archive (mix archive.install). Per-site observation (NDJSON streaming and the progress feed below) rides the mutation runner's single :on_site_completed callback rather than a :telemetry event surface.

NDJSON streaming (--stream)

When --stream is set, MutagenEx.JsonStreamer emits one JSON object per line on the same sink the aggregate document goes to. The line shape is byte-equal to the equivalent entry in the aggregate mutation.results[] / mutation.compile_errors[] array, plus a "kind" discriminator and the "version" literal:

{"version":"2","kind":"start","total":42,"meta":{"id_scheme":"2",...}}
{"version":"2","kind":"result","id":"...","function":"MyApp.Counter.add/2","status":"killed","mutator":"arith","confidence":"high",...}
{"version":"2","kind":"compile_error","id":"...","message":"..."}
{"version":"2","kind":"end","meta":{"id_scheme":"2",...},"aborted":false,"abort_reason":null,"truncated":false,"kill_rate":0.81,...}

Per-site lines ("result" / "compile_error") arrive in input order even under --max-concurrency > 1, because the runner collects via async_stream's :ordered: true default and the streaming callback fires from a sequential post-fold. The final line on the sink is the aggregate document without a kind field. Timings appear only in that document. The end line includes truncated.

Progress feedback

By default, when stderr is a TTY, a one-line-per-site progress feed is written to stderr:

[12/345] killed   lib/foo.ex:42 :arith
[13/345] survived lib/foo.ex:43 :arith
[14/345] timeout  lib/foo.ex:51 :case_drop

Pass --no-progress to suppress unconditionally. The feed is driven by the mutation runner's :on_site_completed callback (the same per-site seam --stream rides) via MutagenEx.Progress.

Resource caps

mix mutagen enforces caps on input and output volume to keep one bad invocation from running away with the host's memory or wall-clock time:

  • --scope and --tests each accept at most 100 occurrences. The 101st is refused at parse time with abort_reason: "too_many_targets". No filesystem touch.
  • --max-sites (default 10_000) caps the enumerated mutation sites. Exceeding the cap aborts with abort_reason: "too_many_sites" BEFORE the mutation runner starts; the error-JSON details map names the count so you can choose between narrowing --scope and raising --max-sites.
  • --budget-ms (optional) caps the aggregate mutation-phase wall-clock. When the budget elapses the runner stops dispatching new sites and the JSON document carries truncated: true (aborted stays false — truncation is a graceful early exit, not an abort). The per-site --timeout-ms still bounds the in-flight site; worst-case overshoot is one timeout_ms.

--json path safety

--json <path> is canonicalised before any mutation runs, in two layers:

  1. Parse-time (pure-string): .. segments and NUL bytes are refused with abort_reason: "unsafe_json_path". No filesystem touch happens.
  2. Filesystem-canonicalisation: every existing component is resolved through File.read_link/1. If the fully-resolved path escapes the project root (resolved through symlinks itself — macOS /var -> /private/var is handled correctly), the run aborts with abort_reason: "unsafe_json_path". The final component is allowed to not yet exist; it is created at write time.

The default policy is inside the project root only. CI workflows writing artifacts to /tmp or a sibling directory must pass --unsafe-json-outside-project; that flag bypasses the inside-root check and emits a one-shot stderr warning naming the resolved target. The symlink-resolution step still runs in either mode — the resolved path you write to is always the fully-canonical one.

Application configuration

mutagen_ex reads one optional knob from Application.get_env/3:

KeyPurpose
:redactA list of compiled regexes or regex source strings. Mutagen validates it after loading runtime configuration and before the test alias and helper. Invalid entries abort with invalid_redact_config. Built-in rules run first. The default adds no custom patterns.

Set it in config/config.exs:

config :mutagen_ex,
  redact: [
    ~r/AWS_(?:ACCESS_KEY_ID|SECRET_ACCESS_KEY)=\S+/,
    ~r/Bearer [A-Za-z0-9._-]+/
  ]

The default report contains repository source and structured test metadata. Treat it as sensitive as the source. Secrets committed in source can appear in excerpts. --no-source omits site excerpts, file, function, and the fixed compiler diagnostic. Scope and test paths, coverage paths, skipped paths, module names, and site IDs still appear. Each site ID contains a path and an unkeyed MD5 hash of the normalized source AST. A reader can guess a low-entropy literal and check the hash offline.

Default reports keep failure counts by module, literal helper tag keys, and counts of lost registered processes. Compiled project exception names count as repository text. Dynamic exception names, test names, helper filter values, host process names, and diagnostic bodies require --include-host-output. Logger and cleanup output stay on stderr.

Treat an opted-in report as sensitive as the test output. Pattern redaction is best-effort. It covers known credential shapes and can miss plain values, prose, and encodings. It examines at most 64 KiB per field. A regex execution error replaces the entire field with [REDACTED: redaction failed].

Emitted text bodies and source excerpts have a 4 KiB payload cap. Source excerpts preserve tabs and newlines and remove other control characters and Unicode format characters. A truncation marker reports bytes dropped by the input and output caps.

Exit codes

CodeMeaning
0Mutagen wrote its report. This includes incomplete runs and runs where every mutation survived.
2Mutagen aborted and wrote an error document to the selected destination. The string field abort_reason names the failure, for example "missing_scope" or "baseline_red".

A host callback that calls System.halt/1 decides the VM exit code. An after_suite callback can therefore leave a completed report with aborted: false and a non-zero exit.

Abort reasons

abort_reasonMeaning
arity_requiredThe function scope omits its arity.
baseline_redOne or more tests failed against the original code.
baseline_timeoutThe baseline phase exceeded its deadline.
colon_syntax_unsupportedA function scope uses colon syntax.
cover_already_runningAn existing OTP coverage session owns modules.
cover_compile_failedOTP coverage could not instrument a selected module.
cover_module_unavailableThe OTP coverage module failed to load.
coverage_timeoutThe coverage phase exceeded its deadline.
diff_unavailableGit cannot provide a usable diff for the changed-code run.
escalation_crashedAn unexpected exception ended automatic test discovery.
ex_unit_run_failedExUnit raised, threw, or exited while running a suite.
file_not_foundA selected source or test file does not exist.
file_read_failedMutagen could not read a required file.
flag_not_supported_in_v1An option requests an unsupported output format.
function_not_foundThe scoped function and arity could not be found.
host_app_start_failedThe host application failed to start.
host_compile_failedMix failed to compile the host project.
host_loadpaths_failedMix failed to load the host project dependencies.
host_setup_timeoutHost setup exceeded its deadline.
invalid_boot_timeoutThe host setup and cleanup timeout is invalid.
invalid_budget_msThe mutation phase time budget is invalid.
invalid_inputAn argument or runner input is malformed.
invalid_lineThe test location has an invalid line number.
invalid_max_concurrencyThe mutation dispatch limit is invalid.
invalid_max_sitesThe mutation site cap is invalid.
invalid_mix_envThe command selected an unsupported Mix environment.
invalid_redact_configThe configured report redaction patterns are invalid.
invalid_seedThe ExUnit seed is invalid.
invalid_siteThe site selector has an invalid format.
invalid_since_ref--since has an invalid Git ref.
invalid_summary_maxThe summary limit must be a non-negative integer.
invalid_tag_nameThe test tag contains unsupported characters or has an invalid length.
invalid_targetThe test selector has an unsupported format.
invalid_timeoutThe per-site timeout is invalid.
missing_scopeNo mutation scope was supplied.
missing_testsNo test selection was supplied.
module_beam_missingThe compiled module could not be located.
module_not_foundThe scope module could not be resolved.
no_tests_matchA test selector matched no files.
no_tests_runExUnit executed no tests in coverage or baseline.
parse_errorMutagen could not parse a source or test file.
runtime_load_failedThe MutagenEx runtime or a report module could not load.
self_mutation_refusedThe scope targets MutagenEx code in the active VM.
site_not_foundThe site ID is absent from the current selection.
summary_path_conflictThe JSON and Markdown paths resolve to the same file.
summary_write_failedMutagen could not write the Markdown file.
test_alias_failedA host test alias setup step failed.
test_file_load_failedA cited test file could not be loaded or matched to its compiled path.
test_helper_load_failedA required test helper could not load.
too_many_sitesThe scope yielded more sites than the configured cap.
too_many_targetsThe command exceeds the scope or test target limit.
umbrella_rootThe command ran from an umbrella root.
unknown_flagThe command contains an unrecognised option.
unrecognised_targetThe scope target has an unsupported format.
unrecoverable_restore_failureMutagen could not restore the original module.
unsafe_json_pathThe report path failed validation.
unsafe_summary_pathThe Markdown path failed validation.
unsupported_elixir_versionThe Elixir release is below 1.18.0.
unsupported_otp_versionThe Erlang/OTP release is below 27.

JSON output

The aggregate and the stream end record include summary. The examples below show the existing pipeline fields. See Reading the summary for the ranked findings and the per-function counts.

Use --json PATH for a guaranteed machine-readable report. The file ends with a newline. Output to stdout is best effort. With --stream, it follows the per-site events.

meta.timings_ms measures elapsed milliseconds for boot, coverage, baseline and mutation. The total also includes work between phases such as enumeration, through report assembly. A phase that did not run shows zero. meta.timings_ms.mutant_deadline records the effective per-mutant deadline in milliseconds. Escalation records the baseline duration and deadline for each pass that runs mutants. Captured host output can contain timestamps, process identifiers, and references.

Successful run

This example comes from a copy of the lane fixture with ExUnit.start() in test/test_helper.exs. The output is indented for readability.

MIX_ENV=test mix mutagen --scope lib/lane_fixture/arith.ex \
  --tests test/lane_fixture/arith_test.exs
{
  "abort_reason": null,
  "aborted": false,
  "baseline": {
    "failed": 0,
    "failures": [],
    "passed": 5
  },
  "coverage": {
    "covered_lines": {
      "lib/lane_fixture/arith.ex": [
        12,
        14,
        16,
        18
      ]
    }
  },
  "details": {},
  "meta": {
    "elixir_version": "1.19.6",
    "exunit_seed": 0,
    "fail_fast": true,
    "id_scheme": "2",
    "max_concurrency": 1,
    "otp_version": "28",
    "timings_ms": {
      "baseline": 2,
      "boot": 93,
      "coverage": 49,
      "mutation": 98,
      "total": 266
    },
    "tool_version": "0.2.0"
  },
  "mutation": {
    "compile_error": 0,
    "compile_errors": [],
    "completed": 7,
    "kill_rate": 0.7142857142857143,
    "killed": 5,
    "results": [
      {
        "after": "a - b",
        "before": "a + b",
        "before_source": "a + b",
        "column": 24,
        "confidence": "high",
        "file": "lib/lane_fixture/arith.ex",
        "function": "LaneFixture.Arith.add/2",
        "id": "lib/lane_fixture/arith.ex:f7fc9974f882553f6b11c5428b0baac6:arith",
        "line": 12,
        "mutator": "arith",
        "status": "killed",
        "tainted_predecessors": false,
        "warnings": []
      },
      {
        "after": "a + b",
        "before": "a - b",
        "before_source": "a - b",
        "column": 24,
        "confidence": "high",
        "file": "lib/lane_fixture/arith.ex",
        "function": "LaneFixture.Arith.sub/2",
        "id": "lib/lane_fixture/arith.ex:02f1348e12b94f92cdcfe1d5cbd6634a:arith",
        "line": 14,
        "mutator": "arith",
        "status": "killed",
        "tainted_predecessors": false,
        "warnings": []
      },
      {
        "after": "a / b",
        "before": "a * b",
        "before_source": "a * b",
        "column": 24,
        "confidence": "high",
        "file": "lib/lane_fixture/arith.ex",
        "function": "LaneFixture.Arith.mul/2",
        "id": "lib/lane_fixture/arith.ex:8d516b89cd37a53d164ad5a2afe29ed0:arith",
        "line": 16,
        "mutator": "arith",
        "status": "killed",
        "tainted_predecessors": false,
        "warnings": []
      },
      {
        "after": "div_safe(a, b)",
        "before": "div_safe(a, b) when b != 0",
        "before_source": "div_safe(a, b) when b != 0",
        "column": 22,
        "confidence": "low",
        "file": "lib/lane_fixture/arith.ex",
        "function": "LaneFixture.Arith.div_safe/2",
        "id": "lib/lane_fixture/arith.ex:50c6b355d96c53576c96a09782b8533d:guard_drop",
        "line": 18,
        "mutator": "guard_drop",
        "status": "survived",
        "tainted_predecessors": false,
        "warnings": []
      },
      {
        "after": "b == 0",
        "before": "b != 0",
        "before_source": "b != 0",
        "column": 29,
        "confidence": "high",
        "file": "lib/lane_fixture/arith.ex",
        "function": "LaneFixture.Arith.div_safe/2",
        "id": "lib/lane_fixture/arith.ex:702f2afcc0d5994b0372219de4bcc4d0:compare",
        "line": 18,
        "mutator": "compare",
        "status": "killed",
        "tainted_predecessors": false,
        "warnings": []
      },
      {
        "after": "1",
        "before": "0",
        "before_source": "0",
        "column": 29,
        "confidence": "high",
        "file": "lib/lane_fixture/arith.ex",
        "function": "LaneFixture.Arith.div_safe/2",
        "id": "lib/lane_fixture/arith.ex:d821575fe16fe3710afb032918744e6b:literal",
        "line": 18,
        "mutator": "literal",
        "status": "survived",
        "tainted_predecessors": false,
        "warnings": []
      },
      {
        "after": "a * b",
        "before": "a / b",
        "before_source": "a / b",
        "column": 41,
        "confidence": "high",
        "file": "lib/lane_fixture/arith.ex",
        "function": "LaneFixture.Arith.div_safe/2",
        "id": "lib/lane_fixture/arith.ex:2edeb721487365fa8cb5322bae381f16:arith",
        "line": 18,
        "mutator": "arith",
        "status": "killed",
        "tainted_predecessors": false,
        "warnings": []
      }
    ],
    "skipped": [],
    "state_drift_warning": {},
    "survived": 2,
    "timeout": 0,
    "total": 7
  },
  "scope": [
    {
      "file": "lib/lane_fixture/arith.ex",
      "line_range": [
        1,
        19
      ],
      "module": "LaneFixture.Arith"
    }
  ],
  "tests": {
    "exclude": [],
    "files": [
      "test/lane_fixture/arith_test.exs"
    ],
    "include": []
  },
  "truncated": false,
  "version": "2",
  "warnings": []
}

Aborted run

When the run cannot proceed — for example, --scope names a module that does not exist, aborted is true and the phase blocks that never ran are null. The default details contains a fixed hint and any available structured facts. The exit code is non-zero. The reason strings are listed in the abort reason table.

{
  "abort_reason": "module_not_found",
  "aborted": true,
  "baseline": null,
  "coverage": null,
  "details": {
    "hint": "Check the module name and that its source compiles.",
    "module": "Elixir.LaneFixture.Missing",
    "note": "Stderr may carry unredacted host output. Do not publish it as a CI artifact.",
    "step": "scope"
  },
  "meta": {
    "elixir_version": "1.19.6",
    "exunit_seed": 0,
    "fail_fast": true,
    "id_scheme": "2",
    "max_concurrency": 1,
    "otp_version": "28",
    "timings_ms": {
      "baseline": 0,
      "boot": 62,
      "coverage": 0,
      "mutation": 0,
      "total": 72
    },
    "tool_version": "0.2.0"
  },
  "mutation": null,
  "scope": [],
  "tests": null,
  "truncated": false,
  "version": "2",
  "warnings": []
}

Both examples are from CLI runs on the fixture. The schema contract is in .spec/specs/json_schema.spec.md.

Baseline guard rail

Before any mutation runs, the baseline phase runs the cited tests once against unmodified code. If any cited test fails, the pipeline aborts with reason: :baseline_red and emits the error-JSON document (aborted: true, populated baseline.failed) — mutation never starts. This catches the "your suite is already red" case so kill rates are never computed against a broken baseline.

The guard works even though the coverage → baseline → mutation phase order drains ExUnit.Server (each ExUnit.run/0 consumes the server's registered-module list) and Code.require_file/1 is one-shot per path. Both the coverage and baseline phases explicitly re-register each cited module with ExUnit.Server.add_module/2 before their own ExUnit.run/0 so baseline sees the cited modules in the registry and reports real failures. The production-condition regression test for this lives in test/mutagen_ex/baseline_red_guard_test.exs (it drives the real ExUnit.Server, not a fake), with end-to-end coverage in test/mutagen_ex/end_to_end_test.exs (:baseline_red_scenario).

An earlier development snapshot documented this as a limitation (the guard "did not trip"). That was fixed by the re-registration above; the manual mix test-first workaround the old docs suggested is no longer required.

Known limitations

  • A :case_drop mutation can remove the only base case of a guarded recursive function. If the remaining guard rejects the base value, the BEAM raises CaseClauseError. The report classifies the site as killed.
  • Mutagen reaps timed-out work spawned by the site runner. Work a test starts through a host-owned supervisor, such as Task.Supervisor.async_nolink/2 or DynamicSupervisor.start_child/2, can outlive the timeout. The host supervisor owns those children.
  • Macros can expand differently when a probe wraps an argument whose AST they pattern-match. The expansion can discard the counter, and a test with weak assertions can pass despite the changed behavior. Mutagen may then report a site as unexecuted even though the original suite ran it.
  • Probes, coverage, and mutation runs only observe work during the suite in the local node with the current module version. Code running on another node, after the suite ends, or in a process still using old code escapes those checks.

Changed-code planning

Planning compares changed lines and the enclosing group's source fingerprint. An edit to a module attribute such as @limit, a defguard, a defmacro, a defdelegate target, alias, import, use options, or defstruct defaults can change runtime behavior and still plan zero runtime sites. Callers of a changed private helper keep their own fingerprints. For a deletion hunk, Mutagen considers the remaining lines next to the deletion. A deletion can plan zero sites when neither neighbor touches a runtime site.

Runtime classification uses syntax. A named def or defp outside quote qualifies even if only compile-time code calls it. An expression inside unquote still falls outside the runtime plan when it sits inside quote, even if a runtime call evaluates the expression.

Performance

The source checkout includes a benchmark harness at scripts/helper_scripts/bench_ast_perf.exs. It drives the full pipeline against scripts/helper_scripts/bench_fixtures/wrd25_200sites/ and reports wall-clock, per-site time, peak :erlang.memory/0, and the SHA-256 of the emitted NDJSON. Save a baseline with the following command. Replace --baseline with --compare to compare a fresh run to the saved file.

mix run scripts/helper_scripts/bench_ast_perf.exs --baseline /tmp/bench.json

The .25 AST/perf epic measured a 1.66× wall-clock speedup on the wrd25 fixture between commit 978a995 (pre-.25) and 78b022f (post-.25.6); see the engineering appendix in CHANGELOG.md for the full table and the follow-up triage of the gap to the spec's documented 2-4× target. The byte-identity of the mutation results (site IDs, before/after slices, kill/survive verdicts, kill_rate) is preserved across the refactor — only timing and the diagnostic warning text changed.

Specs

The behavioural contract for mutagen_ex lives in .spec/:

  • .spec/specs/cli.spec.md — mix mutagen command surface
  • .spec/specs/mutation_pipeline.spec.md — orchestration state machine
  • .spec/specs/mutators.spec.md — mutator catalog and predicates
  • .spec/specs/json_schema.spec.md — version 2 output document shape
  • .spec/specs/coverage.spec.md — coverage phase
  • .spec/specs/mutation_enumeration.spec.md — site enumeration rules
  • .spec/specs/scope_resolution.spec.md — --scope resolution
  • .spec/specs/test_selection.spec.md — --tests resolution
  • .spec/decisions/ — durable architectural decisions

When the implementation and the specs disagree, fix the spec first (see .spec/AGENTS.md).

Development

mix compile --warnings-as-errors
mix test

No third-party runtime dependencies. Requires Elixir 1.18.0 or later for ExUnit case registration and Erlang/OTP 27 or later for JSON and trace sessions.

Test suites

The default mix test run excludes subprocess integration tests, the end-to-end fixture runs, and the coverage spikes.

TagWhat it coversHow to run
:e2e_slowFull mix mutagen pipeline against the lane fixture; takes minutes per scenario.mix test --only e2e_slow
:spikeC1/C2 integration spikes under test/mutagen_ex/integration/ — ~500 cover lifecycles per default run. These are the gating decision artifact for the in-process pipeline (mutagen.decision.in_process_pipeline); they must stay runnable, just not on every default invocation.mix test --only spike
:host_setup_integrationReal Postgres Sandbox and Mox through a path dependency and an installed archive.mix test --only host_setup_integration test/integration/host_setup_test.exs

mix test.integration includes the Postgres fixture. It defaults to the /tmp Unix socket, port 5432, and the current OS user. Set HOST_SETUP_DB_HOST, HOST_SETUP_DB_PORT, HOST_SETUP_DB_USER, HOST_SETUP_DB_PASSWORD, or HOST_SETUP_DB_NAME to override those settings. The fixture adds a unique suffix to its database name and drops the database after the run. Its database user needs permission to create databases.

The C2 spike accepts MUTAGEN_SPIKE_ITERATIONS=<n> to override the loop count. Default is 10; the original gating run used 100:

# Quick smoke (10 iterations)
mix test --only spike

# Full gating run (100 iterations, ~as long as the original spike)
MUTAGEN_SPIKE_ITERATIONS=100 mix test --only spike

C1 retains a fixed 100-iteration count — its loop measures restore contract fidelity across module shapes, where the cycle count is the contract.