MutagenEx.CoverageRunner (mutagen_ex v0.2.0)

Copy Markdown View Source

Runs the cited test suite once under :cover to determine which source lines those tests exercise.

Contract: mutagen.coverage r1-r5, r7.

Lifecycle invariants

  • r1 — refuse another coverage session. The runner can replace an empty Cover server. If it owns compiled or imported modules, the runner returns {:error, :cover_already_running, ...} before instrumenting scope modules or running ExUnit.
  • r2 — cover stopped after return. Whether run/1 succeeds or errors, :cover.stop/0 is invoked at least once. The call is wrapped in try/rescue so repeated stops are idempotent.
  • r3 — modules de-instrumented. After :cover.stop/0, :code.which/1 for every cover-instrumented module returns a real .beam path (or :non_existing) rather than :cover_compiled.
  • r4 — forced ExUnit configuration. Runs ExUnit serially with the configured seed and an unbounded failure count.
  • r5 — scope-filtered output. The returned covered_lines map is keyed by relative file path and contains entries only for files that back the modules in cfg.in_scope_modules.
  • r7 — no disk writes. run/1 does not modify any file on disk.

Input shape

run/1 takes a map with at least:

  • :seed — non-negative integer, the ExUnit seed.
  • :in_scope_modules — list of {module, source_file} tuples. The source_file is a relative path (the same string scope records carry); it's used both as the cover-compile target (we locate the module's .beam and instrument it) and as the result key.
  • :test_filter — %MutagenEx.TestSelector.TestFilter{}. Drives the ExUnit include/exclude/files configuration.

Optional:

  • :ex_unit — module implementing MutagenEx.Test.ExUnitFacade. Defaults to MutagenEx.Test.ExUnit, which delegates to the real ExUnit. The seam exists so the failure-path tests can assert the runner aborts before any ExUnit call.
  • :cover — module implementing MutagenEx.Test.CoverFacade. Defaults to MutagenEx.Test.Cover, which delegates to Erlang's :cover module. Existence solely as a test seam.
  • :test_loader — (path :: String.t() -> any()) — overrides Code.require_file/1. Tests use this to track that test files are loaded once, not per-phase.
  • :test_modules — [{module(), MutagenEx.TestModuleDiscovery.module_cfg()}]. Re-registered with ExUnit.Server.add_module/2 before the coverage phase's ExUnit.run/0 so the cited modules appear in the registry whenever a prior ExUnit.run/0 in the same BEAM drained it (the in-suite multi-scenario hazard documented in mutagen.mutation_pipeline.r10). Defaults to [] (preserves pre-mutagen-wrd.38 behaviour for callers that haven't wired the production payload through; in that mode coverage is vulnerable to the Code.require_file/1 cache hazard whenever it runs after another ExUnit.run/0 in the same BEAM).
  • :ex_unit_server — module implementing MutagenEx.Test.ExUnitServerFacade (the registration seam). Defaults to MutagenEx.Test.ExUnitServer.

Why coverage re-registers cited test modules

ExUnit.Server consumes its registered-module list at every ExUnit.run/0 invocation. The use ExUnit.Case __after_compile__ hook that performs the initial registration only fires when the cited test file is freshly evaluated, and Code.require_file/1 is one-shot per path in a BEAM lifetime.

Production callers running a single mix mutagen invocation per BEAM are unaffected — coverage is the first phase to call ExUnit.run/0, so the __after_compile__ registration is still in the server's registry when coverage runs.

The hazard appears when two scenarios in the same BEAM cite the same *_test.exs file: by the time the second scenario's coverage phase runs, the prior scenario's ExUnit.run/0 has drained the registry and Code.require_file/1 on the same path is a cached no-op. Without explicit re-registration, coverage's ExUnit.run/0 would see zero modules, record zero covered lines, and downstream enumeration would silently produce zero sites (mutagen-wrd.38).

The fix mirrors MutagenEx.Baseline (mutagen-wrd.37) and MutagenEx.MutationRunner.MutationLoop (per site): explicitly call ExUnit.Server.add_module/2 for each cited test module before invoking ExUnit.run/0. After loading the cited files, coverage discovers cases through MutagenEx.TestModuleDiscovery.discover/1 unless the caller provided :test_modules. The :ex_unit_server seam defaults to MutagenEx.Test.ExUnitServer and lets tests record the calls.

S2 spike findings carried forward

  • :cover lives under OTP's lib/tools-*/ebin and is not on the default Mix code path. ensure_cover_loadable/0 does the same path-append the C1 spike did. This is shared with the C1 fixture; production code paths don't always have :cover loaded.
  • :cover.compile_beam/1 requires the Dbgi chunk. Mix's test env defaults to debug_info: true, so project .beam files already carry it. We do not flip Code.compiler_options/1.

Summary

Types

Reasons the runner can return as the second element of {:error, _, _}.

Successful return.

Functions

Run the cited tests under :cover and return per-file covered line sets.

Types

error_reason()

@type error_reason() ::
  :cover_already_running
  | :cover_module_unavailable
  | :module_beam_missing
  | :cover_compile_failed
  | :ex_unit_run_failed
  | :test_file_load_failed
  | :invalid_input

Reasons the runner can return as the second element of {:error, _, _}.

ok_result()

@type ok_result() :: %{
  covered_lines: %{optional(String.t()) => MapSet.t(pos_integer())},
  coverable_lines: %{optional(String.t()) => MapSet.t(pos_integer())},
  instrumented_modules: [module()],
  test_modules: [MutagenEx.TestModuleDiscovery.entry()]
}

Successful return.

Functions

run(input)

@spec run(map()) :: {:ok, ok_result()} | {:error, error_reason(), map()}

Run the cited tests under :cover and return per-file covered line sets.

See module doc for input/output shape and lifecycle invariants.