MutagenEx.Baseline (mutagen_ex v0.2.0)

Copy Markdown View Source

Pre-mutation test pass: do the cited tests pass on unmodified code?

Contract: mutagen.mutation_pipeline r1, r2.

Responsibilities (and what it deliberately is NOT)

  • r1. run/1 runs the cited tests once with no mutation applied. If any test fails, return {:error, :baseline_red, %{failures: [{module, name}, ...], passed: int}}. The orchestrator then routes the failure into the universal error-JSON shape.
  • r2. Runs ExUnit serially with the configured seed and an unbounded failure count. Detects async: true test modules in the cited filter and includes warnings in the report.

Baseline does NOT own:

  • Per-mutation timeouts (no timeout wrapping; if the baseline hangs, the user can Ctrl-C — per mutagen.decision.mutation_loop_private).
  • Stdout/stderr suppression (baseline runs ExUnit's own formatters by default; the orchestrator can ExUnit.CaptureIO-wrap the call if it doesn't want the banner).
  • Cover instrumentation (that's MutagenEx.CoverageRunner).
  • Loading test files (that's the orchestrator: see MutagenEx.CoverageRunner for the precedent; baseline does it once because tests can run only once per require_file per r10).

Why baseline re-registers cited test modules

ExUnit.Server consumes its registered-module list at every ExUnit.run/0 invocation. The use ExUnit.Case __after_compile__ hook that performs the initial registration only fires when the cited test file is freshly evaluated, and Code.require_file/1 is one-shot per path in a BEAM lifetime.

When the coverage phase runs first it loads the cited test files, fires the hook, and consumes the registered modules during its own ExUnit.run/0. By the time baseline runs, the registry is empty and Code.require_file/1 on the same paths is a cached no-op — without explicit re-registration, baseline's ExUnit.run/0 would see zero modules and return %{total: 0, failures: 0}, silently classifying every red baseline as green (mutagen-wrd.37).

The fix mirrors what MutagenEx.MutationRunner.MutationLoop already does per site: explicitly call ExUnit.Server.add_module/2 for each cited test module before invoking ExUnit.run/0. After loading the cited files, baseline discovers cases through MutagenEx.TestModuleDiscovery.discover/1 unless the caller provided :test_modules. The :ex_unit_server seam defaults to MutagenEx.Test.ExUnitServer and lets tests record the calls.

Input shape

A map with:

  • :seed — non-negative integer.
  • :test_filter — %MutagenEx.TestSelector.TestFilter{}.

Optional:

  • :ex_unit — module implementing MutagenEx.Test.ExUnitFacade (test seam). Defaults to MutagenEx.Test.ExUnit, which delegates to the real ExUnit. Tests pass their own module with the same callback surface.
  • :test_loader — (path -> any) overriding Code.require_file/1.
  • :test_modules — [{module(), MutagenEx.TestModuleDiscovery.module_cfg()}]. Re-registered with ExUnit.Server.add_module/2 before ExUnit.run/0 so the cited modules appear in the registry that the prior phase's ExUnit.run/0 drained. Defaults to [] (preserves pre-mutagen-wrd.37 behaviour for callers that haven't wired the production payload through; in that mode baseline is vulnerable to the Code.require_file/1 cache hazard whenever it runs after another ExUnit.run/0 in the same BEAM).
  • :ex_unit_server — module implementing MutagenEx.Test.ExUnitServerFacade (the registration seam). Defaults to MutagenEx.Test.ExUnitServer.
  • :ast_cache — MutagenEx.AstCache.t(). Optional. When present, detect_async_modules/1 (the r2 async-warning path) looks each cited test file up in the cache via AstCache.get/2 and consumes the cached {ast, _source} directly — no re-read from disk. On :error (cache miss), the implementation falls back to the pre-.25 File.read/1 path as a safety net and logs the miss. When :ast_cache is absent, the fall-back path is the only path (preserves pre-.25 behaviour for callers that haven't wired the cache through yet). See mutagen.coverage.r9.

Summary

Types

Reasons the baseline can return.

Successful baseline result.

Functions

Run the baseline. Returns {:ok, %{passed:, failed: 0, failures: [], warnings: [...]}} on full success or {:error, :baseline_red, %{failures: [...], passed: n}} on any failing cited test.

Types

error_reason()

@type error_reason() ::
  :baseline_red | :ex_unit_run_failed | :test_file_load_failed | :invalid_input

Reasons the baseline can return.

ok_result()

@type ok_result() :: %{
  passed: non_neg_integer(),
  failed: non_neg_integer(),
  failures: [{module(), String.t() | atom()}],
  warnings: [String.t()]
}

Successful baseline result.

Functions

run(input)

@spec run(map()) :: {:ok, ok_result()} | {:error, error_reason(), map()}

Run the baseline. Returns {:ok, %{passed:, failed: 0, failures: [], warnings: [...]}} on full success or {:error, :baseline_red, %{failures: [...], passed: n}} on any failing cited test.