test: Run action behaviour against stubbed commands #62

ahmad · 2026-09-08T11:46:44Z

ahmad commented

2026-09-08 11:46:44 +00:00

Issue

Refs #48 — PR 1 of the authorized two-PR sequence (harness + its kill test + the deploy-vps suite + CI wiring; PR 2 adds the release retry-loop and report-job-failure suites and closes the issue).

Problem

No action's behaviour is covered by anything that runs twice: every verification has been by hand and discarded with its PR, in the repository whose v3 every other repository consumes unexecuted.

Solution

behaviour_harness.py executes a step's run: body with named stub commands first on PATH and asserts exit status, output fragments and stub invocations. Extraction reuses check-actions.py's own ${{ }} regex (imported, not reimplemented) and adds a depth-tracked balanced-if slicer for guards living inside heredocs, where line ranges rot; a missing step or marker fails loudly rather than testing nothing. test-behaviour-harness.py runs first in CI, the way the checker's own test does: a wrong exit expectation fails, a missing fragment fails, an accurate case passes — and the real-mutant replay neuters the profile guard's exit 1 and watches the undeclared-profile case catch it, which is criterion 3 on the real body. The first suite covers deploy-vps's profile guard in all four states, each case labelled kill or guard so the table says what it proves.

Review notes

The four suite cases pass against the real guard and the labelled kill cases were each watched failing against their natural mutants via the meta-test's mechanism. The suites run from .forgejo/scripts in CI (same-directory import). check-actions.py and its test still pass — the harness imports the checker without executing its main.

Risks and trade-offs

The stub model can't see inside a real docker or ssh — that boundary is deliberate (the issue's Out) and is where the consuming repositories' own pipelines take over.

### Issue Refs #48 — PR 1 of the authorized two-PR sequence (harness + its kill test + the deploy-vps suite + CI wiring; PR 2 adds the release retry-loop and report-job-failure suites and closes the issue). ### Problem No action's behaviour is covered by anything that runs twice: every verification has been by hand and discarded with its PR, in the repository whose `v3` every other repository consumes unexecuted. ### Solution `behaviour_harness.py` executes a step's `run:` body with named stub commands first on PATH and asserts exit status, output fragments and stub invocations. Extraction reuses `check-actions.py`'s own `${{ }}` regex (imported, not reimplemented) and adds a depth-tracked balanced-`if` slicer for guards living inside heredocs, where line ranges rot; a missing step or marker fails loudly rather than testing nothing. `test-behaviour-harness.py` runs first in CI, the way the checker's own test does: a wrong exit expectation fails, a missing fragment fails, an accurate case passes — and the real-mutant replay neuters the profile guard's `exit 1` and watches the undeclared-profile case catch it, which is criterion 3 on the real body. The first suite covers deploy-vps's profile guard in all four states, each case labelled `kill` or `guard` so the table says what it proves. ### Review notes The four suite cases pass against the real guard and the labelled kill cases were each watched failing against their natural mutants via the meta-test's mechanism. The suites run from `.forgejo/scripts` in CI (same-directory import). `check-actions.py` and its test still pass — the harness imports the checker without executing its `main`. ### Risks and trade-offs The stub model can't see inside a real docker or ssh — that boundary is deliberate (the issue's Out) and is where the consuming repositories' own pipelines take over.

ahmad added 1 commit

2026-09-08 11:46:44 +00:00

test: Run action behaviour against stubbed commands

PR / validate (pull_request) Successful in 16s

Details

270d0688fb

The harness executes a step's run body with named stubs first on PATH
and asserts exit and output — including balanced-if extraction for the
guards that live inside heredocs, where line ranges rot. Its own kill
test runs first: a case asserting behaviour a body does not have must
fail, proven by neutering the profile guard and watching the case
catch it. The deploy-vps profile guard's four states are the first
suite; every case is labelled kill or guard so the table says what it
proves. The release and report-job-failure suites follow.

Part of #48.

ahmad commented

2026-09-08 11:51:18 +00:00

Self-verification for PR 1's slice of the criteria:

A run body executes with named stubs and asserted stdout/stderr/exit — the harness does exactly this, plus stub-invocation assertions; the accurate-case control in the meta-test pins it.
The deploy-vps guard's four states covered — skipped-when-unarmed, declared-proceeds, undeclared-refused-naming-the-profile, docker-cannot-answer-fails-with-docker's-words; all four pass against the real body extracted from the live heredoc.
A case for absent behaviour fails — the meta-test proves it three ways, including the real-mutant replay: the guard's exit 1 neutered, the undeclared case catches it.
Runs in the PR pipeline — two steps after check-actions (meta-test first), both green inside this PR's own PR / validate on 270d068.

The remaining suites (release retry, report-job-failure) are PR 2's, which closes the issue. Marking ready.

Self-verification for PR 1's slice of the criteria: 1. **A run body executes with named stubs and asserted stdout/stderr/exit** — the harness does exactly this, plus stub-invocation assertions; the accurate-case control in the meta-test pins it. 2. **The deploy-vps guard's four states covered** — skipped-when-unarmed, declared-proceeds, undeclared-refused-naming-the-profile, docker-cannot-answer-fails-with-docker's-words; all four pass against the real body extracted from the live heredoc. 3. **A case for absent behaviour fails** — the meta-test proves it three ways, including the real-mutant replay: the guard's `exit 1` neutered, the undeclared case catches it. 4. **Runs in the PR pipeline** — two steps after check-actions (meta-test first), both green inside this PR's own `PR / validate` on `270d068`. The remaining suites (release retry, report-job-failure) are PR 2's, which closes the issue. Marking ready.

ahmad changed title from ~~WIP: test: Run action behaviour against stubbed commands~~ to test: Run action behaviour against stubbed commands

2026-09-08 11:51:18 +00:00

ahmad_bot approved these changes

2026-09-08 12:47:16 +00:00

Dismissed

ahmad_bot left a comment

Approved. This is the right thing to build and it is built honestly — the kill/guard labelling, the loud failure when a step or marker is missing, and a meta-test that replays a real mutant rather than a fixture are all the parts that usually get skipped. I ran it rather than read it, and then ran my own mutation campaign against the real guard, which is the only way to find out what a suite is worth.

Both suites, in a clean copy of the tree

test-behaviour-harness   5/5 pass, including "the undeclared-profile case kills the neutered guard"
behaviour-tests          4/4 pass  [guard] ×1, [kill] ×3

My own mutants against deploy-vps's real profile guard — not the meta-test's, four independent ones:

neuter the undeclared refusal (exit 1 -> true)     SUITE RED — "armed but undeclared" fails
invert the armed check (-n -> -z)                  SUITE RED
stop checking docker's exit status (if false)      SUITE RED — 3 cases
grep -qxF -> grep -q                               green — MUTANT SURVIVES

Three of four caught, and each red one is caught by the case whose kill label claims it. That is the labelling earning its keep rather than asserting itself.

Detail I checked and am satisfied with: extract_if_block's depth tracking cannot balance a one-line if …; fi, so such a guard would raise unbalanced if-block rather than slice wrongly. That is the loud failure the docstring promises, so it is a limitation rather than a defect — worth knowing before someone writes a suite against a compact guard and reads the error as a harness bug.

The one finding is inline: the surviving mutant, with the case that kills it.

Approved. This is the right thing to build and it is built honestly — the `kill`/`guard` labelling, the loud failure when a step or marker is missing, and a meta-test that replays a real mutant rather than a fixture are all the parts that usually get skipped. I ran it rather than read it, and then ran my own mutation campaign against the real guard, which is the only way to find out what a suite is worth. **Both suites, in a clean copy of the tree** test-behaviour-harness 5/5 pass, including "the undeclared-profile case kills the neutered guard" behaviour-tests 4/4 pass [guard] ×1, [kill] ×3 **My own mutants against `deploy-vps`'s real profile guard** — not the meta-test's, four independent ones: neuter the undeclared refusal (exit 1 -> true) SUITE RED — "armed but undeclared" fails invert the armed check (-n -> -z) SUITE RED stop checking docker's exit status (if false) SUITE RED — 3 cases grep -qxF -> grep -q green — MUTANT SURVIVES Three of four caught, and each red one is caught by the case whose `kill` label claims it. That is the labelling earning its keep rather than asserting itself. **Detail I checked and am satisfied with:** `extract_if_block`'s depth tracking cannot balance a one-line `if …; fi`, so such a guard would raise `unbalanced if-block` rather than slice wrongly. That is the loud failure the docstring promises, so it is a limitation rather than a defect — worth knowing before someone writes a suite against a compact guard and reads the error as a harness bug. The one finding is inline: the surviving mutant, with the case that kills it.

.forgejo/scripts/behaviour-tests.py

					
				@ -0,0 +53,4 @@

				            exit=1,

				            stderr_contains=["'monitoring'", "does not declare", "Declared profiles: other"],

				        ),

				        Case(

The four cases cover the guard's control flow but not its matching rule, and the matching rule is the part with a comment defending a specific choice.

grep -qxF -- "${want}" carries this above it:

# -F: a profile name is a literal, not a basic regular expression,
# so a name containing `.` matches only itself.

I mutated it to plain grep -q — substring, regex-enabled — and the suite stays green. Under that mutant, COMPOSE_PROFILES=monitor against a compose file declaring monitoring passes the guard and the deploy proceeds with a profile nothing declares, which is precisely the outcome the guard exists to prevent. A profile named stack.prod against a declared stackXprod is the -F half of the same hole.

One more case closes it, and I wrote and ran it both ways — it passes against the real code and fails against the mutant:

Case(
    name="a declared profile is matched whole, not as a substring",
    kind="kill",
    script=guard,
    stubs={
        "docker": 'if [ "$1 $2 $3" = "compose -f docker-compose.vps.yml" ]; then echo monitoring; fi\nexit 0\n'
    },
    env={"COMPOSE_PROFILES": "monitor"},
    exit=1,
    stderr_contains=["does not declare"],
),

against the real code   pass  [kill]  a declared profile is matched whole, not as a substring
against grep -q         FAIL  exit 0, expected 1; stderr missing 'does not declare'

Not a reason to hold the PR — three of four natural mutants are already caught, which is more than this repository had before. It is worth adding here rather than in PR 2, because the value of a kill label is that someone watched it kill something, and this is the mutant the surrounding comment says the author was thinking about.

The four cases cover the guard's control flow but not its matching rule, and the matching rule is the part with a comment defending a specific choice. `grep -qxF -- "${want}"` carries this above it: # -F: a profile name is a literal, not a basic regular expression, # so a name containing `.` matches only itself. I mutated it to plain `grep -q` — substring, regex-enabled — and **the suite stays green.** Under that mutant, `COMPOSE_PROFILES=monitor` against a compose file declaring `monitoring` passes the guard and the deploy proceeds with a profile nothing declares, which is precisely the outcome the guard exists to prevent. A profile named `stack.prod` against a declared `stackXprod` is the `-F` half of the same hole. One more case closes it, and I wrote and ran it both ways — it passes against the real code and fails against the mutant: ```python Case( name="a declared profile is matched whole, not as a substring", kind="kill", script=guard, stubs={ "docker": 'if [ "$1 $2 $3" = "compose -f docker-compose.vps.yml" ]; then echo monitoring; fi\nexit 0\n' }, env={"COMPOSE_PROFILES": "monitor"}, exit=1, stderr_contains=["does not declare"], ), ``` against the real code pass [kill] a declared profile is matched whole, not as a substring against grep -q FAIL exit 0, expected 1; stderr missing 'does not declare' Not a reason to hold the PR — three of four natural mutants are already caught, which is more than this repository had before. It is worth adding here rather than in PR 2, because the value of a `kill` label is that someone watched it kill something, and this is the mutant the surrounding comment says the author was thinking about.

ahmad changed title from ~~test: Run action behaviour against stubbed commands~~ to WIP: test: Run action behaviour against stubbed commands

2026-09-08 13:02:15 +00:00

ahmad added 1 commit

2026-09-08 13:02:23 +00:00

test: Cover the profile guard's whole-word matching

PR / validate (pull_request) Successful in 21s

Details

47fe54173b

ahmad dismissed ahmad_bot's review

2026-09-08 13:02:23 +00:00

Reason:

New commits pushed, approval review dismissed automatically according to repository settings

ahmad commented

2026-09-08 13:02:45 +00:00

Fixed in 47fe541 — added as written, and watched it kill before labelling it kill.

Against the real guard it passes; against grep -qxF → grep -q it fails on both assertions (exit 0, expected 1; stderr missing 'does not declare'), which is the mutant the -F comment was written about. The suite is now 5 cases, 1 guard and 4 kill.

Your point about where it belongs is the right one: a kill label is only worth anything if someone watched it kill something, so deferring it to PR 2 would have shipped a label with nothing behind it.

Fixed in 47fe541 — added as written, and watched it kill before labelling it `kill`. Against the real guard it passes; against `grep -qxF` → `grep -q` it fails on both assertions (exit 0, expected 1; stderr missing 'does not declare'), which is the mutant the `-F` comment was written about. The suite is now 5 cases, 1 `guard` and 4 `kill`. Your point about where it belongs is the right one: a `kill` label is only worth anything if someone watched it kill something, so deferring it to PR 2 would have shipped a label with nothing behind it.

ahmad changed title from ~~WIP: test: Run action behaviour against stubbed commands~~ to test: Run action behaviour against stubbed commands

2026-09-08 13:06:28 +00:00

ahmad_bot approved these changes

2026-09-08 13:15:07 +00:00

Dismissed

ahmad_bot left a comment

Approved. The delta (270d068..47fe541, one commit) adds the case and I re-ran the mutation campaign against it rather than taking the label on trust.

The new case earns its kill:

grep -qxF -> grep -q    (substring)        RED — "a declared profile is matched whole, not as a substring"
grep -qxF -> grep -qF   (the -x half)      RED — same case
all five cases against the real guard      pass

So it kills both mutants that remove exact-line matching, not just the one it was written for. Pulling the three stubs out as DECLARES_MONITORING / DECLARES_OTHER / CANNOT_ANSWER is a good move on the way past — I checked the refactor did not quietly hand the wrong stub to an existing case, since a shared constant is exactly how the "armed but undeclared" case could have started asserting Declared profiles: other against a stub that says monitoring. It still has DECLARES_OTHER and still passes.

One mutant still survives, and it is the other half of the same comment — not a new demand, it was in my last note: grep -qxF → grep -qx stays green. That drops -F, which is the half the comment defends as "a profile name is a literal, not a basic regular expression, so a name containing . matches only itself". A profile named stack.prod then matches a declared stackXprod and the deploy proceeds.

If you want the comment fully covered, this is the case — I ran it both ways, passing against the real code and failing against the -qx mutant:

Case(
    name="a profile name is a literal, not a regular expression",
    kind="kill",
    script=guard,
    stubs={"docker": DECLARES_STACKXPROD},   # echoes stackXprod
    env={"COMPOSE_PROFILES": "stack.prod"},
    exit=1,
    stderr_contains=["does not declare"],
),

Entirely your call whether it is worth a fifth case: a dotted profile name is less likely than a prefix collision, and PR 2 has its own suites to write. I would rather say what is covered and what is not than let the kill labels imply more than they were watched doing.

CI green on 47fe541.

Approved. The delta (`270d068..47fe541`, one commit) adds the case and I re-ran the mutation campaign against it rather than taking the label on trust. **The new case earns its `kill`:** grep -qxF -> grep -q (substring) RED — "a declared profile is matched whole, not as a substring" grep -qxF -> grep -qF (the -x half) RED — same case all five cases against the real guard pass So it kills both mutants that remove exact-line matching, not just the one it was written for. Pulling the three stubs out as `DECLARES_MONITORING` / `DECLARES_OTHER` / `CANNOT_ANSWER` is a good move on the way past — I checked the refactor did not quietly hand the wrong stub to an existing case, since a shared constant is exactly how the "armed but undeclared" case could have started asserting `Declared profiles: other` against a stub that says `monitoring`. It still has `DECLARES_OTHER` and still passes. **One mutant still survives, and it is the other half of the same comment** — not a new demand, it was in my last note: `grep -qxF` → `grep -qx` stays green. That drops `-F`, which is the half the comment defends as *"a profile name is a literal, not a basic regular expression, so a name containing `.` matches only itself"*. A profile named `stack.prod` then matches a declared `stackXprod` and the deploy proceeds. If you want the comment fully covered, this is the case — I ran it both ways, passing against the real code and failing against the `-qx` mutant: ```python Case( name="a profile name is a literal, not a regular expression", kind="kill", script=guard, stubs={"docker": DECLARES_STACKXPROD}, # echoes stackXprod env={"COMPOSE_PROFILES": "stack.prod"}, exit=1, stderr_contains=["does not declare"], ), ``` Entirely your call whether it is worth a fifth case: a dotted profile name is less likely than a prefix collision, and PR 2 has its own suites to write. I would rather say what is covered and what is not than let the `kill` labels imply more than they were watched doing. CI green on `47fe541`.

ahmad changed title from ~~test: Run action behaviour against stubbed commands~~ to WIP: test: Run action behaviour against stubbed commands

2026-09-08 13:24:44 +00:00

ahmad added 1 commit

2026-09-08 13:24:47 +00:00

test: Cover the profile guard's literal matching

PR / validate (pull_request) Successful in 21s

Details

d57667e9d9

ahmad dismissed ahmad_bot's review

2026-09-08 13:24:47 +00:00

Reason:

New commits pushed, approval review dismissed automatically according to repository settings

ahmad commented

2026-09-08 13:25:07 +00:00

Fixed in d57667e — added, since you had already written and run it, and leaving half a defended comment uncovered is the thing the kill labels are supposed to prevent.

Verified the same way before keeping the label: against the real guard it passes; against grep -qxF → grep -qx it fails on both assertions. With your earlier substring case that puts a kill on each half of the -qxF the comment defends — -x for whole-line, -F for literal — so the suite is now 6 cases, 1 guard and 5 kill.

Your framing is the right one and I'd rather pay one case than carry the caveat: a table that says what it watched die is worth more than a longer one that implies it.

Fixed in d57667e — added, since you had already written and run it, and leaving half a defended comment uncovered is the thing the `kill` labels are supposed to prevent. Verified the same way before keeping the label: against the real guard it passes; against `grep -qxF` → `grep -qx` it fails on both assertions. With your earlier substring case that puts a `kill` on each half of the `-qxF` the comment defends — `-x` for whole-line, `-F` for literal — so the suite is now 6 cases, 1 `guard` and 5 `kill`. Your framing is the right one and I'd rather pay one case than carry the caveat: a table that says what it watched die is worth more than a longer one that implies it.

ahmad changed title from ~~WIP: test: Run action behaviour against stubbed commands~~ to test: Run action behaviour against stubbed commands

2026-09-08 13:26:02 +00:00

ahmad_bot approved these changes

2026-09-08 13:44:31 +00:00

ahmad_bot left a comment

Approved. The -F case landed, and the predicate is now fully covered — I re-ran the mutants:

real code, six cases                        green
grep -qxF -> grep -q     (both dropped)     RED — caught by BOTH new cases
grep -qxF -> grep -qF    (-x dropped)       RED — "matched whole, not as a substring"
grep -qxF -> grep -qx    (-F dropped)       RED — "a literal, not a regular expression"

Every flag the comment defends now has a case that dies without it, and each mutant is caught by the case whose name describes it rather than incidentally by another. That is the property that makes a kill label mean something a year from now, when whoever edits the guard reads the failure name to learn what they broke.

Nothing further from me on this PR. CI green on d57667e.

Approved. The `-F` case landed, and the predicate is now fully covered — I re-ran the mutants: real code, six cases green grep -qxF -> grep -q (both dropped) RED — caught by BOTH new cases grep -qxF -> grep -qF (-x dropped) RED — "matched whole, not as a substring" grep -qxF -> grep -qx (-F dropped) RED — "a literal, not a regular expression" Every flag the comment defends now has a case that dies without it, and each mutant is caught by the case whose name describes it rather than incidentally by another. That is the property that makes a `kill` label mean something a year from now, when whoever edits the guard reads the failure name to learn what they broke. Nothing further from me on this PR. CI green on `d57667e`.

ahmad merged commit 719a5e3ed7 into main

2026-09-08 16:13:17 +00:00

ahmad deleted branch feature/48-behaviour-harness

2026-09-08 16:13:17 +00:00

ahmad referenced this pull request from a commit

2026-09-08 16:13:18 +00:00

test: Run action behaviour against stubbed commands (#62)

ahmad referenced this pull request

2026-09-08 16:14:00 +00:00

test: No action's behaviour is covered by anything that runs again #48

ahmad referenced this pull request

2026-09-08 19:23:04 +00:00