test: No action's behaviour is covered by anything that runs again

ahmad commented

2026-09-05 07:37:04 +00:00

Owner

Context

check-actions.py parses every action.yml, checks the fields a composite action needs, and runs bash -n over each run: body with ${{ }} substituted. It cannot execute one, so no action's behaviour is tested by anything that runs again.

Every recent change here has been verified by hand, by extracting a run: body into a file, stubbing the commands it calls, and driving it through its cases:

release@v3        the retry loop, against a stub curl returning 500 / 000 / 409 / 201
report-job-failure  four log states and three "cannot post" paths, against a local HTTP server
deploy-vps        the profile guard, against a stub `docker compose config --profiles`

Each of those found something a reading would not have: a || echo 000 that concatenated with curl's own 000 to give 000000, a mutation that was a syntax error rather than a behavioural change, and an unguarded curl reporting HTTP with an empty status. None of that verification survives the pull request it was written in.

This is the repository whose v3 tag every other repository consumes, and a change here reaches all of them unexecuted — the README says as much. Hand verification that is discarded is the weakest possible answer to that.

Scope

In: a harness that runs an action's run: bodies against stubbed commands and asserts on output and exit status, plus the existing verifications for release, report-job-failure and deploy-vps written into it.
Out: executing actions against a real host, registry or Forgejo — that is what the consuming repositories' own pipelines are for.
Out: changing any action's behaviour.

Acceptance criteria

Given an action with a run: body, when the harness runs, then that body can be executed with named commands stubbed and its stdout, stderr and exit status asserted.
Given the guard in deploy-vps that refuses an undeclared profile, when the harness runs, then all four of its states are covered.
Given a case is written for behaviour an action does not have, when the harness runs, then it fails — a case that passes against both the fixed and broken versions is worse than none.
Given the PR pipeline, when the harness is added, then it runs there alongside check-actions.py.

Notes

The third criterion is the one that matters and the one most likely to be quietly skipped. Every verification listed above was written by first watching the case fail against the unfixed code; a harness that only ever ran against working code would have recorded four passing tables and caught none of the four defects.

Extraction is the fiddly part: a run: body inside a heredoc keeps the heredoc's indentation, so textwrap.dedent on the YAML-parsed string is not always enough — the deploy-vps guard had to be pulled from the raw file by line range. Worth solving once in the harness rather than in each verification.

check-actions.py already substitutes ${{ }} and knows how to find every action; the harness should reuse that rather than reimplement it.

### Context `check-actions.py` parses every `action.yml`, checks the fields a composite action needs, and runs `bash -n` over each `run:` body with `${{ }}` substituted. It cannot execute one, so **no action's behaviour is tested by anything that runs again**. Every recent change here has been verified by hand, by extracting a `run:` body into a file, stubbing the commands it calls, and driving it through its cases: ``` release@v3 the retry loop, against a stub curl returning 500 / 000 / 409 / 201 report-job-failure four log states and three "cannot post" paths, against a local HTTP server deploy-vps the profile guard, against a stub `docker compose config --profiles` ``` Each of those found something a reading would not have: a `|| echo 000` that concatenated with curl's own `000` to give `000000`, a mutation that was a syntax error rather than a behavioural change, and an unguarded `curl` reporting `HTTP ` with an empty status. None of that verification survives the pull request it was written in. This is the repository whose `v3` tag every other repository consumes, and a change here reaches all of them unexecuted — the README says as much. Hand verification that is discarded is the weakest possible answer to that. ### Scope - In: a harness that runs an action's `run:` bodies against stubbed commands and asserts on output and exit status, plus the existing verifications for `release`, `report-job-failure` and `deploy-vps` written into it. - Out: executing actions against a real host, registry or Forgejo — that is what the consuming repositories' own pipelines are for. - Out: changing any action's behaviour. ### Acceptance criteria - [ ] Given an action with a `run:` body, when the harness runs, then that body can be executed with named commands stubbed and its stdout, stderr and exit status asserted. - [ ] Given the guard in `deploy-vps` that refuses an undeclared profile, when the harness runs, then all four of its states are covered. - [ ] Given a case is written for behaviour an action does not have, when the harness runs, then it fails — a case that passes against both the fixed and broken versions is worse than none. - [ ] Given the PR pipeline, when the harness is added, then it runs there alongside `check-actions.py`. ### Notes The third criterion is the one that matters and the one most likely to be quietly skipped. Every verification listed above was written by first watching the case *fail* against the unfixed code; a harness that only ever ran against working code would have recorded four passing tables and caught none of the four defects. Extraction is the fiddly part: a `run:` body inside a heredoc keeps the heredoc's indentation, so `textwrap.dedent` on the YAML-parsed string is not always enough — the `deploy-vps` guard had to be pulled from the raw file by line range. Worth solving once in the harness rather than in each verification. `check-actions.py` already substitutes `${{ }}` and knows how to find every action; the harness should reuse that rather than reimplement it.

ahmad added the

labels

2026-09-05 07:37:04 +00:00

ahmad referenced this issue

2026-09-05 07:52:40 +00:00

fix: Refuse a profile the compose file does not declare #47

ahmad commented

2026-09-07 18:46:19 +00:00

Author

Owner

Escalating rather than implementing: measured honestly, this lands well over the loop's ~5-file/~150-line ceiling, so per the ground rules it stops here — with the shape written down so the next start is immediate.

Measured scope. A harness module (extraction reusing check-actions.py's ${{ }} substitution and action discovery, stub-command PATH injection, stdout/stderr/exit assertions — ~100 lines), the three existing verifications rewritten into it (release retry loop against stub curl 500/000/409/201; report-job-failure's four log states and three cannot-post paths; deploy-vps's four profile-guard states — ~60-100 lines each), a meta-test proving criterion 3 the way test-check-actions.py does (drive a case asserting behaviour a body does not have, assert the harness fails — ~30 lines), and the pr.yml step. Total ~350-450 lines across ~6 files.

Proposed layout, matching the repo's no-framework style: .forgejo/scripts/behaviour_harness.py (library), .forgejo/scripts/behaviour-tests.py (the cases, one function per action), .forgejo/scripts/test-behaviour-harness.py (the criterion-3 meta-test, run first in CI like the checker's own test). Each case declares: action, step name, env, stub scripts, expected stdout/stderr patterns and exit status. Kill-vs-guard is recorded per case (the knowledge file's discipline): a kill case is documented with the mutation it was watched failing against.

Two ways to proceed — your call:
(a) Authorize the loop to implement despite the ceiling — cleanest as two PRs: harness + meta-test + deploy-vps suite + CI wiring (Refs #48), then the release and report-job-failure suites (Closes #48); each is independently green and reviewable.
(b) Take it yourself from this shape.

Leaving it state/ready and unassigned; nothing regresses while it waits — hand verification continues as now.

Escalating rather than implementing: measured honestly, this lands well over the loop's ~5-file/~150-line ceiling, so per the ground rules it stops here — with the shape written down so the next start is immediate. **Measured scope.** A harness module (extraction reusing `check-actions.py`'s `${{ }}` substitution and action discovery, stub-command PATH injection, stdout/stderr/exit assertions — ~100 lines), the three existing verifications rewritten into it (release retry loop against stub curl 500/000/409/201; report-job-failure's four log states and three cannot-post paths; deploy-vps's four profile-guard states — ~60-100 lines each), a meta-test proving criterion 3 the way `test-check-actions.py` does (drive a case asserting behaviour a body does not have, assert the harness fails — ~30 lines), and the `pr.yml` step. Total ~350-450 lines across ~6 files. **Proposed layout**, matching the repo's no-framework style: `.forgejo/scripts/behaviour_harness.py` (library), `.forgejo/scripts/behaviour-tests.py` (the cases, one function per action), `.forgejo/scripts/test-behaviour-harness.py` (the criterion-3 meta-test, run first in CI like the checker's own test). Each case declares: action, step name, env, stub scripts, expected stdout/stderr patterns and exit status. Kill-vs-guard is recorded per case (the knowledge file's discipline): a kill case is documented with the mutation it was watched failing against. **Two ways to proceed — your call:** (a) Authorize the loop to implement despite the ceiling — cleanest as two PRs: harness + meta-test + deploy-vps suite + CI wiring (`Refs #48`), then the release and report-job-failure suites (`Closes #48`); each is independently green and reviewable. (b) Take it yourself from this shape. Leaving it `state/ready` and unassigned; nothing regresses while it waits — hand verification continues as now.

ahmad commented

2026-09-08 09:32:55 +00:00

Author

Owner

Human decision recorded (2026-09-08): option (a) — the loop implements the two-PR sequence from the design comment (harness + meta-test + deploy-vps suite + CI wiring, then the remaining suites).

ahmad self-assigned this

2026-09-08 11:43:31 +00:00

ahmad added

state/in-progress

and removed

state/ready

labels

2026-09-08 11:43:32 +00:00

ahmad commented

2026-09-08 11:43:33 +00:00

Author

Owner

Starting under the recorded authorization, as PR 1 of the design's two-PR sequence (Refs #48): behaviour_harness.py (run-body extraction reusing check-actions.py's ${{ }} substitution, stub commands via PATH injection, stdout/stderr/exit assertions, kill-vs-guard recorded per case), test-behaviour-harness.py (criterion 3 the way test-check-actions.py does it: a case asserting behaviour a body does not have must fail), the deploy-vps profile-guard suite (all four states), and the pr.yml step running meta-test → suites. PR 2 adds the release retry-loop and report-job-failure suites and closes the issue.

Starting under the recorded authorization, as PR 1 of the design's two-PR sequence (`Refs #48`): `behaviour_harness.py` (run-body extraction reusing check-actions.py's `${{ }}` substitution, stub commands via PATH injection, stdout/stderr/exit assertions, kill-vs-guard recorded per case), `test-behaviour-harness.py` (criterion 3 the way test-check-actions.py does it: a case asserting behaviour a body does not have must fail), the deploy-vps profile-guard suite (all four states), and the pr.yml step running meta-test → suites. PR 2 adds the release retry-loop and report-job-failure suites and closes the issue.

ahmad referenced this issue from a commit

2026-09-08 11:46:22 +00:00

test: Run action behaviour against stubbed commands

ahmad referenced this issue

2026-09-08 11:46:44 +00:00

test: Run action behaviour against stubbed commands #62

ahmad added

state/ready

and removed

state/in-progress

labels

2026-09-08 16:13:29 +00:00

ahmad removed their assignment

2026-09-08 16:14:00 +00:00

ahmad commented

2026-09-08 16:14:00 +00:00

Author

Owner

PR 1 (#62) is merged: the harness, its meta-test, and the deploy-vps profile-guard suite — six cases, five of them kill, each watched failing against the mutant it was written for. PR 2 (the release retry-loop and report-job-failure suites) is still to come, so this stays open and state/ready.

PR 1 (#62) is merged: the harness, its meta-test, and the deploy-vps profile-guard suite — six cases, five of them `kill`, each watched failing against the mutant it was written for. PR 2 (the release retry-loop and report-job-failure suites) is still to come, so this stays open and `state/ready`.

ahmad self-assigned this

2026-09-08 19:13:38 +00:00

ahmad added

state/in-progress

and removed

state/ready

labels

2026-09-08 19:13:39 +00:00

ahmad referenced this issue from a pull request that will close it,

2026-09-08 19:23:04 +00:00

test: Cover the release retry loop and the failure reporter #63

ahmad added