test: No action's behaviour is covered by anything that runs again #48

New issue

Open

opened 2026-09-05 07:37:04 +00:00 by ahmad · 0 comments

ahmad commented

2026-09-05 07:37:04 +00:00

Owner

Context

check-actions.py parses every action.yml, checks the fields a composite action needs, and runs bash -n over each run: body with ${{ }} substituted. It cannot execute one, so no action's behaviour is tested by anything that runs again.

Every recent change here has been verified by hand, by extracting a run: body into a file, stubbing the commands it calls, and driving it through its cases:

release@v3        the retry loop, against a stub curl returning 500 / 000 / 409 / 201
report-job-failure  four log states and three "cannot post" paths, against a local HTTP server
deploy-vps        the profile guard, against a stub `docker compose config --profiles`

Each of those found something a reading would not have: a || echo 000 that concatenated with curl's own 000 to give 000000, a mutation that was a syntax error rather than a behavioural change, and an unguarded curl reporting HTTP with an empty status. None of that verification survives the pull request it was written in.

This is the repository whose v3 tag every other repository consumes, and a change here reaches all of them unexecuted — the README says as much. Hand verification that is discarded is the weakest possible answer to that.

Scope

In: a harness that runs an action's run: bodies against stubbed commands and asserts on output and exit status, plus the existing verifications for release, report-job-failure and deploy-vps written into it.
Out: executing actions against a real host, registry or Forgejo — that is what the consuming repositories' own pipelines are for.
Out: changing any action's behaviour.

Acceptance criteria

Given an action with a run: body, when the harness runs, then that body can be executed with named commands stubbed and its stdout, stderr and exit status asserted.
Given the guard in deploy-vps that refuses an undeclared profile, when the harness runs, then all four of its states are covered.
Given a case is written for behaviour an action does not have, when the harness runs, then it fails — a case that passes against both the fixed and broken versions is worse than none.
Given the PR pipeline, when the harness is added, then it runs there alongside check-actions.py.

Notes

The third criterion is the one that matters and the one most likely to be quietly skipped. Every verification listed above was written by first watching the case fail against the unfixed code; a harness that only ever ran against working code would have recorded four passing tables and caught none of the four defects.

Extraction is the fiddly part: a run: body inside a heredoc keeps the heredoc's indentation, so textwrap.dedent on the YAML-parsed string is not always enough — the deploy-vps guard had to be pulled from the raw file by line range. Worth solving once in the harness rather than in each verification.

check-actions.py already substitutes ${{ }} and knows how to find every action; the harness should reuse that rather than reimplement it.

### Context `check-actions.py` parses every `action.yml`, checks the fields a composite action needs, and runs `bash -n` over each `run:` body with `${{ }}` substituted. It cannot execute one, so **no action's behaviour is tested by anything that runs again**. Every recent change here has been verified by hand, by extracting a `run:` body into a file, stubbing the commands it calls, and driving it through its cases: ``` release@v3 the retry loop, against a stub curl returning 500 / 000 / 409 / 201 report-job-failure four log states and three "cannot post" paths, against a local HTTP server deploy-vps the profile guard, against a stub `docker compose config --profiles` ``` Each of those found something a reading would not have: a `|| echo 000` that concatenated with curl's own `000` to give `000000`, a mutation that was a syntax error rather than a behavioural change, and an unguarded `curl` reporting `HTTP ` with an empty status. None of that verification survives the pull request it was written in. This is the repository whose `v3` tag every other repository consumes, and a change here reaches all of them unexecuted — the README says as much. Hand verification that is discarded is the weakest possible answer to that. ### Scope - In: a harness that runs an action's `run:` bodies against stubbed commands and asserts on output and exit status, plus the existing verifications for `release`, `report-job-failure` and `deploy-vps` written into it. - Out: executing actions against a real host, registry or Forgejo — that is what the consuming repositories' own pipelines are for. - Out: changing any action's behaviour. ### Acceptance criteria - [ ] Given an action with a `run:` body, when the harness runs, then that body can be executed with named commands stubbed and its stdout, stderr and exit status asserted. - [ ] Given the guard in `deploy-vps` that refuses an undeclared profile, when the harness runs, then all four of its states are covered. - [ ] Given a case is written for behaviour an action does not have, when the harness runs, then it fails — a case that passes against both the fixed and broken versions is worse than none. - [ ] Given the PR pipeline, when the harness is added, then it runs there alongside `check-actions.py`. ### Notes The third criterion is the one that matters and the one most likely to be quietly skipped. Every verification listed above was written by first watching the case *fail* against the unfixed code; a harness that only ever ran against working code would have recorded four passing tables and caught none of the four defects. Extraction is the fiddly part: a `run:` body inside a heredoc keeps the heredoc's indentation, so `textwrap.dedent` on the YAML-parsed string is not always enough — the `deploy-vps` guard had to be pulled from the raw file by line range. Worth solving once in the harness rather than in each verification. `check-actions.py` already substitutes `${{ }}` and knows how to find every action; the harness should reuse that rather than reimplement it.

ahmad added the

labels

2026-09-05 07:37:04 +00:00

ahmad referenced this issue

2026-09-05 07:52:40 +00:00

fix: Refuse a profile the compose file does not declare #47