fix: Gate the release on the checks that gate a pull request #85
No reviewers
Labels
No labels
priority/P0
priority/P1
priority/P2
priority/P3
size/L
size/M
size/S
size/XL
size/XS
state/blocked
state/done
state/in-progress
state/in-review
state/needs-refinement
state/ready
type/bug
type/chore
type/feature
type/refactor
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
amtronics/platform-actions!85
Loading…
Reference in a new issue
No description provided.
Delete branch "feature/84-release-checks"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Issue
Closes #84
Problem
A release published and deployed without running any of the checks that gate a
pull request.
ahmad/lardashipped v0.19.0 and v0.20.0 over amainwhosepnpm typecheckwas failing, with every job green.Solution
A
checksjob running the same commands the repository's PR workflow runs, whichthe rest of the flow needs.
It gates the whole flow rather than sitting between
versionandpublish.That is the design decision worth reviewing:
versiontags and publishes theRelease, so a check failing after it would leave a tag and release notes for a
version that never shipped — and consume that version number for the next one.
Gating first means a red tree produces nothing at all.
notifyneedscheckstoo. Without that a blocked release either reports ageneric failure or, since
notifyisif: always(), reports on three jobs thatnever ran. Its
detailsline names the case: "the checks on main did not pass(failure) — nothing was tagged, published or deployed", because that outcome
wants different people looking than a deploy falling over.
Review notes
The
checkssteps in the template are the common pnpm shape and are meant to bereplaced with whatever the consumer's PR workflow runs — the README says so, and
so does the comment above the job. Keeping the two lists identical is the
point; a check that gates a pull request and not a release is a check with a
hole in it.
The status expression was evaluated against all nine combinations that can reach
notify— table in the comment below — including the two new ones,checksfailed and
checkscancelled.check-actions.pynow validates 13run:steps in this template, up from 7, sothe new job's shell is covered by the existing gate.
The cost, measured
Every merge to
mainnow runs the check suite, including merges that warrant norelease. Measured from the median of each repository's recent check-job runs:
suhba/khittaBayan/kidpointsBayan/stemahmad/launchpadsuhba/suhba-web,suhba/suhba-docsBayan/bayan-webahmad/portfolioahmad/imamahahmad/expiroahmad/nameyahmad/lardaMedian 29 seconds, worst case 96. Nothing here is slow enough to be a reason not
to, which is the answer criterion 4 asked for.
Follow-ups
All thirteen consumers have one, each carrying its own measured cost, filed
state/blockedon this:ahmad/trip#148,ahmad/portfolio#94,ahmad/expiro#90,ahmad/larda#79,ahmad/namey#87,ahmad/imamah#165,ahmad/launchpad#91,Bayan/stem#22,Bayan/bayan-web#37,Bayan/kidpoints#216,suhba/suhba-web#47,suhba/suhba-docs#20,suhba/khitta#60.Risks and trade-offs
The template only; no action changes, so no consumer's behaviour moves until its
own follow-up lands.
All five
validatesteps pass locally by exit code.WIP: fix: Gate the release on the checks that gate a pull requestto fix: Gate the release on the checks that gate a pull requestSelf-verified on
2525c81;PR / validategreen.mainwhose checks fail publishes and deploys nothing. Everything needschecks:versiongainsneeds: [checks], andpublishanddeployalready chain fromversion. A failed gate means no tag either, which is stronger than the criterion asked and is the reason for gating first rather than betweenversionandpublish—versionpublishes the Release, so a later failure would leave notes for a version that never shipped and burn the number.mainwhose checks pass releases as before. The only change downstream ofchecksisversion'sneeds.notifyneedschecks, and its details line reads "the checks on main did not pass (failure) — nothing was tagged, published or deployed".suhba/khitta9s at the fastest andahmad/larda96s at the slowest, taken from the median of each repository's recent check-job runs rather than estimated.state/blockedon this and each carrying its own measured cost.The status expression evaluated against all nine reachable combinations, including the two new ones:
successsuccessskippedfailurecancelledfailurefailurefailurecancelledGates by exit code:
test-check-actions.py0,check-actions.py0,test-behaviour-harness.py0,behaviour-tests.py0, README lint 0. Worth notingcheck-actions.pynow validates 13run:steps in this template where it validated 7 — the new job's shell is covered by the gate that already existed.One thing I want a second opinion on: the
checkssteps I wrote are the common pnpm shape, and every consumer will replace them. An alternative would be leaving them as<CHECKS>placeholders like<OWNER>/<APP>. I chose runnable-but-wrong over obviously-blank because it makes the shape clear — install, then the four gates — and because the comment and the README both say to replace them. If you would rather it be a placeholder, that is a one-line change and I have no strong view.Requesting changes on one finding, with a second worth fixing in the same edit. The design is right and the status expression is sound — both were checked rather than taken from the description.
What I verified holds.
The status expression: I enumerated it under expression semantics (
a && byieldsawhen falsy elseb;a || byieldsawhen truthy elseb) over all 512 combinations of the four job results andreleased. Every one returns one of the four status words, and no combination with a non-successcheckscan returnsuccess— the gate cannot be reported green. The nine reachable rows in your comment all match; I evaluated a tenth,checkssuccess withversioncancelled, which returnscancelledcorrectly.Gating the whole flow rather than sitting between
versionandpublishis the right call for the reason you give —versiontags and publishes the Release, so a later failure would burn the version number — andnotifyneedingchecksis what makes the blocked case reportable at all.The finding.
PNPM_VERSIONis undefined in this template, so the new job's corepack line silently activates the newest pnpm instead of the pinned one. Measured in the job's own image: it exits 0 and installs 12.4.0 whereahmad/larda's PR workflow pins 11.17.0. Detail inline.The second point, inline as well: the instruction says to replace the steps, and for larda the gate lives partly in the job's
services:block, a step-levelDATABASE_URLand an explicit anti-skip assertion — without thempnpm testreports green with the store suites skipped, which is the statelarda#49closed.One pre-existing wart in the line you are editing, not a reason to hold the merge and not in scope for #84 — worth an issue rather than a fix here. When
versionfails,releasedis empty, sodetailsfalls to its last branch and the notification reads ❌ Release — no release warranted by the commits. The status is right; the sentence beside it describes a different outcome. Your new first branch is what made me look, and it is the same shape: thecheckscase got named precisely because a generic line wants the wrong people looking.On your open question — runnable-but-wrong over
<CHECKS>placeholders: I would keep what you chose. The shape carries information a placeholder does not, and the two<PLACEHOLDER>conventions already in the file are for values, not blocks. It does argue for thePNPM_VERSIONfix rather than against it: if the block is meant to be read as a working example, the example should work.@ -57,0 +72,4 @@# `main` is now reported when it happens, rather than discovered by whoever# opens the next pull request.## Replace the steps below with exactly what your PR workflow runs. KeepingCopying the step list alone loses the gate for at least one consumer — and it is
ahmad/larda, the repository this issue was opened over.Its
validatejob carries three things that are notrun:lines:services:block runningpostgres:17-alpinewith a health check;env: DATABASE_URLon the test step;Its own comment says why: "The store suites are skipped without DATABASE_URL, so without a database here they skipped on every pull request and a green
validatesaid nothing about the SQL (#49)." And the assertion step is there because "a greenvalidatewith the store suites skipped ... looks identical to a real pass".So a
checksjob assembled from the step list — install, format, lint, typecheck,pnpm test— passes green on larda while the integration suites skip. That is the statelarda#49closed, reintroduced in the job whose whole purpose is to stop a green pipeline from meaning nothing.Fix: say in this comment, and in the README paragraph, that what is copied is the PR workflow's job, not its step list —
services:, job- and step-levelenv:, and any assertion steps come with it. Copying the shape one line lower than the instruction says is exactly the mistake this text is trying to prevent, and thirteen follow-up issues are already pointed at it.@ -57,0 +81,4 @@image: node:22-bookwormsteps:- uses: actions/checkout@v4- run: corepack enable && corepack prepare pnpm@${PNPM_VERSION} --activatePNPM_VERSIONis not defined anywhere in this template, so this line does not pin pnpm — it floats.grep -n PNPM_VERSION templates/release.ymlreturns exactly one hit: this line. The workflow-levelenv:block holdsREGISTRYandIMAGE_NAMEand nothing else, and there is no job- or step-levelenv:onchecks. The step list came from a consumer's PR workflow, but theenv:block that gave the variable a value did not come with it —ahmad/larda'spr.ymlsetsPNPM_VERSION: 11.17.0at workflow level and uses this exact line.It does not fail, which is the problem. Measured in a throwaway
node:22-bookworm, which is the image this job runs in:The empty expansion leaves
pnpm@, corepack reads that aspnpm@*and installs the newest release. So the job that gates the release runs pnpm 12.4.0 where the job that gates the pull request runs 11.17.0 — a major apart — and exits 0 either way. I checked whether it breaks the install outright and it does not: alockfileVersion: '9.0'lockfile written by 11.17.0 installs clean under 12.4.0 with--frozen-lockfile(exit 0), so nothing downstream reports it. It also floats: the next pnpm release changes this gate for every consumer with no commit anywhere.That is the same hole this pull request is closing, one level down — a check that gates a release under a different toolchain than the one that gated the pull request.
Fix: give the template the variable, beside the two that are already there, and list it with the other placeholders in the header:
fix: Gate the release on the checks that gate a pull requestto WIP: fix: Gate the release on the checks that gate a pull requestBoth findings were right and both are fixed in
f8b3152.PNPM_VERSIONfloating. Confirmed before fixing:grep -n PNPM_VERSION templates/release.ymlreturned one hit, the corepack line itself, and theworkflow-level
env:block held onlyREGISTRYandIMAGE_NAME. The variableis now a placeholder beside those two and is listed in the header block with the
others, so it is filled in the same pass as the image name rather than
discovered later. Your measurement is the part that made this worth stopping
for — it exits 0, installs a major version ahead, and the lockfile installs
clean under it, so nothing downstream would ever have reported it.
Copying the steps rather than the job. Also fixed, in both places you named.
The template comment and the README paragraph now say to copy the PR workflow's
validate job —
services:, job- and step-levelenv:and assertion steps comewith it — and both cite
ahmad/larda#49as the concrete case, because "copy thejob" without the reason is the kind of instruction that gets read as pedantry and
shortened back to the step list. The README no longer says "the same commands"
in its opening line either; that phrasing was the same mistake one paragraph up.
On the pre-existing
detailswart —versionfailing falls through to"no release warranted by the commits", which describes a different outcome. I
agree it is real and I agree it is not #84's. Filed as
amtronics/platform-actions#86rather than folded in here.
On the open question, thank you — I have kept the runnable steps.
check-actions.pyexits 0, the template still parses as YAML, and the READMElints clean.
WIP: fix: Gate the release on the checks that gate a pull requestto fix: Gate the release on the checks that gate a pull requestBoth findings are addressed. Re-reviewed the delta only —
f8b3152against2525c81— and the rest of the template is untouched, including the status expression I enumerated last time.The pin.
PNPM_VERSION: <PNPM-VERSION>is in the workflowenv:block and listed with the other placeholders in the header. I checked the case this introduces, which is the one worth checking: a consumer who copies the template and forgets to replace it. It fails loudly rather than trading one silent state for another — measured innode:22-bookworm:Exit 1 on the second step of the gate, before anything is installed. That is the right shape: the unpinned case was dangerous precisely because it exited 0, and an unfilled placeholder now cannot.
Copy the job, not the steps. Both the job comment and the README section now say the whole job —
services:, job- and step-levelenv:, assertion steps — and nameahmad/larda#49as the concrete case. That is what makes the instruction actionable for the thirteen follow-ups; a consumer reading it will look for itsservices:block instead of pastingrun:lines.PR / validateis green on this head.The pre-existing
detailswart I mentioned stands unchanged and is still out of scope for #84 — worth its own issue: whenversionfails,releasedis empty, so a red release reports "no release warranted by the commits".