chore: Carry the non-blocking notify fetch to the eight consumers #68

Closed
opened 2026-09-09 04:53:57 +00:00 by ahmad · 6 comments
Owner

Context

#65 made openbao-fetch able to tolerate an unreachable vault (required: false) and set it on templates/release.yml's notify job. The template is copied into consumers rather than called — no repository references a reusable workflow — so the tolerance reaches a repository only when its own notify job gains the line.

Until then, an OpenBao blip still turns a build that was fine into a red run in every repository below. This is not a regression from #65; it is the failure #65 describes, still unfixed for existing consumers.

Surveyed 2026-09-09 from the checked-out trees. Every job here fetches from OpenBao purely to send a notification:

Repository Workflows with a fetching notify job
ahmad/portfolio release.yml, pr.yml, deploy.yml, monitoring.yml
ahmad/trip release.yml, deploy.yml
ahmad/imamah release.yml, deploy.yml
bayan/kidpoints release.yml, deploy.yml, monitoring.yml
bayan/stem release.yml
suhba/khitta release.yml, deploy.yml, monitoring.yml
suhba/suhba-docs release.yml, deploy.yml
suhba/suhba-web release.yml, deploy.yml

ahmad/expiro and ahmad/larda have no fetching notify job and need nothing.

Scope

  • In: adding required: false to the openbao-fetch step of every notify job listed above, and making the load step beside it tolerate an empty or absent file the way the template now does; one pull request per repository.
  • Out: the deploy, publish, version and dashboard jobs — those genuinely need their secrets and must keep the fatal default (#65 surveyed them: most have no guard after the fetch, so a tolerant fetch there would start containers with no DATABASE_URL and report success); changing which events notify; the message format.

Acceptance criteria

  • Given OpenBao is unreachable, when a release runs in any repository listed above, then the run's conclusion is unchanged and the job log carries a warning naming the reason.
  • Given the same, when the run is inspected, then no notification was sent and nothing claims one was.
  • Given a deploy, publish, version or dashboard job in those repositories, when its fetch fails, then it still fails the job — the flag is not spread beyond notify.
  • Given every repository in the table, when its workflows are grepped, then each notify job's openbao-fetch step carries required: false.
  • Given a repository whose release pipeline is exercised after the change, then a normal run still notifies exactly as before.
### Context `#65` made `openbao-fetch` able to tolerate an unreachable vault (`required: false`) and set it on `templates/release.yml`'s `notify` job. The template is **copied** into consumers rather than called — no repository references a reusable workflow — so the tolerance reaches a repository only when its own `notify` job gains the line. Until then, an OpenBao blip still turns a build that was fine into a red run in every repository below. This is not a regression from `#65`; it is the failure `#65` describes, still unfixed for existing consumers. Surveyed 2026-09-09 from the checked-out trees. Every job here fetches from OpenBao purely to send a notification: | Repository | Workflows with a fetching `notify` job | | --- | --- | | `ahmad/portfolio` | `release.yml`, `pr.yml`, `deploy.yml`, `monitoring.yml` | | `ahmad/trip` | `release.yml`, `deploy.yml` | | `ahmad/imamah` | `release.yml`, `deploy.yml` | | `bayan/kidpoints` | `release.yml`, `deploy.yml`, `monitoring.yml` | | `bayan/stem` | `release.yml` | | `suhba/khitta` | `release.yml`, `deploy.yml`, `monitoring.yml` | | `suhba/suhba-docs` | `release.yml`, `deploy.yml` | | `suhba/suhba-web` | `release.yml`, `deploy.yml` | `ahmad/expiro` and `ahmad/larda` have no fetching `notify` job and need nothing. ### Scope - In: adding `required: false` to the `openbao-fetch` step of every `notify` job listed above, and making the load step beside it tolerate an empty or absent file the way the template now does; one pull request per repository. - Out: the `deploy`, `publish`, `version` and `dashboard` jobs — those genuinely need their secrets and **must** keep the fatal default (`#65` surveyed them: most have no guard after the fetch, so a tolerant fetch there would start containers with no `DATABASE_URL` and report success); changing which events notify; the message format. ### Acceptance criteria - [ ] Given OpenBao is unreachable, when a release runs in any repository listed above, then the run's conclusion is unchanged and the job log carries a warning naming the reason. - [ ] Given the same, when the run is inspected, then no notification was sent and nothing claims one was. - [ ] Given a `deploy`, `publish`, `version` or `dashboard` job in those repositories, when its fetch fails, then it still fails the job — the flag is not spread beyond `notify`. - [ ] Given every repository in the table, when its workflows are grepped, then each `notify` job's `openbao-fetch` step carries `required: false`. - [ ] Given a repository whose release pipeline is exercised after the change, then a normal run still notifies exactly as before.
Author
Owner

Correcting the table in the body: it is short, and short in a way that matters. Review of #65 pointed out that my survey walked local checkouts rather than the fleet, so it could only see repositories I happen to have cloned and up to date. Re-run against the API — every repository the server lists, every file under .forgejo/workflows/, parsed rather than grepped:

12 repositories use openbao-fetch, across 35 workflow files. My original list had 9 and the review's had 11; Bayan/bayan-web is in neither.

Classifying every call by whether the job's only use of the secrets is sending a notification:

Jobs to change — 26, across 10 repositories

Repository notify jobs
ahmad/portfolio release.yml, deploy.yml, pr.yml, monitoring.yml
Bayan/bayan-web release.yml, deploy.yml, pr.yml, monitoring.yml
Bayan/kidpoints release.yml, deploy.yml, monitoring.yml
ahmad/namey release.yml, deploy.yml, pr.yml
suhba/khitta release.yml, deploy.yml, monitoring.yml
ahmad/trip release.yml, deploy.yml
ahmad/imamah release.yml, deploy.yml
suhba/suhba-docs release.yml, deploy.yml
suhba/suhba-web release.yml, deploy.yml
Bayan/stem release.yml

ahmad/expiro and ahmad/larda have no notification-only job and need nothing.

pr.yml and monitoring.yml matter as much as release.yml, which the original scoping missed: a pull-request check or a monitoring sync going red because the vault blinked is the same failure as a release doing it, and arguably worse — it lands on somebody's PR.

Jobs that must not change — 57

Every deploy, publish, version, dashboard, sync, validate and open-pr job in those repositories genuinely reads the secrets. #65 surveyed what follows the fetch in each: most have no guard, so a tolerant fetch there would start containers with no DATABASE_URL and report success. The flag goes on notify and nowhere else, and criterion 3 below is what proves it did not spread.

The method, since the first table was wrong

Enumerate repositories from GET /orgs/{org}/repos and GET /users/{u}/repos, list .forgejo/workflows/ per repository through the contents API, and attribute each openbao-fetch@ line to its job by walking back to the enclosing ^ <job>:. A claim of the form "every consumer does X" is only as wide as the enumeration behind it, and a local checkout is not that enumeration.

**Correcting the table in the body: it is short, and short in a way that matters.** Review of `#65` pointed out that my survey walked local checkouts rather than the fleet, so it could only see repositories I happen to have cloned and up to date. Re-run against the API — every repository the server lists, every file under `.forgejo/workflows/`, parsed rather than grepped: **12 repositories use `openbao-fetch`, across 35 workflow files.** My original list had 9 and the review's had 11; `Bayan/bayan-web` is in neither. Classifying every call by whether the job's *only* use of the secrets is sending a notification: ### Jobs to change — 26, across 10 repositories | Repository | `notify` jobs | | --- | --- | | `ahmad/portfolio` | `release.yml`, `deploy.yml`, `pr.yml`, `monitoring.yml` | | `Bayan/bayan-web` | `release.yml`, `deploy.yml`, `pr.yml`, `monitoring.yml` | | `Bayan/kidpoints` | `release.yml`, `deploy.yml`, `monitoring.yml` | | `ahmad/namey` | `release.yml`, `deploy.yml`, `pr.yml` | | `suhba/khitta` | `release.yml`, `deploy.yml`, `monitoring.yml` | | `ahmad/trip` | `release.yml`, `deploy.yml` | | `ahmad/imamah` | `release.yml`, `deploy.yml` | | `suhba/suhba-docs` | `release.yml`, `deploy.yml` | | `suhba/suhba-web` | `release.yml`, `deploy.yml` | | `Bayan/stem` | `release.yml` | `ahmad/expiro` and `ahmad/larda` have no notification-only job and need nothing. **`pr.yml` and `monitoring.yml` matter as much as `release.yml`**, which the original scoping missed: a pull-request check or a monitoring sync going red because the vault blinked is the same failure as a release doing it, and arguably worse — it lands on somebody's PR. ### Jobs that must **not** change — 57 Every `deploy`, `publish`, `version`, `dashboard`, `sync`, `validate` and `open-pr` job in those repositories genuinely reads the secrets. `#65` surveyed what follows the fetch in each: most have no guard, so a tolerant fetch there would start containers with no `DATABASE_URL` and report success. The flag goes on `notify` and nowhere else, and criterion 3 below is what proves it did not spread. ### The method, since the first table was wrong Enumerate repositories from `GET /orgs/{org}/repos` and `GET /users/{u}/repos`, list `.forgejo/workflows/` per repository through the contents API, and attribute each `openbao-fetch@` line to its job by walking back to the enclosing `^ <job>:`. A claim of the form "every consumer does X" is only as wide as the enumeration behind it, and a local checkout is not that enumeration.
Author
Owner

Starting this, one repository at a time, and beginning with ahmad/portfolio — it has four affected jobs, more than any other, and its pr.yml case is the one that lands somewhere a person is already looking.

Scope of this first pull request: portfolio's notify jobs in release.yml, deploy.yml, pr.yml and monitoring.yml gain required: false on their openbao-fetch step, and the load step beside each tolerates an absent or empty file the way the template now does. Nothing else in those workflows changes, and no deploy, publish, version or sync job is touched — those genuinely need their secrets and the table above says so.

Ordering. #67 is merged, so openbao-fetch@v3 already carries the input; a consumer setting required: false today gets the behaviour immediately rather than waiting on anything.

What I will check before opening it, since "the flag is on the right jobs" is the whole of this change: every openbao-fetch step in each edited file, listed with its enclosing job and whether it now carries the flag — so the diff can be read against the table rather than trusted.

The remaining nine repositories follow in later iterations, one pull request each as the scope says. I will keep the list here updated as they land, so what is left is visible without reading ten PRs.

Starting this, one repository at a time, and beginning with `ahmad/portfolio` — it has **four** affected jobs, more than any other, and its `pr.yml` case is the one that lands somewhere a person is already looking. **Scope of this first pull request:** `portfolio`'s `notify` jobs in `release.yml`, `deploy.yml`, `pr.yml` and `monitoring.yml` gain `required: false` on their `openbao-fetch` step, and the load step beside each tolerates an absent or empty file the way the template now does. Nothing else in those workflows changes, and no `deploy`, `publish`, `version` or `sync` job is touched — those genuinely need their secrets and the table above says so. **Ordering.** `#67` is merged, so `openbao-fetch@v3` already carries the input; a consumer setting `required: false` today gets the behaviour immediately rather than waiting on anything. **What I will check before opening it**, since "the flag is on the right jobs" is the whole of this change: every `openbao-fetch` step in each edited file, listed with its enclosing job and whether it now carries the flag — so the diff can be read against the table rather than trusted. The remaining nine repositories follow in later iterations, one pull request each as the scope says. I will keep the list here updated as they land, so what is left is visible without reading ten PRs.
ahmad self-assigned this 2026-09-09 07:44:55 +00:00
Author
Owner

1 of 10 open. ahmad/portfolio → #86, green and ready. Four notify jobs (release, deploy, pr, monitoring); its five secret-reading jobs untouched, audited job by job in the PR body.

Worth noting from doing the first one: PR / notify on that repository's own pull-request pipeline ran green against this change, which is the fetch path in question exercised for real rather than reasoned about. It does not prove the tolerant branch — the vault answered — but it does prove the flag does not break the ordinary case, which is the half a reviewer cannot check by reading.

Remaining, in the order I intend to take them (most affected first, so the biggest reductions land soonest):

# Repository Jobs
✅ ahmad/portfolio 4
2 Bayan/bayan-web 4
3 Bayan/kidpoints 3
4 ahmad/namey 3
5 suhba/khitta 3
6 ahmad/trip 2
7 ahmad/imamah 2
8 suhba/suhba-docs 2
9 suhba/suhba-web 2
10 Bayan/stem 1
**1 of 10 open.** `ahmad/portfolio` → [`#86`](https://git.ahmadelmasri.com/ahmad/portfolio/pulls/86), green and ready. Four `notify` jobs (`release`, `deploy`, `pr`, `monitoring`); its five secret-reading jobs untouched, audited job by job in the PR body. Worth noting from doing the first one: **`PR / notify` on that repository's own pull-request pipeline ran green against this change**, which is the fetch path in question exercised for real rather than reasoned about. It does not prove the tolerant branch — the vault answered — but it does prove the flag does not break the ordinary case, which is the half a reviewer cannot check by reading. Remaining, in the order I intend to take them (most affected first, so the biggest reductions land soonest): | # | Repository | Jobs | | --- | --- | --- | | ✅ | `ahmad/portfolio` | 4 | | 2 | `Bayan/bayan-web` | 4 | | 3 | `Bayan/kidpoints` | 3 | | 4 | `ahmad/namey` | 3 | | 5 | `suhba/khitta` | 3 | | 6 | `ahmad/trip` | 2 | | 7 | `ahmad/imamah` | 2 | | 8 | `suhba/suhba-docs` | 2 | | 9 | `suhba/suhba-web` | 2 | | 10 | `Bayan/stem` | 1 |
Author
Owner

Correction to the survey above. It counted jobs by whether the fetch
declared required, and never looked for continue-on-error. Two of the
26 were already tolerant by that other mechanism:

Repo Workflow Job Already guarded
Bayan/bayan-web pr.yml notify continue-on-error: true on the fetch and the load step
ahmad/namey pr.yml notify same

So the fleet total is 24 jobs that fail on a vault outage, not 26.
The other eight repositories and every non-pr.yml workflow have no
guard of any kind — re-checked against the API, not local checkouts.

That both instances are pr.yml and both are the same shape suggests
someone hit this on a PR run and patched it where it hurt, without
carrying it to deploy, release or monitoring. It is the same bug
this issue is about, found earlier and fixed locally.

I am converting those two to required: false in the same PRs rather
than leaving them, so all ten repositories use one mechanism. It is a
strictly narrower guard: continue-on-error would swallow a genuine
defect in the action as readily as an outage. Both PRs call this out
separately so it can be refused without refusing the rest.

Related: removing continue-on-error from the load step is only
safe because that step survives an empty vault on its own. It runs under
set -euo pipefail and its middle line is
[ -n "$v" ] && echo "::add-mask::$v", which returns non-zero when the
value is empty. Exercised against a missing .env, an empty one and a
populated one, it exits 0 in all three — a short-circuited && list is
not a set -e failure, so the value simply lands empty and
notify-telegram skips.

Progress: 2 of 10 open.

Repo PR State
ahmad/portfolio #86 green, ready
Bayan/bayan-web Bayan/bayan-web#32 opened, CI running

Remaining, in order: Bayan/kidpoints, ahmad/namey, suhba/khitta,
ahmad/trip, ahmad/imamah, suhba/suhba-docs, suhba/suhba-web,
Bayan/stem.

**Correction to the survey above.** It counted jobs by whether the fetch declared `required`, and never looked for `continue-on-error`. Two of the 26 were already tolerant by that other mechanism: | Repo | Workflow | Job | Already guarded | |---|---|---|---| | `Bayan/bayan-web` | `pr.yml` | `notify` | `continue-on-error: true` on the fetch **and** the load step | | `ahmad/namey` | `pr.yml` | `notify` | same | So the fleet total is **24 jobs that fail on a vault outage**, not 26. The other eight repositories and every non-`pr.yml` workflow have no guard of any kind — re-checked against the API, not local checkouts. That both instances are `pr.yml` and both are the same shape suggests someone hit this on a PR run and patched it where it hurt, without carrying it to `deploy`, `release` or `monitoring`. It is the same bug this issue is about, found earlier and fixed locally. I am converting those two to `required: false` in the same PRs rather than leaving them, so all ten repositories use one mechanism. It is a strictly narrower guard: `continue-on-error` would swallow a genuine defect in the action as readily as an outage. Both PRs call this out separately so it can be refused without refusing the rest. Related: removing `continue-on-error` from the **load** step is only safe because that step survives an empty vault on its own. It runs under `set -euo pipefail` and its middle line is `[ -n "$v" ] && echo "::add-mask::$v"`, which returns non-zero when the value is empty. Exercised against a missing `.env`, an empty one and a populated one, it exits 0 in all three — a short-circuited `&&` list is not a `set -e` failure, so the value simply lands empty and `notify-telegram` skips. **Progress: 2 of 10 open.** | Repo | PR | State | |---|---|---| | `ahmad/portfolio` | #86 | green, ready | | `Bayan/bayan-web` | Bayan/bayan-web#32 | opened, CI running | Remaining, in order: `Bayan/kidpoints`, `ahmad/namey`, `suhba/khitta`, `ahmad/trip`, `ahmad/imamah`, `suhba/suhba-docs`, `suhba/suhba-web`, `Bayan/stem`.
Author
Owner

All ten repositories now have a PR. One merged, nine open.

Repo PR Notify jobs State
ahmad/portfolio #86 4 merged 7bf823f
Bayan/bayan-web Bayan/bayan-web#32 4 (1 pre-guarded) green, ready
Bayan/kidpoints Bayan/kidpoints#211 3 green, ready
ahmad/namey ahmad/namey#76 3 (1 pre-guarded) green, ready
suhba/khitta suhba/khitta#56 3 green, ready
ahmad/trip ahmad/trip#138 2 green, ready
suhba/suhba-docs suhba/suhba-docs#14 2 green, ready
ahmad/imamah ahmad/imamah#158 2 CI running
suhba/suhba-web suhba/suhba-web#43 2 CI running
Bayan/stem Bayan/stem#13 1 CI running

That totals 26 notify jobs, of which 2 were already guarded by
continue-on-error — so 24 fetches changed from required to not,
which is the corrected figure from the comment above, arrived at
independently by counting the PRs. Every PR was verified by re-parsing
its workflows and reading required back per job, not by reading the
diff; each carries that audit as a comment.

A gap found while surveying, not folded in. Checking all 25
repositories on the server for workflows that deploy or release without a
notify job turned up exactly one: Bayan/stem's deploy.yml runs a
deploy job and has no notification at all, so deployments there succeed
or fail silently. Its release.yml does notify, so the gap is specific
to the deploy path. Opened as Bayan/stem#14 rather than added to
Bayan/stem#13, which is scoped to the required flag.

Nothing here needs an answer — the one open question is still whether to
keep continue-on-error in the two pr.yml files instead of converting
them, called out on Bayan/bayan-web#32 and ahmad/namey#76 so it can be
refused there without refusing the rest.

**All ten repositories now have a PR.** One merged, nine open. | Repo | PR | Notify jobs | State | |---|---|---|---| | `ahmad/portfolio` | #86 | 4 | **merged** `7bf823f` | | `Bayan/bayan-web` | Bayan/bayan-web#32 | 4 (1 pre-guarded) | green, ready | | `Bayan/kidpoints` | Bayan/kidpoints#211 | 3 | green, ready | | `ahmad/namey` | ahmad/namey#76 | 3 (1 pre-guarded) | green, ready | | `suhba/khitta` | suhba/khitta#56 | 3 | green, ready | | `ahmad/trip` | ahmad/trip#138 | 2 | green, ready | | `suhba/suhba-docs` | suhba/suhba-docs#14 | 2 | green, ready | | `ahmad/imamah` | ahmad/imamah#158 | 2 | CI running | | `suhba/suhba-web` | suhba/suhba-web#43 | 2 | CI running | | `Bayan/stem` | Bayan/stem#13 | 1 | CI running | That totals **26 notify jobs**, of which 2 were already guarded by `continue-on-error` — so **24 fetches changed from required to not**, which is the corrected figure from the comment above, arrived at independently by counting the PRs. Every PR was verified by re-parsing its workflows and reading `required` back per job, not by reading the diff; each carries that audit as a comment. **A gap found while surveying, not folded in.** Checking all 25 repositories on the server for workflows that deploy or release without a `notify` job turned up exactly one: `Bayan/stem`'s `deploy.yml` runs a deploy job and has no notification at all, so deployments there succeed or fail silently. Its `release.yml` does notify, so the gap is specific to the deploy path. Opened as Bayan/stem#14 rather than added to Bayan/stem#13, which is scoped to the `required` flag. Nothing here needs an answer — the one open question is still whether to keep `continue-on-error` in the two `pr.yml` files instead of converting them, called out on Bayan/bayan-web#32 and ahmad/namey#76 so it can be refused there without refusing the rest.
Author
Owner

Done. All ten repositories merged.

Repository PR Merge
ahmad/portfolio #86 7bf823f
Bayan/bayan-web Bayan/bayan-web#32 0b91a05
Bayan/kidpoints Bayan/kidpoints#211 1c97d58
ahmad/namey ahmad/namey#76 383d3f1
suhba/khitta suhba/khitta#56 1b18a15
ahmad/trip ahmad/trip#138 b99586e
ahmad/imamah ahmad/imamah#158 2f496fa
suhba/suhba-docs suhba/suhba-docs#14 45a62e6
suhba/suhba-web suhba/suhba-web#43 a79f82a
Bayan/stem Bayan/stem#13 d574bee

24 notify fetches now declare required: false, out of 26 notify jobs
— the other two were already tolerant via continue-on-error and were
converted to the same mechanism, so no notify job on the fleet fails a
run when OpenBao is unreachable.

Two corrections were made along the way, both recorded above rather
than quietly fixed: the original survey counted 26 because it looked for
the absence of required and never for continue-on-error; and the
continue-on-error on two pr.yml load steps turned out to be
guarding a failure that cannot happen, since a short-circuited && list
does not trip set -e.

What is still unobserved. No repository has yet had a release or
deploy run while OpenBao was actually unreachable, so the behaviour is
verified against the action's contract and a local exercise of the load
step, not against a real outage. The next OpenBao blip is the test; if a
notify job fails then, this issue is where to come back to.

Two follow-ups came out of the pass and are filed separately:
Bayan/stem#14 (merged as Bayan/stem#15 — the one deploy workflow on the
server that reported nothing) and #72 (six repositories announce a
skipped deploy as a failure).

**Done. All ten repositories merged.** | Repository | PR | Merge | |---|---|---| | `ahmad/portfolio` | #86 | `7bf823f` | | `Bayan/bayan-web` | Bayan/bayan-web#32 | `0b91a05` | | `Bayan/kidpoints` | Bayan/kidpoints#211 | `1c97d58` | | `ahmad/namey` | ahmad/namey#76 | `383d3f1` | | `suhba/khitta` | suhba/khitta#56 | `1b18a15` | | `ahmad/trip` | ahmad/trip#138 | `b99586e` | | `ahmad/imamah` | ahmad/imamah#158 | `2f496fa` | | `suhba/suhba-docs` | suhba/suhba-docs#14 | `45a62e6` | | `suhba/suhba-web` | suhba/suhba-web#43 | `a79f82a` | | `Bayan/stem` | Bayan/stem#13 | `d574bee` | **24 notify fetches** now declare `required: false`, out of 26 notify jobs — the other two were already tolerant via `continue-on-error` and were converted to the same mechanism, so no notify job on the fleet fails a run when OpenBao is unreachable. **Two corrections were made along the way**, both recorded above rather than quietly fixed: the original survey counted 26 because it looked for the absence of `required` and never for `continue-on-error`; and the `continue-on-error` on two `pr.yml` **load** steps turned out to be guarding a failure that cannot happen, since a short-circuited `&&` list does not trip `set -e`. **What is still unobserved.** No repository has yet had a release or deploy run while OpenBao was actually unreachable, so the behaviour is verified against the action's contract and a local exercise of the load step, not against a real outage. The next OpenBao blip is the test; if a notify job fails then, this issue is where to come back to. Two follow-ups came out of the pass and are filed separately: Bayan/stem#14 (merged as Bayan/stem#15 — the one deploy workflow on the server that reported nothing) and #72 (six repositories announce a skipped deploy as a failure).
ahmad 2026-09-09 09:49:41 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
amtronics/platform-actions#68
No description provided.