fix: openbao-fetch breaks when apt transiently fails #57

New issue

Closed

opened 2026-09-07 21:15:37 +00:00 by ahmad · 3 comments

ahmad commented

2026-09-07 21:15:37 +00:00

Owner

Context

openbao-fetch installs its two runtime dependencies (curl, jq) with an in-job apt-get when the job container lacks them. On 2026-09-07 ahmad/trip's main run 2462 went red because Debian's bullseye-security InRelease was momentarily expired ("invalid since 34s"): apt failed, jq stayed absent, the OpenBao fetch exited 127, and the downstream notify-telegram step never ran. A transient mirror or clock hiccup at any consumer thus fails jobs that need no network beyond the vault and Telegram — and it will strike deploy jobs the same way, not just notifications. The action already runs inside node images, where a JSON parser is guaranteed without any package install; report-job-failure shed its jq dependency for the same class of reason (#53).

Scope

In: openbao-fetch works without installing packages at runtime, or its dependency install tolerates transient apt failures; existing consumers unchanged.
Out: retrying the vault or Telegram calls themselves; other actions' dependencies.

Acceptance criteria

Given a job container where apt is unavailable or failing, when openbao-fetch runs, then the secret fetch still succeeds.
Given the existing consumers' workflows unchanged, when they run, then behaviour is identical to today on the happy path.
Given the change, when check-actions.py runs, then it passes.

### Context openbao-fetch installs its two runtime dependencies (`curl`, `jq`) with an in-job `apt-get` when the job container lacks them. On 2026-09-07 ahmad/trip's main run 2462 went red because Debian's bullseye-security `InRelease` was momentarily expired ("invalid since 34s"): apt failed, `jq` stayed absent, the OpenBao fetch exited 127, and the downstream notify-telegram step never ran. A transient mirror or clock hiccup at any consumer thus fails jobs that need no network beyond the vault and Telegram — and it will strike deploy jobs the same way, not just notifications. The action already runs inside node images, where a JSON parser is guaranteed without any package install; report-job-failure shed its jq dependency for the same class of reason (#53). ### Scope - In: openbao-fetch works without installing packages at runtime, or its dependency install tolerates transient apt failures; existing consumers unchanged. - Out: retrying the vault or Telegram calls themselves; other actions' dependencies. ### Acceptance criteria - [ ] Given a job container where apt is unavailable or failing, when openbao-fetch runs, then the secret fetch still succeeds. - [ ] Given the existing consumers' workflows unchanged, when they run, then behaviour is identical to today on the happy path. - [ ] Given the change, when `check-actions.py` runs, then it passes.

ahmad added the

labels

2026-09-07 21:15:38 +00:00

ahmad commented

2026-09-07 21:22:33 +00:00

Author

Owner

Second occurrence within the hour, widening the evidence beyond openbao-fetch: ahmad/expiro PR #66's image job died at its own in-workflow apt-get on the identical signature — Release file for …/bullseye-security/InRelease is expired (invalid since 3min 50s) (job 6329, 21:16Z; trip's notify was invalid since 34s at 21:13Z). The invalid-since delta tracking wall time says the mirror's Valid-Until lapsed and the runner kept fetching the stale file — so every in-job apt-get in the fleet shares this failure mode, not just this action's. Worth keeping this issue scoped to openbao-fetch (guaranteed-parser fix), but the general pattern is: an apt failure inside a job is a transient signature, and jobs should not depend on apt at runtime where the image already carries what they need.

Second occurrence within the hour, widening the evidence beyond openbao-fetch: ahmad/expiro PR #66's `image` job died at its own in-workflow `apt-get` on the identical signature — `Release file for …/bullseye-security/InRelease is expired (invalid since 3min 50s)` (job 6329, 21:16Z; trip's notify was `invalid since 34s` at 21:13Z). The invalid-since delta tracking wall time says the mirror's Valid-Until lapsed and the runner kept fetching the stale file — so every in-job `apt-get` in the fleet shares this failure mode, not just this action's. Worth keeping this issue scoped to openbao-fetch (guaranteed-parser fix), but the general pattern is: an apt failure inside a job is a transient signature, and jobs should not depend on apt at runtime where the image already carries what they need.

ahmad commented

2026-09-07 21:52:26 +00:00

Author

Owner

Permanence update, third repo and fifth failure: this is not a lazy CDN. suhba/khitta run 2468 (21:43Z) and two more expiro re-runs show the identical signature with the invalid-since delta still tracking wall time (now 30min+). Debian 11 LTS ended 2026-08-31; the affected jobs run in the runner's default bullseye-based container, and bullseye-security's final InRelease has now aged out for good — no retry will ever fix an in-job apt-get update there again. Fleet fix filed as an infrastructure issue (runner default image); this issue remains the right action-level fix: openbao-fetch runs on node images where a JSON parser needs no package install at all.

Permanence update, third repo and fifth failure: this is not a lazy CDN. suhba/khitta run 2468 (21:43Z) and two more expiro re-runs show the identical signature with the invalid-since delta still tracking wall time (now 30min+). Debian 11 LTS ended 2026-08-31; the affected jobs run in the runner's *default* bullseye-based container, and bullseye-security's final InRelease has now aged out for good — no retry will ever fix an in-job `apt-get update` there again. Fleet fix filed as an infrastructure issue (runner default image); this issue remains the right action-level fix: openbao-fetch runs on node images where a JSON parser needs no package install at all.

ahmad self-assigned this

2026-09-07 22:44:12 +00:00

ahmad added

state/in-progress

and removed

state/ready

labels

2026-09-07 22:44:13 +00:00

ahmad commented

2026-09-07 22:44:13 +00:00

Author

Owner

Plan. One file, openbao-fetch/action.yml, the fetch step. The deciding fact, read from the runner config: the default job label maps to node:20-bullseye, and every declared container in the fleet is a node image too — so node is on PATH wherever this action runs, and both jq parses (login token extract, kv-to-KEY=VALUE render) become node -e one-liners with no package install at all. curl stays required (present in every node Debian image); the on-demand apt fallback shrinks to curl-only and is effectively dead code on the fleet, kept as a last resort. A missing node fails fast with a message naming the requirement rather than reaching for apt. Behavioural note recorded here: a malformed vault response now fails the step (JSON.parse throws) where jq would emit empty output and exit 0 — stricter, and the right direction. Verification: bash -n via check-actions.py, plus a local round trip of both parsers against captured-shape fixtures (login response with and without client_token; kv response with several field types), and the existing consumers' inputs untouched — criterion 2.

Plan. One file, openbao-fetch/action.yml, the fetch step. The deciding fact, read from the runner config: the default job label maps to `node:20-bullseye`, and every declared container in the fleet is a node image too — so `node` is on PATH wherever this action runs, and both jq parses (login token extract, kv-to-KEY=VALUE render) become `node -e` one-liners with no package install at all. curl stays required (present in every node Debian image); the on-demand apt fallback shrinks to curl-only and is effectively dead code on the fleet, kept as a last resort. A missing node fails fast with a message naming the requirement rather than reaching for apt. Behavioural note recorded here: a malformed vault response now fails the step (JSON.parse throws) where jq would emit empty output and exit 0 — stricter, and the right direction. Verification: bash -n via check-actions.py, plus a local round trip of both parsers against captured-shape fixtures (login response with and without client_token; kv response with several field types), and the existing consumers' inputs untouched — criterion 2.

ahmad referenced this issue from a pull request that will close it,

2026-09-07 22:45:46 +00:00

fix: Parse OpenBao responses with node, not jq #58

ahmad

2026-09-07 22:45:46 +00:00