amtronics/platform-actions

Watch

Fork 0

Extract the observability contract into reusable actions #1

New issue

Open

opened 2026-07-28 13:25:14 +00:00 by ahmad · 2 comments

ahmad commented

2026-07-28 13:25:14 +00:00

Owner

Design agreed, nothing implemented. This tracks turning the contract in the
README into reusable actions.

What ships here

Action	What it does
`observe-metrics`	adds the alloy sidecar and wires its credential
`observe-logs`	the compose fragment and log labels for Loki
`observe-analytics`	the umami build-time variables
`register-uptime`	declares an app's public hostnames as probe targets

Rules the implementation has to hold

Degrade to a skip, never to a failure. Every piece must no-op when its
variables are unset. The portfolio's monitoring workflow already does this and
it is the reason merges stayed green for weeks before grafana existed.
Observability that can break a deployment gets deleted by someone under
pressure at 2am, and they will be right to do it.

One credential per app. Revoking one app must not rotate a secret the
others hold.

Consumers pin a tag. An action referenced by every repository is production
infrastructure — a bad commit on main breaks every pipeline simultaneously,
including the one needed to ship the fix.

Order

ahmad/portfolio is the reference implementation: it already does metrics,
analytics and monitoring sync by hand. Extract from it rather than designing in
the abstract, then migrate it to consume what was extracted. If extraction makes
the portfolio's own setup worse, the abstraction is wrong.

Second consumer proves it. One consumer only proves it was copied.

Open question

register-uptime is the awkward one: the target list lives in
infrastructure-vps, so an app declaring a hostname either opens a PR there or
the probes discover targets from traefik labels. Discovery is less coupled and
harder to review. Decide before building it.

Depends on

The endpoints existing — infrastructure-vps #23 through #28.

Design agreed, nothing implemented. This tracks turning the contract in the README into reusable actions. ## What ships here | Action | What it does | |---|---| | `observe-metrics` | adds the alloy sidecar and wires its credential | | `observe-logs` | the compose fragment and log labels for Loki | | `observe-analytics` | the umami build-time variables | | `register-uptime` | declares an app's public hostnames as probe targets | ## Rules the implementation has to hold **Degrade to a skip, never to a failure.** Every piece must no-op when its variables are unset. The portfolio's monitoring workflow already does this and it is the reason merges stayed green for weeks before grafana existed. Observability that can break a deployment gets deleted by someone under pressure at 2am, and they will be right to do it. **One credential per app.** Revoking one app must not rotate a secret the others hold. **Consumers pin a tag.** An action referenced by every repository is production infrastructure — a bad commit on `main` breaks every pipeline simultaneously, including the one needed to ship the fix. ## Order `ahmad/portfolio` is the reference implementation: it already does metrics, analytics and monitoring sync by hand. Extract from it rather than designing in the abstract, then migrate it to consume what was extracted. If extraction makes the portfolio's own setup worse, the abstraction is wrong. Second consumer proves it. One consumer only proves it was copied. ## Open question `register-uptime` is the awkward one: the target list lives in `infrastructure-vps`, so an app declaring a hostname either opens a PR there or the probes discover targets from traefik labels. Discovery is less coupled and harder to review. Decide before building it. ## Depends on The endpoints existing — infrastructure-vps #23 through #28.

ahmad commented

2026-08-07 15:46:33 +00:00

Author

Owner

Revisited before implementing. Four findings change the shape of this, and one
of them changes whether the main action exists at all.

The dependency list is wrong

This says it depends on infrastructure-vps #23 through #28. Two of the four
endpoints already exist:

	Status
`observe-metrics`	Unblocked. The remote-write endpoint, traefik basic auth and per-app htpasswd entries have been live since #19 and #24. #23 moves the stack to its own VPS, but the hostname is deliberately stable, so nothing here waits on it.
`observe-analytics`	Unblocked. Umami has been running since #7.
`observe-logs`	Blocked. Loki is #25, unbuilt.
`register-uptime`	Blocked. Blackbox #26 and Kuma #27 are unbuilt, and the open question below cannot be answered against something that does not exist.

Waiting on the whole range was waiting for nothing.

The collector model was the real question, and it has been decided

observe-metrics assumed each application ships its own alloy. Two already do,
and in under a month they diverged on image version, user, memory cap, buffer
strategy, network and destination — the exact drift this repository exists to
stop, reproduced inside the thing meant to stop it.

The alternative is that the alloy already running on every managed host
discovers application containers by docker label and scrapes them, the same way
traefik already discovers routes. Decided in favour of that; recorded as an ADR
in infrastructure-vps#43.

Consequences here:

observe-metrics is not an action. It becomes a five-line compose
fragment: a data attachment and two labels. No sidecar, no credential.
observe-logs will very likely go the same way once #25 lands, since the
same host collector will carry logs.
"One credential per application" no longer applies to metrics. An
application holds nothing. That serves the intent better than the rule did —
nothing to leak, rotate or revoke — but the README states it as a general
property and needs rewriting.

`observe-analytics` probably should not be an action either

It is three build arguments on the image build. A composite action cannot inject
build arguments into a build step it does not own; the most it can return is a
string. The real content is the rule — both variables present or no script is
injected, and the allowed-domain list must equal the deploy domain. That is a
documented fragment. Making it an action to complete a table of four is building
an abstraction for symmetry.

The most duplicated thing in the estate is not in this issue

Both consumers already provision Grafana dashboards and alert rules from their
own repositories, by hand, with separate implementations. The same bug was then
fixed independently in each: ahmad/portfolio#49 and ahmad/trip#104 are the same
defect, found and fixed twice, weeks apart.

That is the strongest evidence for extraction anywhere in the estate, and
sync-grafana is not in the four this issue lists. With metrics and logs
collapsing to fragments, it becomes the repository's actual content.

Consumption requires this repository to be public

Every repository here is private, and Forgejo scopes the per-run token to the
running repository. A consumer's runner cannot clone a private action
repository without a shared read token held by every consumer — which is
precisely the "revoking one application rotates a secret the others hold"
failure this issue forbids. This repository holds YAML and shell, no secrets.
Agreed to make it public.

Proposed sequence

infrastructure-vps#43 — ADR, label discovery, absent-job alert, docs.
ahmad/portfolio migrates off its sidecar. Cheapest first: its collector is
not running anywhere today, so this is a deletion plus labels, and it also
moves the repository off Grafana Cloud onto the self-hosted endpoint.
ahmad/trip migrates. Its collector is live and pushing, so this is a real
cutover and proves the contract on a second consumer.
sync-grafana extracted here, consumed by both.
This issue rewritten as the tracker once the fragment's shape is known from
step 2 rather than guessed now.

Logs and uptime stay closed until #25 and #26/#27 land. The register-uptime
open question is deferred with them.

Revisited before implementing. Four findings change the shape of this, and one of them changes whether the main action exists at all. ## The dependency list is wrong This says it depends on infrastructure-vps #23 through #28. Two of the four endpoints already exist: | | Status | |---|---| | `observe-metrics` | **Unblocked.** The remote-write endpoint, traefik basic auth and per-app htpasswd entries have been live since #19 and #24. #23 moves the stack to its own VPS, but the hostname is deliberately stable, so nothing here waits on it. | | `observe-analytics` | **Unblocked.** Umami has been running since #7. | | `observe-logs` | Blocked. Loki is #25, unbuilt. | | `register-uptime` | Blocked. Blackbox #26 and Kuma #27 are unbuilt, and the open question below cannot be answered against something that does not exist. | Waiting on the whole range was waiting for nothing. ## The collector model was the real question, and it has been decided `observe-metrics` assumed each application ships its own alloy. Two already do, and in under a month they diverged on image version, user, memory cap, buffer strategy, network and destination — the exact drift this repository exists to stop, reproduced inside the thing meant to stop it. The alternative is that the alloy already running on every managed host discovers application containers by docker label and scrapes them, the same way traefik already discovers routes. Decided in favour of that; recorded as an ADR in infrastructure-vps#43. Consequences here: - **`observe-metrics` is not an action.** It becomes a five-line compose fragment: a `data` attachment and two labels. No sidecar, no credential. - **`observe-logs` will very likely go the same way** once #25 lands, since the same host collector will carry logs. - **"One credential per application" no longer applies to metrics.** An application holds nothing. That serves the intent better than the rule did — nothing to leak, rotate or revoke — but the README states it as a general property and needs rewriting. ## `observe-analytics` probably should not be an action either It is three build arguments on the image build. A composite action cannot inject build arguments into a build step it does not own; the most it can return is a string. The real content is the rule — both variables present or no script is injected, and the allowed-domain list must equal the deploy domain. That is a documented fragment. Making it an action to complete a table of four is building an abstraction for symmetry. ## The most duplicated thing in the estate is not in this issue Both consumers already provision Grafana dashboards and alert rules from their own repositories, by hand, with separate implementations. The same bug was then fixed independently in each: ahmad/portfolio#49 and ahmad/trip#104 are the same defect, found and fixed twice, weeks apart. That is the strongest evidence for extraction anywhere in the estate, and `sync-grafana` is not in the four this issue lists. With metrics and logs collapsing to fragments, it becomes the repository's actual content. ## Consumption requires this repository to be public Every repository here is private, and Forgejo scopes the per-run token to the running repository. A consumer's runner cannot clone a private action repository without a shared read token held by every consumer — which is precisely the "revoking one application rotates a secret the others hold" failure this issue forbids. This repository holds YAML and shell, no secrets. Agreed to make it public. ## Proposed sequence 1. infrastructure-vps#43 — ADR, label discovery, absent-job alert, docs. 2. `ahmad/portfolio` migrates off its sidecar. Cheapest first: its collector is not running anywhere today, so this is a deletion plus labels, and it also moves the repository off Grafana Cloud onto the self-hosted endpoint. 3. `ahmad/trip` migrates. Its collector is live and pushing, so this is a real cutover and proves the contract on a second consumer. 4. `sync-grafana` extracted here, consumed by both. 5. This issue rewritten as the tracker once the fragment's shape is known from step 2 rather than guessed now. Logs and uptime stay closed until #25 and #26/#27 land. The `register-uptime` open question is deferred with them.

ahmad commented

2026-09-04 02:33:17 +00:00

Author

Owner

Refining rather than starting — my implementation slot is elsewhere and every image build on the fleet is currently blocked on the runner's Docker Hub egress, so nothing merges tonight regardless.

Labelled type/feature, priority/P2, size/XL, and left state/needs-refinement rather than moved to ready. The design here is good and the revisit comment is better, but as one issue this cannot be finished, and an issue that cannot be finished sits in in-progress for weeks looking like slow work rather than blocked work.

The four parts are in three different states, per your own revisit:

	State	What it waits on
`observe-metrics`	ready to build	nothing
`observe-analytics`	ready to build	nothing
`observe-logs`	blocked	Loki, `infrastructure-vps#25`, unbuilt
`register-uptime`	undecided	whether probes discover targets from traefik labels or an app opens a PR against the target list

My recommendation is to split this into four, one per action, with this issue kept as the parent holding the contract and the rules. Two children start immediately; two carry their blocker explicitly. That is the difference between "this epic is 50% done" and "two of these are shipped and two are waiting on named things".

I have not created them — splitting someone else's design is a bigger intervention than refinement, and the parent's framing is yours. Say the word and I will.

The rules in the body are the valuable part and should survive the split, especially the first one. "Degrade to a skip, never to a failure" is the reason ahmad/portfolio's monitoring stayed green for weeks before Grafana existed, and the argument for it — that observability which can break a deployment gets deleted at 2am by someone who is right to delete it — is worth repeating in each child rather than linking to.

One thing I would add to the rules, from a failure on ahmad/expiro tonight: a piece that no-ops when unconfigured must also say that it no-opped. expiro's invite delivery returns skipped and logs it; the sweep leaves a reminder due rather than recording it as sent. A silent skip and a working system are indistinguishable until someone asks why no data arrived, which is usually months later.

On the open question — register-uptime, discovery versus a PR. Discovery from traefik labels is less coupled and harder to review, as you say. Worth adding that it is also the option that fails silently: a mislabelled container simply is not probed, and nothing anywhere reports a target that was never registered. The PR approach is more friction and the friction is visible. That is a genuine trade-off and yours to make, not mine.

Refining rather than starting — my implementation slot is elsewhere and every image build on the fleet is currently blocked on the runner's Docker Hub egress, so nothing merges tonight regardless. Labelled `type/feature`, `priority/P2`, `size/XL`, and left **`state/needs-refinement`** rather than moved to ready. The design here is good and the revisit comment is better, but as one issue this cannot be finished, and an issue that cannot be finished sits in `in-progress` for weeks looking like slow work rather than blocked work. **The four parts are in three different states**, per your own revisit: | | State | What it waits on | |---|---|---| | `observe-metrics` | ready to build | nothing | | `observe-analytics` | ready to build | nothing | | `observe-logs` | blocked | Loki, `infrastructure-vps#25`, unbuilt | | `register-uptime` | undecided | whether probes discover targets from traefik labels or an app opens a PR against the target list | **My recommendation is to split this into four**, one per action, with this issue kept as the parent holding the contract and the rules. Two children start immediately; two carry their blocker explicitly. That is the difference between "this epic is 50% done" and "two of these are shipped and two are waiting on named things". I have not created them — splitting someone else's design is a bigger intervention than refinement, and the parent's framing is yours. Say the word and I will. **The rules in the body are the valuable part and should survive the split**, especially the first one. "Degrade to a skip, never to a failure" is the reason `ahmad/portfolio`'s monitoring stayed green for weeks before Grafana existed, and the argument for it — that observability which can break a deployment gets deleted at 2am by someone who is right to delete it — is worth repeating in each child rather than linking to. **One thing I would add to the rules**, from a failure on `ahmad/expiro` tonight: a piece that no-ops when unconfigured must also **say that it no-opped**. `expiro`'s invite delivery returns `skipped` and logs it; the sweep leaves a reminder due rather than recording it as sent. A silent skip and a working system are indistinguishable until someone asks why no data arrived, which is usually months later. **On the open question** — `register-uptime`, discovery versus a PR. Discovery from traefik labels is less coupled and harder to review, as you say. Worth adding that it is also the option that fails silently: a mislabelled container simply is not probed, and nothing anywhere reports a target that was never registered. The PR approach is more friction and the friction is visible. That is a genuine trade-off and yours to make, not mine.

ahmad added the

priority/P2

size/XL

state/needs-refinement

type/feature

labels

2026-09-04 02:33:19 +00:00