amtronics/platform-actions

Watch

Fork 0

deploy-vps fails deployments that succeed: the health wait is 60s #22

New issue

Closed

opened 2026-09-02 20:54:19 +00:00 by ahmad · 1 comment

ahmad commented

2026-09-02 20:54:19 +00:00

Owner

Context

deploy-vps waits sixty seconds for a container to report healthy — for attempt in $(seq 1 30) with sleep 2 — and fails the deployment when it has not. On a loaded host that is shorter than a container takes to start, so the action reports a failure for a deployment that then completes successfully. The pipeline cries wolf, and the next red deploy is one nobody trusts.

Observed on ahmad/portfolio v0.15.1, 2 September 2026 (times UTC):

event	time	outcome
release deploy (run 5574)	20:23:42 → 20:26:02	failure
manual retry (run 5576)	20:33:16 → 20:50:45	failure
`https://ahmadelmasri.com/api/health`	20:46–20:51	502
same endpoint	20:52	200, `{"status":"ok","version":"v0.15.1"}`

The deployment worked. The api came healthy a minute or two after the second job had already given up and marked the run red, and production has been serving the new version since. Two red runs, nothing wrong with the release.

The host is the fleet's I/O-saturated runner host — see amtronics/infrastructure-vps#390 — so "the container is slow to start" is the normal case here rather than the exception.

Scope

In: how long deploy-vps waits for a container's healthcheck, and what it does when the wait runs out.
Out: making the containers start faster, the host's I/O saturation, and any change to which containers are waited on.

Acceptance criteria

Given a container that becomes healthy after more than sixty seconds, when the deployment runs, then it is reported as a success.
Given a container that never becomes healthy, when the wait runs out, then the deployment still fails and prints the container's logs as it does today.
Given a container that reports unhealthy rather than starting, when the deployment runs, then it fails without waiting out the full budget — a crash-looping image should not cost the whole timeout.
Given a caller that wants a different budget, when it sets the input, then the wait uses it; unset keeps a documented default.
Given the change, when README.md is read, then the new input and its default are documented alongside the other inputs.

Notes

Waiting on Docker's own health status is right; only the budget is wrong. Distinguishing starting from unhealthy is what lets the budget grow without making a genuinely broken image slow to report — Docker sets unhealthy only after the image's own retries are exhausted, so it is already a settled verdict when it appears.

### Context `deploy-vps` waits sixty seconds for a container to report healthy — `for attempt in $(seq 1 30)` with `sleep 2` — and fails the deployment when it has not. On a loaded host that is shorter than a container takes to start, so the action reports a failure for a deployment that then completes successfully. The pipeline cries wolf, and the next red deploy is one nobody trusts. Observed on `ahmad/portfolio` v0.15.1, 2 September 2026 (times UTC): | event | time | outcome | | --- | --- | --- | | release deploy (run 5574) | 20:23:42 → 20:26:02 | failure | | manual retry (run 5576) | 20:33:16 → 20:50:45 | failure | | `https://ahmadelmasri.com/api/health` | 20:46–20:51 | 502 | | same endpoint | 20:52 | **200, `{"status":"ok","version":"v0.15.1"}`** | The deployment worked. The api came healthy a minute or two after the second job had already given up and marked the run red, and production has been serving the new version since. Two red runs, nothing wrong with the release. The host is the fleet's I/O-saturated runner host — see `amtronics/infrastructure-vps#390` — so "the container is slow to start" is the normal case here rather than the exception. ### Scope - In: how long `deploy-vps` waits for a container's healthcheck, and what it does when the wait runs out. - Out: making the containers start faster, the host's I/O saturation, and any change to which containers are waited on. ### Acceptance criteria - [ ] Given a container that becomes healthy after more than sixty seconds, when the deployment runs, then it is reported as a success. - [ ] Given a container that never becomes healthy, when the wait runs out, then the deployment still fails and prints the container's logs as it does today. - [ ] Given a container that reports `unhealthy` rather than `starting`, when the deployment runs, then it fails without waiting out the full budget — a crash-looping image should not cost the whole timeout. - [ ] Given a caller that wants a different budget, when it sets the input, then the wait uses it; unset keeps a documented default. - [ ] Given the change, when `README.md` is read, then the new input and its default are documented alongside the other inputs. ### Notes Waiting on Docker's own health status is right; only the budget is wrong. Distinguishing `starting` from `unhealthy` is what lets the budget grow without making a genuinely broken image slow to report — Docker sets `unhealthy` only after the image's own retries are exhausted, so it is already a settled verdict when it appears.

ahmad added the

labels

2026-09-02 20:54:23 +00:00

ahmad self-assigned this

2026-09-02 22:02:57 +00:00

ahmad added

state/in-progress

and removed

state/ready

labels

2026-09-02 22:02:58 +00:00

ahmad commented

2026-09-02 22:03:17 +00:00

Author

Owner

Taking this. It is a shared action, so I want the change and its blast radius stated before the diff exists.

What changes. The health wait becomes a budget in seconds rather than a fixed thirty attempts, exposed as a health-timeout input with a default long enough for these hosts. The loop also distinguishes Docker's two negative answers: starting means keep waiting, unhealthy means stop now — Docker only reports unhealthy after the image's own retries are exhausted, so it is a settled verdict and waiting out the rest of the budget on it would make a genuinely broken image slow to report.

What does not change. Callers that set nothing keep working; the only difference they see is that a slow-starting container is no longer called a failure. A container that never becomes healthy still fails the deployment and still prints its logs.

Blast radius, stated plainly. Merging here moves the v3 tag, so every repository consuming deploy-vps@v3 picks this up on its next deployment. That is the mechanism working as designed, and it is why the default matters more than the input: I am choosing ten minutes, which is roughly six times the observed worst case on ahmad/portfolio and still far short of a run that hangs.

Ambiguity, and how I am resolving it. I cannot exercise this against a real host — the deploy key is in OpenBao and out of bounds for me — so I will test the loop as a shell function against a stubbed docker inspect, covering healthy-immediately, healthy-late, never-healthy and unhealthy-early, and say plainly in the pull request that the first real deployment is the live proof.

Taking this. It is a shared action, so I want the change and its blast radius stated before the diff exists. **What changes.** The health wait becomes a budget in seconds rather than a fixed thirty attempts, exposed as a `health-timeout` input with a default long enough for these hosts. The loop also distinguishes Docker's two negative answers: `starting` means keep waiting, `unhealthy` means stop now — Docker only reports `unhealthy` after the image's own retries are exhausted, so it is a settled verdict and waiting out the rest of the budget on it would make a genuinely broken image slow to report. **What does not change.** Callers that set nothing keep working; the only difference they see is that a slow-starting container is no longer called a failure. A container that never becomes healthy still fails the deployment and still prints its logs. **Blast radius, stated plainly.** Merging here moves the `v3` tag, so every repository consuming `deploy-vps@v3` picks this up on its next deployment. That is the mechanism working as designed, and it is why the default matters more than the input: I am choosing ten minutes, which is roughly six times the observed worst case on `ahmad/portfolio` and still far short of a run that hangs. **Ambiguity, and how I am resolving it.** I cannot exercise this against a real host — the deploy key is in OpenBao and out of bounds for me — so I will test the loop as a shell function against a stubbed `docker inspect`, covering healthy-immediately, healthy-late, never-healthy and unhealthy-early, and say plainly in the pull request that the first real deployment is the live proof.

ahmad referenced this issue from a pull request that will close it,

2026-09-02 22:05:35 +00:00

fix: Wait longer for a container to report healthy #24

ahmad added

state/in-review

and removed

state/in-progress

labels

2026-09-02 22:05:37 +00:00

ahmad

2026-09-02 23:36:41 +00:00