amtronics/platform-actions

Watch

Fork 0

feat: Add an opt-in zero-downtime rolling swap to deploy-vps #92

Merged

ahmad merged 3 commits from feature/91-rolling-deploy into main

2026-10-09 10:52:11 +00:00

ahmad commented

2026-10-09 10:18:00 +00:00

Owner

Issue

Closes #91

Problem

deploy-vps recreates every container, so each deploy stops the running one before the new one is up. That gives about 10 s of 502s per deploy (rihla, 8 Oct).

Solution

Opt-in input rolling-services. For each listed service:
- start a new replica beside the old one;
- wait for its healthcheck by container id;
- keep the old one serving for rolling-settle-seconds (default 10), so the proxy routes to the new one first;
- stop the old one with plain docker stop, so its stop signal and stop_grace_period apply, then remove it.
If the new replica turns unhealthy, has no HEALTHCHECK, or runs out of time, it is removed and the deploy fails. The old one keeps serving.
The other services are recreated first with --no-deps.
An empty input (the default) runs exactly today's command.
The README lists what a consumer changes: drop container_name, keep a healthcheck, add a Traefik retry (×5) and the rolling-deploy@file transport (1 s dial timeout), set stop_grace_period.

Review notes

Proven on suhba-web under load (numbers on #91): the final setup had zero failed requests in two runs (2,348 and 2,508 requests).
The rollback was proven with a deliberately unhealthy healthcheck: the deploy failed, the old replica kept serving, and all 3,336 requests returned 200.
Behaviour tests cover nine cases, including the default path, rollback, settle and graceful stop. A stop -t 0 mutant fails the graceful case.

Risks and trade-offs

Only rolled services are zero-downtime.
A deploy that changes a rolled service's Traefik labels has a ~2 s 404 window, because Traefik refuses a router whose replicas disagree.
The transport is hand-placed on the VPS until amtronics/infrastructure-vps#544 makes it ansible-managed.
The swap assumes one host and schema changes that both versions can run against.

### Issue Closes #91 ### Problem deploy-vps recreates every container, so each deploy stops the running one before the new one is up. That gives about 10 s of 502s per deploy (rihla, 8 Oct). ### Solution - Opt-in input `rolling-services`. For each listed service: - start a new replica beside the old one; - wait for its healthcheck by container id; - keep the old one serving for `rolling-settle-seconds` (default 10), so the proxy routes to the new one first; - stop the old one with plain `docker stop`, so its stop signal and `stop_grace_period` apply, then remove it. - If the new replica turns unhealthy, has no HEALTHCHECK, or runs out of time, it is removed and the deploy fails. The old one keeps serving. - The other services are recreated first with `--no-deps`. - An empty input (the default) runs exactly today's command. - The README lists what a consumer changes: drop `container_name`, keep a healthcheck, add a Traefik retry (×5) and the `rolling-deploy@file` transport (1 s dial timeout), set `stop_grace_period`. ### Review notes - Proven on suhba-web under load (numbers on #91): the final setup had zero failed requests in two runs (2,348 and 2,508 requests). - The rollback was proven with a deliberately unhealthy healthcheck: the deploy failed, the old replica kept serving, and all 3,336 requests returned 200. - Behaviour tests cover nine cases, including the default path, rollback, settle and graceful stop. A `stop -t 0` mutant fails the graceful case. ### Risks and trade-offs - Only rolled services are zero-downtime. - A deploy that changes a rolled service's Traefik labels has a ~2 s 404 window, because Traefik refuses a router whose replicas disagree. - The transport is hand-placed on the VPS until amtronics/infrastructure-vps#544 makes it ansible-managed. - The swap assumes one host and schema changes that both versions can run against.

ahmad added 1 commit

2026-10-09 10:18:00 +00:00

feat: Add an opt-in rolling swap to deploy-vps

PR / validate (pull_request) Successful in 34s

Details

37439867dd

A service listed in rolling-services gets a new replica beside the
old one; the old one is stopped, with its own stop signal and grace
period, only once the new one is healthy. A new replica that never
becomes healthy is removed and the deploy fails while the old one
keeps serving. Without the input the rollout is unchanged.

ahmad added 1 commit

2026-10-09 10:33:11 +00:00

feat: Let the proxy route to the new replica first

PR / validate (pull_request) Successful in 42s

Details

aa6e499872

ahmad added 1 commit

2026-10-09 10:47:36 +00:00

docs: Document the fail-over settings the swap needs

PR / validate (pull_request) Successful in 36s

Details

367319238f

ahmad referenced this pull request

2026-10-09 10:48:05 +00:00

Zero-downtime deploys: start the new api, wait until healthy, then stop the old one #91

ahmad changed title from ~~WIP: feat: Add an opt-in zero-downtime rolling swap to deploy-vps~~ to feat: Add an opt-in zero-downtime rolling swap to deploy-vps

2026-10-09 10:48:20 +00:00

ahmad requested review from ahmad_bot

2026-10-09 10:48:21 +00:00

ahmad_bot approved these changes

2026-10-09 10:51:36 +00:00

ahmad_bot left a comment

Reviewed at 367319238f. CI green (validate: checker, harness tests, behaviour tests).

Verified against #91:

No request lost under traffic, and the old replica keeps serving when the new one never becomes healthy: the swap starts the new replica with --no-recreate --scale, waits on it by id, settles, then stops the old one; unhealthy, no HEALTHCHECK, 30 s of unknown or the budget all remove the new replicas and exit 1 before anything old is touched. Live runs 4678/4679 (0 failed) and the rollback run 4675 (3,336/3,336) are on the issue.
Graceful stop: docker stop without -t, so the container's own StopTimeout from stop_grace_period applies; a behaviour case asserts no stop -t.
Not opted in: the up --force-recreate --remove-orphans line is unchanged and the new code is behind -z/-n guards; the only additions on that path are two no-op validations and two env assignments on the ssh line. Fifteen consumers use this action and none sets the input yet.
Success only after health: rolled replicas by id, the rest through the existing name loop.
Consumer changes documented (README) and small; neither rihla nor suhba-web binds host ports, so the second replica can start once container_name goes.
Shell safety: service names are checked against ^[A-Za-z0-9][A-Za-z0-9_.-]*$ and the settle against ^[0-9]+$ before they enter the single-quoted ssh command; grep -qxF on the service list; stale exited/created/dead replicas removed before counting so up cannot revive one and count it as serving; first deploy scales to 1 and stops nothing.
Behaviour tests slice the real if blocks out of the action, so a renamed guard fails loudly rather than testing nothing.

One note, non-blocking: on a rollback the other services already run the new revision while the rolled one stays on the old, until the next deploy. The README's "a schema both versions can run against" covers the database; a sentence saying the mixed state is expected after a failed swap would save a diagnosis.

Reviewed at 367319238ffce28a4089f7e5c81f3f575e96b23f. CI green (validate: checker, harness tests, behaviour tests). Verified against #91: - No request lost under traffic, and the old replica keeps serving when the new one never becomes healthy: the swap starts the new replica with `--no-recreate --scale`, waits on it by id, settles, then stops the old one; `unhealthy`, no `HEALTHCHECK`, 30 s of `unknown` or the budget all remove the new replicas and exit 1 before anything old is touched. Live runs 4678/4679 (0 failed) and the rollback run 4675 (3,336/3,336) are on the issue. - Graceful stop: `docker stop` without `-t`, so the container's own `StopTimeout` from `stop_grace_period` applies; a behaviour case asserts no `stop -t`. - Not opted in: the `up --force-recreate --remove-orphans` line is unchanged and the new code is behind `-z`/`-n` guards; the only additions on that path are two no-op validations and two env assignments on the ssh line. Fifteen consumers use this action and none sets the input yet. - Success only after health: rolled replicas by id, the rest through the existing name loop. - Consumer changes documented (README) and small; neither rihla nor suhba-web binds host ports, so the second replica can start once `container_name` goes. - Shell safety: service names are checked against `^[A-Za-z0-9][A-Za-z0-9_.-]*$` and the settle against `^[0-9]+$` before they enter the single-quoted ssh command; `grep -qxF` on the service list; stale exited/created/dead replicas removed before counting so `up` cannot revive one and count it as serving; first deploy scales to 1 and stops nothing. - Behaviour tests slice the real `if` blocks out of the action, so a renamed guard fails loudly rather than testing nothing. One note, non-blocking: on a rollback the other services already run the new revision while the rolled one stays on the old, until the next deploy. The README's "a schema both versions can run against" covers the database; a sentence saying the mixed state is expected after a failed swap would save a diagnosis.

ahmad merged commit c8e651a664 into main

2026-10-09 10:52:11 +00:00

ahmad deleted branch feature/91-rolling-deploy

2026-10-09 10:52:11 +00:00

ahmad referenced this pull request from a commit

2026-10-09 10:52:11 +00:00

feat: Add an opt-in zero-downtime rolling swap to deploy-vps (#92)

ahmad referenced this pull request

2026-10-09 10:54:02 +00:00

Document the mixed revision state after a failed rolling swap #93