Zero-downtime deploys: start the new api, wait until healthy, then stop the old one

ahmad commented

2026-10-08 22:11:26 +00:00

Owner

Context

Every deploy through the shared deploy-vps path stops the api container before starting the new one. That gives about 10 s of 502s per deploy, which was seen on rihla in the morning of 8 Oct 2026. While families are registering, that window costs real requests. Ahmad approved this as a follow-up on 9 Oct 2026 (overnight ticket w4rpP0Jy). Until it lands, rihla avoids deploys in its registration windows and keeps its migrations additive (suhba/rihla#417).

Scope

In: a rolling swap in the shared deploy path: start the new api, wait for its healthcheck, shift traffic, then stop the old one. Devops's outline: either docker-rollout or scale api=2 with Traefik healthcheck labels, and dropping the fixed container_name this needs. Rollback when the new one never becomes healthy. A doc for consuming repos.
Out: database migration strategy (stays additive-only in each app); multi-host.

Acceptance criteria

Given a deploy under steady traffic, when it runs, then no request gets a 502 or connection error (measured with a load loop during the deploy).
Given a new version that never becomes healthy, when deployed, then the old one keeps serving and the deploy reports failure.
Given a consuming repo, when it opts in, then the change it needs is documented and small.
Deploy success is reported only after every container is healthy.
Given the old replica is still doing work, when the swap stops it, then it receives SIGTERM and gets its configured stop grace period to finish in-flight work before it is killed.
Given a consuming repo that does not set the new input, when it deploys, then the behaviour is exactly today's.

Estimated half a day to a day with testing. Rihla's adoption: suhba/rihla#417.

### Context Every deploy through the shared deploy-vps path stops the api container before starting the new one. That gives about 10 s of 502s per deploy, which was seen on rihla in the morning of 8 Oct 2026. While families are registering, that window costs real requests. Ahmad approved this as a follow-up on 9 Oct 2026 (overnight ticket w4rpP0Jy). Until it lands, rihla avoids deploys in its registration windows and keeps its migrations additive (suhba/rihla#417). ### Scope - In: a rolling swap in the shared deploy path: start the new api, wait for its healthcheck, shift traffic, then stop the old one. Devops's outline: either docker-rollout or `scale api=2` with Traefik healthcheck labels, and dropping the fixed `container_name` this needs. Rollback when the new one never becomes healthy. A doc for consuming repos. - Out: database migration strategy (stays additive-only in each app); multi-host. ### Acceptance criteria - [ ] Given a deploy under steady traffic, when it runs, then no request gets a 502 or connection error (measured with a load loop during the deploy). - [ ] Given a new version that never becomes healthy, when deployed, then the old one keeps serving and the deploy reports failure. - [ ] Given a consuming repo, when it opts in, then the change it needs is documented and small. - [ ] Deploy success is reported only after every container is healthy. - [ ] Given the old replica is still doing work, when the swap stops it, then it receives SIGTERM and gets its configured stop grace period to finish in-flight work before it is killed. - [ ] Given a consuming repo that does not set the new input, when it deploys, then the behaviour is exactly today's. Estimated half a day to a day with testing. Rihla's adoption: suhba/rihla#417.

ahmad added the

labels

2026-10-08 22:11:26 +00:00

ahmad added

state/in-progress

and removed

state/ready

labels

2026-10-09 10:12:21 +00:00

ahmad self-assigned this

2026-10-09 10:12:21 +00:00

ahmad commented

2026-10-09 10:12:32 +00:00

Author

Owner

Picked up.

Proving app: suhba/suhba-web. It runs on the same VPS as rihla, behind the same Traefik v3.3 docker provider, so it tests the mechanism rihla will actually use. portfolio and namey sit behind NPM with an nginx that proxies the api by name, which works differently.
Approach: a rolling swap in plain compose, with no new host tooling such as docker-rollout. Opt-in input rolling-services; empty keeps today's path byte-for-byte. For each listed service:
1. up --scale <svc>=2N --no-recreate --no-deps
2. wait for the new containers' healthchecks by id
3. Traefik routes to them once they are healthy
4. docker stop the old ones, so the image's stop signal and the compose stop_grace_period apply
5. remove the old ones
If a new container goes unhealthy or runs out of time, it is removed, the old ones keep serving, and the deploy fails.
Added two acceptance criteria to the body: a graceful stop, and no change for consumers that don't opt in.

Picked up. - **Proving app: `suhba/suhba-web`.** It runs on the same VPS as rihla, behind the same Traefik v3.3 docker provider, so it tests the mechanism rihla will actually use. portfolio and namey sit behind NPM with an nginx that proxies the api by name, which works differently. - **Approach:** a rolling swap in plain compose, with no new host tooling such as docker-rollout. Opt-in input `rolling-services`; empty keeps today's path byte-for-byte. For each listed service: 1. `up --scale <svc>=2N --no-recreate --no-deps` 2. wait for the new containers' healthchecks by id 3. Traefik routes to them once they are healthy 4. `docker stop` the old ones, so the image's stop signal and the compose `stop_grace_period` apply 5. remove the old ones If a new container goes unhealthy or runs out of time, it is removed, the old ones keep serving, and the deploy fails. - Added two acceptance criteria to the body: a graceful stop, and no change for consumers that don't opt in.

ahmad referenced this issue from a pull request that will close it,

2026-10-09 10:18:00 +00:00

feat: Add an opt-in zero-downtime rolling swap to deploy-vps #92

ahmad commented

2026-10-09 10:25:38 +00:00

Author

Owner

Progress, proving on suhba-web (suhba/suhba-web#102, deploy run from the branch). The load loop sends 4 parallel GET https://suhba.nl/ about every 50 ms from outside.

Run 1 (the move from the fixed name suhba-web-1, run 4664): success, 2,224 × 200 and 32 × 404 in a single 2 s window.
- Cause: Traefik logged "Router defined multiple times with different configurations". The old and new replicas carried different router labels (the new retry middleware), and Traefik drops a router whose replicas disagree.
- This only happens on a deploy that changes Traefik labels. It will be documented.
Run 2 (identical labels, run 4667): success, 2,126 × 200 and 2 × 000.
- The access log shows both requests went to the old replica's IP just after it died and hung about 10 s with no retry (RetryAttempts: 0).
- Traefik only reloads on the die event, so for about 2 s it still dials an address nobody answers. The dial hangs until Traefik's 30 s default dial timeout, and the retry middleware never gets an error to retry on.
Fix in progress: a shared Traefik serversTransport with a 2 s dial timeout that rolled services reference. A dead address then fails fast and the retry sends the request to the new replica. Testing it on the VPS next. The durable home is the traefik role in infrastructure-vps; I will open an issue there.

Progress, proving on suhba-web (suhba/suhba-web#102, deploy run from the branch). The load loop sends 4 parallel `GET https://suhba.nl/` about every 50 ms from outside. - **Run 1** (the move from the fixed name `suhba-web-1`, run 4664): success, 2,224 × 200 and 32 × 404 in a single 2 s window. - Cause: Traefik logged "Router defined multiple times with different configurations". The old and new replicas carried different router labels (the new retry middleware), and Traefik drops a router whose replicas disagree. - This only happens on a deploy that changes Traefik labels. It will be documented. - **Run 2** (identical labels, run 4667): success, 2,126 × 200 and 2 × 000. - The access log shows both requests went to the old replica's IP just after it died and hung about 10 s with no retry (`RetryAttempts: 0`). - Traefik only reloads on the `die` event, so for about 2 s it still dials an address nobody answers. The dial hangs until Traefik's 30 s default dial timeout, and the retry middleware never gets an error to retry on. - **Fix in progress:** a shared Traefik `serversTransport` with a 2 s dial timeout that rolled services reference. A dead address then fails fast and the retry sends the request to the new replica. Testing it on the VPS next. The durable home is the traefik role in infrastructure-vps; I will open an issue there.

ahmad commented

2026-10-09 10:48:05 +00:00

Author

Owner

Results on suhba-web. Every run deployed v0.22.0 from the branch suhba/suhba-web#102 using deploy-vps from #92. The load loop sends 4 parallel GET https://suhba.nl/ every ~50 ms from outside the VPS; any non-200 or connection error counts as a failure.

Run	What	Requests	Failed
4664	move off the fixed name, new router labels	2,256	32 × 404 in 2 s: Traefik refuses a router whose replicas carry different labels
4667	swap, no settle, retry only	2,128	2 × timeout: dial to the stopped replica hung, no retry
4670	+ 2 s dial-timeout transport	2,380	0
4676	+ 10 s settle, 3 retries	2,420	1 × 504: an in-flight request's retries stayed on the old routing table and hit the stopped replica 3×
4678	final: settle 10 s, dial timeout 1 s, retry ×5	2,348	0
4679	final, again	2,508	0

In the final setup, a request that hits the stopping replica costs 1–3 s instead of failing (4 such requests in the access log across both runs, all 200).

Rollback (run 4675): the healthcheck was pointed at a dead port, under load. The new replica went unhealthy and was removed, the deploy reported failure, the old replica kept serving, and 3,336 of 3,336 requests returned 200.

What changed after the first comment:

#92 gained rolling-settle-seconds (default 10): the old replica keeps serving until Traefik has picked up the new one. Without it, Traefik's only server could be the stopped one (run 4676's cause).
The Traefik transport rolling-deploy@file (dial timeout 1 s) sits on the VPS by hand for now. Making it ansible-managed is amtronics/infrastructure-vps#544.
The README documents the consumer's four changes, and the caveat that a deploy changing a rolled service's Traefik labels has a ~2 s 404 window.
Graceful stop: the old replica gets plain docker stop, so its stop signal and its stop_grace_period apply (nginx images: SIGQUIT). A behaviour test kills the stop -t 0 mutant.

Rihla was not touched; its adoption stays suhba/rihla#417. Rihla's api needs stop_grace_period above its longest welcome-email send.

Results on suhba-web. Every run deployed v0.22.0 from the branch suhba/suhba-web#102 using deploy-vps from #92. The load loop sends 4 parallel `GET https://suhba.nl/` every ~50 ms from outside the VPS; any non-200 or connection error counts as a failure. | Run | What | Requests | Failed | |---|---|---|---| | 4664 | move off the fixed name, new router labels | 2,256 | 32 × 404 in 2 s: Traefik refuses a router whose replicas carry different labels | | 4667 | swap, no settle, retry only | 2,128 | 2 × timeout: dial to the stopped replica hung, no retry | | 4670 | + 2 s dial-timeout transport | 2,380 | 0 | | 4676 | + 10 s settle, 3 retries | 2,420 | 1 × 504: an in-flight request's retries stayed on the old routing table and hit the stopped replica 3× | | **4678** | **final: settle 10 s, dial timeout 1 s, retry ×5** | **2,348** | **0** | | **4679** | **final, again** | **2,508** | **0** | In the final setup, a request that hits the stopping replica costs 1–3 s instead of failing (4 such requests in the access log across both runs, all 200). **Rollback** (run 4675): the healthcheck was pointed at a dead port, under load. The new replica went unhealthy and was removed, the deploy reported failure, the old replica kept serving, and 3,336 of 3,336 requests returned 200. **What changed after the first comment:** - #92 gained `rolling-settle-seconds` (default 10): the old replica keeps serving until Traefik has picked up the new one. Without it, Traefik's only server could be the stopped one (run 4676's cause). - The Traefik transport `rolling-deploy@file` (dial timeout 1 s) sits on the VPS by hand for now. Making it ansible-managed is amtronics/infrastructure-vps#544. - The README documents the consumer's four changes, and the caveat that a deploy changing a rolled service's Traefik labels has a ~2 s 404 window. - Graceful stop: the old replica gets plain `docker stop`, so its stop signal and its `stop_grace_period` apply (nginx images: SIGQUIT). A behaviour test kills the `stop -t 0` mutant. Rihla was not touched; its adoption stays suhba/rihla#417. Rihla's api needs `stop_grace_period` above its longest welcome-email send.

ahmad

2026-10-09 10:48:21 +00:00

closed this issue
added
state/done
and removed
state/in-progress
labels

ahmad commented

2026-10-09 11:08:06 +00:00

Author

Owner

Merged and released as v0.23.0. Its release deploy (run 4687) went through deploy-vps@v3 (v3.9.0) under the same load loop: 8,468 of 8,468 requests returned 200 over 467 s, with a new image and the rolling swap. web now runs as suhba-web-web-11.

Merged and released as v0.23.0. Its release deploy (run 4687) went through `deploy-vps@v3` (v3.9.0) under the same load loop: **8,468 of 8,468 requests returned 200 over 467 s**, with a new image and the rolling swap. `web` now runs as `suhba-web-web-11`.

Rows
Columns

Zero-downtime deploys: start the new api, wait until healthy, then stop the old one #91

Context

Scope

Acceptance criteria