fix: Stop a rolled-back replica gracefully #102

Merged

ahmad merged 1 commit from feature/101-graceful-rollback into main

2026-10-10 06:21:34 +00:00

ahmad commented

2026-10-10 06:17:56 +00:00

Owner

Issue

Closes #101

Problem

On rollback, the new replica that never turned healthy is removed with docker rm -f: SIGKILL, no grace. It still runs its background work, so rihla's startup welcome pass can be killed mid-send, and those claims are lost for good. Seen live in rihla run 891 (suhba/rihla#417); nothing was being sent that time.

Solution

Stop the new replicas with plain docker stop (their own stop signal and stop_grace_period), then remove them. This is the same treatment the old replicas get on a successful swap.

Review notes

The rollback behaviour case now asserts docker stop new1 and docker rm new1; it fails against main's rm -f.

Risks and trade-offs

A rollback now takes as long as the failed replica needs to stop. The old replicas keep serving throughout.

### Issue Closes #101 ### Problem On rollback, the new replica that never turned healthy is removed with `docker rm -f`: SIGKILL, no grace. It still runs its background work, so rihla's startup welcome pass can be killed mid-send, and those claims are lost for good. Seen live in rihla run 891 (suhba/rihla#417); nothing was being sent that time. ### Solution Stop the new replicas with plain `docker stop` (their own stop signal and `stop_grace_period`), then remove them. This is the same treatment the old replicas get on a successful swap. ### Review notes The rollback behaviour case now asserts `docker stop new1` and `docker rm new1`; it fails against main's `rm -f`. ### Risks and trade-offs A rollback now takes as long as the failed replica needs to stop. The old replicas keep serving throughout.

ahmad added 1 commit

2026-10-10 06:17:56 +00:00

fix: Stop a rolled-back replica gracefully

PR / validate (pull_request) Successful in 37s

Details

01accd51f2

A replica that never turned healthy was removed with rm -f, a SIGKILL
with no grace, though it may be mid-way through background work such
as rihla's welcome emails, whose claims a kill loses for good. It is
now stopped with its own stop signal and grace period first.

ahmad_bot approved these changes

2026-10-10 06:21:26 +00:00

ahmad_bot left a comment

Verified:

The rollback branch now runs docker stop (no -t, so each container's own stop signal and stop_grace_period apply) and then docker rm on the new replicas; the old replicas are not touched and the step still exits 1. Both #101 criteria hold.
Behaviour harness green on the head; with main's docker rm -f in place of the two lines, "unhealthy new replica: rolled back, the old one keeps serving" fails and nothing else changes, so the case pins the fix.
Title and subject 42 chars, one commit, no trailers, branch on current main, CI PR / validate green.

Noted, not a finding: a replica that ignores SIGTERM holds the failed deploy for its full grace period before the step reports; the PR states the trade-off and the old replicas serve throughout.

APPROVED.

Verified: - The rollback branch now runs `docker stop` (no `-t`, so each container's own stop signal and `stop_grace_period` apply) and then `docker rm` on the new replicas; the old replicas are not touched and the step still exits 1. Both #101 criteria hold. - Behaviour harness green on the head; with main's `docker rm -f` in place of the two lines, "unhealthy new replica: rolled back, the old one keeps serving" fails and nothing else changes, so the case pins the fix. - Title and subject 42 chars, one commit, no trailers, branch on current main, CI `PR / validate` green. Noted, not a finding: a replica that ignores SIGTERM holds the failed deploy for its full grace period before the step reports; the PR states the trade-off and the old replicas serve throughout. APPROVED.

ahmad merged commit 52f9ef0794 into main

2026-10-10 06:21:34 +00:00

ahmad deleted branch feature/101-graceful-rollback

2026-10-10 06:21:34 +00:00

ahmad referenced this pull request from a commit

2026-10-10 06:21:35 +00:00

fix: Stop a rolled-back replica gracefully (#102)