Rollback kills the new replica without its grace period #101

New issue

Closed

opened 2026-10-10 06:16:24 +00:00 by ahmad · 0 comments

ahmad commented

2026-10-10 06:16:24 +00:00

Owner

Context

When a new replica never turns healthy, the rolling swap removes it with docker rm -f: SIGKILL, with no stop signal and no grace period. An unhealthy replica still runs its background work. Rihla's api runs a welcome-email pass at startup, so a rollback that lands mid-pass kills the sends it has claimed, and those are never resent. Seen live in rihla run 891 (suhba/rihla#417); harmless that time because the pass had nothing to send.

Scope

In: stop new replicas on rollback with plain docker stop, so their stop signal and stop_grace_period apply, then remove them.
Out: the swap's happy path, which already stops the old replicas gracefully.

Acceptance criteria

Given a new replica that never turns healthy, when the swap rolls back, then it is stopped with its own stop signal and grace period before it is removed.
Given that rollback, then the old replicas are untouched and the deploy fails.

### Context When a new replica never turns healthy, the rolling swap removes it with `docker rm -f`: SIGKILL, with no stop signal and no grace period. An unhealthy replica still runs its background work. Rihla's api runs a welcome-email pass at startup, so a rollback that lands mid-pass kills the sends it has claimed, and those are never resent. Seen live in rihla run 891 (suhba/rihla#417); harmless that time because the pass had nothing to send. ### Scope - In: stop new replicas on rollback with plain `docker stop`, so their stop signal and `stop_grace_period` apply, then remove them. - Out: the swap's happy path, which already stops the old replicas gracefully. ### Acceptance criteria - [ ] Given a new replica that never turns healthy, when the swap rolls back, then it is stopped with its own stop signal and grace period before it is removed. - [ ] Given that rollback, then the old replicas are untouched and the deploy fails.

ahmad added the

labels

2026-10-10 06:16:24 +00:00

ahmad self-assigned this

2026-10-10 06:16:24 +00:00

ahmad referenced this issue from a pull request that will close it,

2026-10-10 06:17:56 +00:00

fix: Stop a rolled-back replica gracefully #102