fix: Retry the release POST; a transient 500 strands the tag #40

New issue

Closed

opened 2026-09-04 11:47:57 +00:00 by ahmad · 1 comment

ahmad commented

2026-09-04 11:47:57 +00:00

Owner

Context

release/action.yml pushes the new tag and then immediately POSTs the release that
references it. When Forgejo has not finished indexing the tag, that POST returns 500,
curl -fsS exits 22, and the job fails — after the tag has been pushed.

The failure mode is bad out of proportion to the cause. The repository is left with a tag
and no release; publish and deploy are skipped because they need version; and the
next ordinary dispatch finds the tag already present, decides no release is warranted, and
exits 0. Recovering means either publishing the release by hand or force-bumping to a new
version, which strands the tagged one forever.

It is transient. Observed on ahmad/imamah v0.5.0 (its #148); replaying the identical call
by hand minutes later returned 201. The notes had already been generated and were lost
with the failed run — the rewrite had succeeded, notes_fallback=false.

Scope

In: making the release POST survive a transient failure.
Out: the notes rewrite; the tagging itself; recovery of releases already stranded.

Acceptance criteria

Given the release POST returns 500, when the step runs, then it retries with backoff
before failing the job.
Given a retry then succeeds, when the job finishes, then it is green and publish
runs as normal.
Given every retry fails, when the job fails, then the log names the status code and
the response body — not only curl: (22).
Given the POST succeeds first time, when the job runs, then behaviour and timing are
unchanged.

Notes

Two things are worth separating. The retry is the fix. But the ordering is what makes the
failure expensive: the tag is pushed before the release is known to be creatable, so a
failure anywhere after that point strands a version number. If the tag push can be moved
after a successful release creation — or made conditional on it — the same class of failure
becomes harmless and re-runnable. That may be the more valuable change; it is larger, so it
is worth deciding deliberately rather than folding in here.

curl -fsS also swallows the response body, which is why the original diagnosis took a log
dig. --write-out '%{http_code}' with the body kept would have said so directly.

### Context `release/action.yml` pushes the new tag and then immediately `POST`s the release that references it. When Forgejo has not finished indexing the tag, that POST returns **500**, `curl -fsS` exits 22, and the job fails — *after* the tag has been pushed. The failure mode is bad out of proportion to the cause. The repository is left with a tag and no release; `publish` and `deploy` are skipped because they need `version`; and the next ordinary dispatch finds the tag already present, decides no release is warranted, and exits 0. Recovering means either publishing the release by hand or force-bumping to a new version, which strands the tagged one forever. It is transient. Observed on `ahmad/imamah` v0.5.0 (its #148); replaying the identical call by hand minutes later returned `201`. The notes had already been generated and were lost with the failed run — the rewrite had succeeded, `notes_fallback=false`. ### Scope - In: making the release POST survive a transient failure. - Out: the notes rewrite; the tagging itself; recovery of releases already stranded. ### Acceptance criteria - [ ] Given the release POST returns 500, when the step runs, then it retries with backoff before failing the job. - [ ] Given a retry then succeeds, when the job finishes, then it is green and `publish` runs as normal. - [ ] Given every retry fails, when the job fails, then the log names the status code and the response body — not only `curl: (22)`. - [ ] Given the POST succeeds first time, when the job runs, then behaviour and timing are unchanged. ### Notes Two things are worth separating. The retry is the fix. But the ordering is what makes the failure expensive: the tag is pushed before the release is known to be creatable, so a failure anywhere after that point strands a version number. If the tag push can be moved after a successful release creation — or made conditional on it — the same class of failure becomes harmless and re-runnable. That may be the more valuable change; it is larger, so it is worth deciding deliberately rather than folding in here. `curl -fsS` also swallows the response body, which is why the original diagnosis took a log dig. `--write-out '%{http_code}'` with the body kept would have said so directly.

ahmad added the

labels

2026-09-04 11:47:57 +00:00

ahmad commented

2026-09-04 13:33:14 +00:00

Author

Owner

A second instance of the ordering problem, from a different cause — which I think settles that the ordering, not the 500, is the thing worth fixing here.

ahmad/imamah release #283 (merge 4ddb512d): version succeeded, tagged v0.5.1 and published the release with proper notes. publish (web) succeeded. publish (api) has now been running 53 minutes and has produced no output for the last 28:

12:39:18Z  job starts
13:04:57Z  #16 [9/9] WORKDIR /usr/src/apps/api   DONE 483.1s
13:04:57Z  #17 exporting to image
13:04:57Z  #17 exporting layers
13:32:37Z  (still nothing)

A WORKDIR instruction taking 483 seconds is the tell: this is host I/O, not the registry and not the network — and the sibling publish (web) completing normally rules out the runner being dead. It belongs with amtronics/infrastructure-vps#390 and #393, and I have not touched it.

Why it matters here. If that job dies, v0.5.1 ends up a tag and a release with a web image and no api image, and deploy never runs — the same stranded shape as v0.5.0, reached by a completely different route. And the recovery is the same dead end: the tag exists, so a re-dispatch derives no release and skips publish, meaning the image can never be built for that version.

That is the argument for reordering rather than only retrying. A retry fixes a transient 500. It does nothing for a build that hangs, a runner that is killed, or any other failure after the tag is pushed — and each of those strands a version number just as thoroughly. Tagging only once the artefacts exist, or making the tag reproducible so a re-dispatch can rebuild for it, closes the whole class.

I am not widening this issue: the retry is still the small, correct fix and should land as scoped. But the note under Notes about ordering being "the more valuable change" now has two independent instances behind it in one day, which is worth knowing when it is prioritised.

A second instance of the *ordering* problem, from a different cause — which I think settles that the ordering, not the 500, is the thing worth fixing here. `ahmad/imamah` release #283 (merge `4ddb512d`): `version` succeeded, tagged `v0.5.1` and published the release with proper notes. `publish (web)` succeeded. `publish (api)` has now been running **53 minutes** and has produced no output for the last **28**: ``` 12:39:18Z job starts 13:04:57Z #16 [9/9] WORKDIR /usr/src/apps/api DONE 483.1s 13:04:57Z #17 exporting to image 13:04:57Z #17 exporting layers 13:32:37Z (still nothing) ``` A `WORKDIR` instruction taking **483 seconds** is the tell: this is host I/O, not the registry and not the network — and the sibling `publish (web)` completing normally rules out the runner being dead. It belongs with `amtronics/infrastructure-vps#390` and `#393`, and I have not touched it. **Why it matters here.** If that job dies, `v0.5.1` ends up a tag and a release with a `web` image and no `api` image, and `deploy` never runs — the same stranded shape as `v0.5.0`, reached by a completely different route. And the recovery is the same dead end: the tag exists, so a re-dispatch derives no release and skips `publish`, meaning the image can never be built for that version. That is the argument for reordering rather than only retrying. A retry fixes a transient 500. It does nothing for a build that hangs, a runner that is killed, or any other failure after the tag is pushed — and each of those strands a version number just as thoroughly. Tagging only once the artefacts exist, or making the tag reproducible so a re-dispatch can rebuild for it, closes the whole class. I am not widening this issue: the retry is still the small, correct fix and should land as scoped. But the note under **Notes** about ordering being "the more valuable change" now has two independent instances behind it in one day, which is worth knowing when it is prioritised.

ahmad self-assigned this

2026-09-04 18:03:42 +00:00

ahmad added

state/in-progress

and removed

state/ready

labels

2026-09-04 18:03:43 +00:00

ahmad referenced this issue from a commit

2026-09-04 18:05:54 +00:00

fix: Retry the release POST instead of stranding the tag

ahmad referenced this issue from a pull request that will close it,