fix: Parallel image pushes race on shared layers and fail the release #66

New issue

Closed

opened 2026-09-08 20:41:29 +00:00 by ahmad · 0 comments

ahmad commented

2026-09-08 20:41:29 +00:00

Owner

Context

A release can fail because two of its own jobs push the same layer at the same moment.

ahmad/imamah run 2564 is the example: publish is a matrix over api and web, both pushing to the same package, and the two images share base layers. The api push died with failed to push …: unknown, which is buildkit reporting a server-side failure it could not classify. On the server the same second shows what actually happened:

GetOrInsertBlob() [E] ... duplicate key value violates unique constraint "UQE_package_blob_blake2b"
container/blob.go:75 [E] Error inserting package blob: unique constraint violation
router: failed PUT /v2/ahmad/imamah/blobs/uploads/... panic ... nil pointer dereference

Two identical blobs are inserted concurrently, one loses the unique-constraint race, and the loser's error path dereferences nil. The server bug is tracked separately (amtronics/infrastructure-vps); this issue is about not standing in front of it. The failure is intermittent by nature, which is why it reads as "CI is flaky" rather than as a defect.

The blast radius is every repository that publishes more than one image from one matrix — the copy-in release template, so imamah, larda, expiro and trip at least.

Scope

In: the publish matrix in templates/release.yml, so concurrent pushes of shared layers cannot collide; the same change carried to the consumer repositories that already copied the template.
Out: the Forgejo-side panic; the Docker CLI install step (#59); the notify path (#65).

Acceptance criteria

Given a repository publishing two or more images that share base layers, when a release runs, then no two image pushes upload the same blob concurrently.
Given the same release, then every image it declares is still published under both the version tag and the commit tag.
Given a push that fails for a transient reason, then the failure is retried a bounded number of times before the job fails.
Given the template is changed, then the consumer repositories carrying the old copy are listed on this issue, each with its own follow-up.

Note on the trade-off

Serialising the matrix (max-parallel: 1) is the smallest change and removes the race by construction, at the cost of wall-clock time on multi-image releases. A bounded retry keeps the parallelism and treats the race as the transient it is. Either satisfies the criteria; a retry is the better answer if it can be made to distinguish a transient push failure from a real one, and serialising is the honest fallback if it cannot.

### Context A release can fail because two of its own jobs push the same layer at the same moment. `ahmad/imamah` run 2564 is the example: `publish` is a matrix over `api` and `web`, both pushing to the same package, and the two images share base layers. The api push died with `failed to push …: unknown`, which is buildkit reporting a server-side failure it could not classify. On the server the same second shows what actually happened: ``` GetOrInsertBlob() [E] ... duplicate key value violates unique constraint "UQE_package_blob_blake2b" container/blob.go:75 [E] Error inserting package blob: unique constraint violation router: failed PUT /v2/ahmad/imamah/blobs/uploads/... panic ... nil pointer dereference ``` Two identical blobs are inserted concurrently, one loses the unique-constraint race, and the loser's error path dereferences nil. The server bug is tracked separately (`amtronics/infrastructure-vps`); this issue is about not standing in front of it. The failure is intermittent by nature, which is why it reads as "CI is flaky" rather than as a defect. The blast radius is every repository that publishes more than one image from one matrix — the copy-in release template, so imamah, larda, expiro and trip at least. ### Scope - In: the publish matrix in `templates/release.yml`, so concurrent pushes of shared layers cannot collide; the same change carried to the consumer repositories that already copied the template. - Out: the Forgejo-side panic; the Docker CLI install step (#59); the notify path (#65). ### Acceptance criteria - [ ] Given a repository publishing two or more images that share base layers, when a release runs, then no two image pushes upload the same blob concurrently. - [ ] Given the same release, then every image it declares is still published under both the version tag and the commit tag. - [ ] Given a push that fails for a transient reason, then the failure is retried a bounded number of times before the job fails. - [ ] Given the template is changed, then the consumer repositories carrying the old copy are listed on this issue, each with its own follow-up. ### Note on the trade-off Serialising the matrix (`max-parallel: 1`) is the smallest change and removes the race by construction, at the cost of wall-clock time on multi-image releases. A bounded retry keeps the parallelism and treats the race as the transient it is. Either satisfies the criteria; a retry is the better answer if it can be made to distinguish a transient push failure from a real one, and serialising is the honest fallback if it cannot.

ahmad added the

labels

2026-09-08 20:41:37 +00:00

ahmad referenced this issue

2026-09-08 21:34:49 +00:00

fix: A failed notification must not fail the release #65

ahmad self-assigned this

2026-09-09 06:14:15 +00:00