chore: Publish images from one job in the six multi-image repositories #69
Labels
No labels
priority/P0
priority/P1
priority/P2
priority/P3
size/L
size/M
size/S
size/XL
size/XS
state/blocked
state/done
state/in-progress
state/in-review
state/needs-refinement
state/ready
type/bug
type/chore
type/feature
type/refactor
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Depends on
#66 fix: Parallel image pushes race on shared layers and fail the release
amtronics/platform-actions
Reference
amtronics/platform-actions#69
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Context
#66serialised thepublishjob intemplates/release.ymlso a release cannot race its own image pushes. The template is copied into consumers, so the fix reaches a repository only when its ownrelease.ymlchanges shape.Two images from one release share base layers, and the registry inserts a blob by content hash. Pushing the same layer concurrently makes one insert lose a unique-constraint race (
UQE_package_blob_blake2b) whose error path dereferences nil on the server; buildkit can only reportfailed to push …: unknown, so it reads as flaky CI.The shape is one job with one step per image, not a matrix.
max-parallel: 1and job-levelconcurrencywere both measured on this runner and both are ignored — three legs that should have serialised overlapped either way. Steps within one job are the only ordering this scheduler honours.94b67b7established this in the template; the analysis is in the comments below.Surveyed from the API — every repository the server lists,
release.ymlfetched through the contents API and itspublishjob parsed — rather than from local checkouts, which see only what happens to be cloned here.Repositories to change — six
Each publishes more than one image from a matrix, so their pushes run concurrently today:
ahmad/nameymarketing,portal,apiahmad/imamahapi,webahmad/portfolioapi,webahmad/tripapp,portalBayan/kidpointsapi,webBayan/bayan-webThe matrix currently supplies
matrix.service(or a wholeincludeentry) to the Dockerfile path, both tags and both cache refs; each becomes a literal per step.Bayan/bayan-webandahmad/nameyuseincludewith a per-image package name, so their steps differ by more than a suffix.Bayan/bayan-webpublishes into two separate packages and is still affected — the constraint is on the blob rather than the package, so two images sharing a base layer collide whatever they are named. Worth stating because "different packages, so no shared blob" is the natural and wrong assumption.Not affected
ahmad/larda,ahmad/expiroandsuhba/suhba-webalready build two images from two sequential steps in one job — they are the worked example, not merely unaffected.Bayan/stem,suhba/khittaandsuhba/suhba-docspublish a single image.Scope
publishjob of the six repositories above from a matrix to one job with one step per image, carrying the reasoning across rather than the bare shape; one pull request per repository.max-parallel: 1anywhere — it is inert on this runner.amtronics/infrastructure-vps); the repositories that already push serially; changing what any repository publishes.Acceptance criteria
release.ymlis read, then thepublishjob declares nostrategy/matrixand has onedocker/build-push-actionstep per image.publishjob is compared against the matrix it replaced, then every image is still built from the same context and Dockerfile, with the same tags, build-args and cache refs.Rewriting the body's premise:
max-parallel: 1does not work, so this issue is not "add a line" but "change a shape".#66originally serialised the template withmax-parallel: 1. Review would not approve it because nothing on the fleet used the key and so nothing showed the scheduler honours it. It does not. Measured on a throwaway job, three legs each sleeping 25s:Serialised, three 25-second legs take ~81s; both runs took ~55. Forgejo
16.0.3+gitea-1.22.0ignores both keys. Propagatingmax-parallel: 1to six repositories would have shipped an inert line and left the next collision looking exactly like the last one.What to do instead
Publish every image from one job, one step per image, with no matrix. Steps within a job run in order — that is what a job is, not a scheduler feature that may be unimplemented.
Three repositories already have this shape and are the worked example:
ahmad/larda,ahmad/expiroandsuhba/suhba-webeach build two images from two sequentialdocker/build-push-actionsteps. The first version of this issue listed them as "not affected"; they are better than that — they are the answer.Repositories to change — six
ahmad/nameymarketing,portal,apiahmad/imamahapi,webahmad/portfolioapi,webahmad/tripapp,portalBayan/kidpointsapi,webBayan/bayan-webThe matrix currently supplies
matrix.service(or a wholeincludeentry) to the Dockerfile path, both tags and both cache refs; each becomes a literal per step.Bayan/bayan-webandahmad/nameyuseincludewith a per-image package name, so their steps differ by more than a suffix.Cost, stated plainly
The setup — checkout, Docker CLI install, buildx, two logins — currently runs once per matrix leg in parallel. As one job it runs once, and only the builds are serial. On a runner that was giving these legs two slots at best, that is not obviously slower and may be faster; it is worth timing one release rather than assuming either way.
Acceptance criteria
release.ymlis read, thenpublishhas nostrategy.matrixand one build-and-push step per image.GET /repos/{owner}/{repo}/actions/tasksis read, then there is onepublishrow rather than one per image — which is what makes the ordering unarguable.One warning for whoever counts occurrences
Review found
ahmad/imamahrun 307: two publish legs failing without overlapping, each dead in under ten seconds — the#65shape (a vault fetch failing the job), not this one. A tally of redpublishjobs is not a tally of this bug.#65is merged, so that source should stop producing new cases.chore: Serialise the publish matrix in the six multi-image repositoriesto chore: Publish images from one job in the six multi-image repositoriesBlocking this on
#70— the pull request that establishes the shape it propagates — and recording the dependency rather than leaving it as an ordering I happen to remember.The first version of this issue would have propagated
max-parallel: 1, which is ignored on this runner (measured; see the comment above). Had it gone out before#70was reviewed, six repositories would have gained an inert line and the next collision would have looked exactly like the last. That is not a hypothetical: the issue said to do it and I filed it.So the ordering is load-bearing rather than tidy.
#70's shape — one job, one step per image — is what this copies, and it is not settled until that merges.#70is open, green and awaiting a verdict. When it closes, this returns tostate/readyon its own.Unblocked.
#66is closed — read back fromGET /issues/69/dependencies, which reports itclosed, rather than from the fact that I merged it a minute ago.94b67b7is onmain, so the shape this propagates now exists in the template.Carrying forward the sentence review asked for, because it is the one a quiet week will erase: this fixes a release racing its own pushes, not two repositories racing each other. The second needs the server-side nil dereference in
amtronics/infrastructure-vps. If collisions continue after all six repositories are converted, that is the reason and not a failure of this work.One measurement from the probes that belongs here rather than only on
#66: this runner's capacity appears to be 2. Three 25-second legs took 55 seconds end to end, with legs starting exactly as earlier ones finished, two at a time. So the matrix these six repositories use was never getting more than two concurrent builds anyway — which means converting to sequential steps costs less wall-clock than the diff suggests, and is worth saying to whoever times the first converted release and expects it to double.state/ready.Taking this. One pull request per repository, six in all, each touching
only
.forgejo/workflows/release.yml.Approach. Delete the
strategy/matrixblock and expand the singleBuild and pushstep into one step per image, substituting the matrixvalue into the Dockerfile path, both tags and both cache refs. The job's
setup — checkout, Docker CLI, buildx, both logins — is already shared and
does not move; only the build step multiplies.
94b67b7's comment on thetemplate job travels with it, so each repository carries why it has this
shape rather than a bare diff.
Two need more than a suffix substitution.
Bayan/bayan-webandahmad/nameyuseincludewith a per-image package name, so each stepgets its own
imageliteral.ahmad/tripneeds care of a differentkind: its build-arg is conditional on the matrix value
(
VITE_API_URL=${{ matrix.service == 'portal' && '/api' || '' }}), whichbecomes a literal
/apion the portal step and an empty value on the appstep — the conditional disappears rather than being carried across.
Ambiguity, and how I am reading it. The acceptance criterion "the
pushes did not overlap" cannot be checked from a pull request; it needs a
release to run. I am verifying the pre-merge half — that each converted
job declares no matrix, and that every image keeps the same context,
Dockerfile, tags, build-args and cache refs as the leg it replaced,
compared field by field rather than by reading the diff — and leaving the
timing check to the first release each repository cuts. Whether that is
enough to close the issue is the human's call; I will say so on each PR
rather than claim the criterion is met.
I have also corrected the issue body, which still specified
max-parallel: 1in its scope and first acceptance criterion. Thecomments had rewritten the premise but the body still said to ship the
inert line — and the body is what a reader starts from.
All six pull requests are open, green and ready.
ahmad/imamahahmad/portfolioahmad/tripBayan/kidpointsBayan/bayan-webahmad/nameyEach carries its own equivalence check as a comment:
main's matrixexpanded into its legs, and each leg's
context,file,push,tags,build-args,cache-fromandcache-tocompared against the new steps.All thirteen images across the six repositories are identical to the legs
they replace.
Two carry an edit that is not part of the conversion, flagged on each
so it can be refused separately. Expanding a matrix strands its comments
on whichever image comes first, and in
ahmad/tripandahmad/portfoliothat put a note about the portal/web bundle directly over a build that
ignores every argument it names. Moved to the step it describes, with the
first step gaining a line saying why it still receives arguments it
ignores — because the matrix passed them to every leg, and dropping them
would be a behaviour change this issue does not cover.
The body needed correcting before I could work to it. Its scope and
first acceptance criterion still said to add
max-parallel: 1; thecomments had rewritten the premise but not the part a reader starts from.
Rewritten to the shape actually agreed, with the measurement kept.
One criterion stays unmet by construction. "Given a release, when the
step timings are read, then the pushes did not overlap" needs a release
to run — no pull request can show it. Each PR says so rather than
claiming it. The first release each repository cuts is where to check it,
along with wall-clock: the per-leg setup now runs once, and this runner
was giving the legs about two slots anyway, so it should not be the
doubling the diff suggests.
Done. All six repositories merged.
ahmad/imamah20d02b3ahmad/portfolioc94f7eaahmad/trip94f6106Bayan/kidpoints9a2a271Bayan/bayan-web282def6ahmad/namey0977932Thirteen images across six repositories now build from one step each,
in order, rather than from a matrix that raced its own pushes. Together
with
ahmad/larda,ahmad/expiroandsuhba/suhba-web, which alreadyhad this shape, no repository on the fleet publishes more than one
image concurrently.
Every conversion was checked mechanically rather than by reading the
diff. For each repository,
main's matrix was expanded into its legs bysubstituting the matrix values into the build step, and each leg's
context,file,push,tags,build-args,cache-fromandcache-tocompared field by field against the new steps. All thirteenimages are identical to the legs they replace; each PR carries its own
comparison as a comment.
Two things worth keeping.
The first version of this issue said to add
max-parallel: 1. That key isignored on this runner, measured rather than assumed (#66), so the
issue would have propagated an inert line to six repositories and left the
next collision looking exactly like the last. The body was rewritten
before implementation because its scope and first acceptance criterion
still specified it.
Expanding a matrix silently relocates its comments. In
ahmad/tripandahmad/portfolioa note about the portal and web bundles landed above abuild that ignores every argument it names. Both were moved and flagged on
their PRs as edits beyond the mechanical conversion.
One acceptance criterion is not met and cannot be met here. "Given a
release, when the step timings are read, then the pushes did not overlap."
That needs a release to run; no pull request can show it. The mechanism is
steps-within-a-job, which is the one ordering this runner was measured to
honour — but that is an argument, not an observation of these six
repositories. The first release each cuts is where to confirm it, along
with wall-clock: the setup that ran per matrix leg now runs once, and the
runner was giving these legs about two slots anyway, so it should not be
the doubling the diff suggests.
And the sentence this fixes is narrower than it looks: this removes a
release racing its own pushes. Two repositories releasing in the same
second can still collide, and that needs the server-side nil dereference
in
amtronics/infrastructure-vps. If collisions continue, that is why —not a failure of this work.