fix: Validate the release-notes rewrite before publishing it #79

New issue

Closed

opened 2026-09-09 18:22:17 +00:00 by ahmad · 0 comments

ahmad commented

2026-09-09 18:22:17 +00:00

Owner

Context

ahmad/imamah v0.7.1 was published with these release notes, verbatim and public:

I don't see any commit subjects in your message. Could you please provide the list of commit subjects you'd like me to rewrite as release notes?

The release step accepts whatever the summariser returns as long as it is not an empty string (release/action.yml, the if [ -n "${SUMMARY}" ] guard). A model that answers with a question, an apology, or a refusal passes that check and is published to users as the description of the release.

Two faults meet here:

The rewrite is called with nothing to rewrite. v0.7.1 came from a manual dispatch with force-bump: patch — the documented rebake path — so the commit range was empty. The prompt was sent with an empty commit list, and the model reasonably asked what it was supposed to summarise.
Nothing checks the answer. Non-empty is treated as valid. The action already has a correct fallback to commit-derived notes and a notes-fallback output to flag it; the fallback simply never fired, because the failure did not look like a failure.

Scope

In: not calling the summariser when there is nothing to summarise; validating the summariser's answer against the shape the prompt asks for before publishing it; using the existing commit-based fallback when validation fails; a release with no commits getting honest notes instead.
Out: the choice of model or provider; the notification path (#65); the wording of the prompt beyond what validation requires.

Acceptance criteria

Given a release whose commit range is empty, when the release is published, then the summariser is not called and the notes state plainly that nothing changed.
Given a summariser answer that carries none of the expected headings or bullets, when the release is published, then the commit-derived notes are used instead and notes-fallback is true.
Given a summariser answer that is a question, an apology, a refusal or an explanation of what it cannot do, then it is treated as a failed rewrite, not as notes.
Given a valid summary, then it is published unchanged — validation must not rewrite or truncate a good answer.
Given any fallback, then the job log says which rule rejected the answer, so a bad prompt is diagnosable without re-running it.
Given the repository's AGENTS.md, then it carries the rule that model-generated text is never published without a shape check and a deterministic fallback.

### Context `ahmad/imamah` v0.7.1 was published with these release notes, verbatim and public: > I don't see any commit subjects in your message. Could you please provide the list of commit subjects you'd like me to rewrite as release notes? The release step accepts whatever the summariser returns as long as it is not an empty string (`release/action.yml`, the `if [ -n "${SUMMARY}" ]` guard). A model that answers with a question, an apology, or a refusal passes that check and is published to users as the description of the release. Two faults meet here: 1. **The rewrite is called with nothing to rewrite.** v0.7.1 came from a manual dispatch with `force-bump: patch` — the documented rebake path — so the commit range was empty. The prompt was sent with an empty commit list, and the model reasonably asked what it was supposed to summarise. 2. **Nothing checks the answer.** Non-empty is treated as valid. The action already has a correct fallback to commit-derived notes and a `notes-fallback` output to flag it; the fallback simply never fired, because the failure did not look like a failure. ### Scope - In: not calling the summariser when there is nothing to summarise; validating the summariser's answer against the shape the prompt asks for before publishing it; using the existing commit-based fallback when validation fails; a release with no commits getting honest notes instead. - Out: the choice of model or provider; the notification path (`#65`); the wording of the prompt beyond what validation requires. ### Acceptance criteria - [ ] Given a release whose commit range is empty, when the release is published, then the summariser is not called and the notes state plainly that nothing changed. - [ ] Given a summariser answer that carries none of the expected headings or bullets, when the release is published, then the commit-derived notes are used instead and `notes-fallback` is `true`. - [ ] Given a summariser answer that is a question, an apology, a refusal or an explanation of what it cannot do, then it is treated as a failed rewrite, not as notes. - [ ] Given a valid summary, then it is published unchanged — validation must not rewrite or truncate a good answer. - [ ] Given any fallback, then the job log says which rule rejected the answer, so a bad prompt is diagnosable without re-running it. - [ ] Given the repository's `AGENTS.md`, then it carries the rule that model-generated text is never published without a shape check and a deterministic fallback.

ahmad added the

labels

2026-09-09 18:22:26 +00:00

ahmad self-assigned this

2026-09-09 18:22:26 +00:00

ahmad referenced this issue from a pull request that will close it,

2026-09-09 18:27:24 +00:00

fix: Validate the release-notes rewrite before publishing #80