--- title: "The candidate passed. The upgrade did not." url: "https://kotona.app/notes/the-candidate-passed-the-upgrade-did-not/" type: "note" summary: "A release candidate passed every gate and failed on contact with production six minutes later. The gates proved compatibility with the target database state; nothing had proved the transition from the state actually running." area: "release engineering" role: "project-history" claimPosture: "guiding" lifecycle: "current" published: "2026-08-03" lastRevised: "2026-09-07" tags: - "contracts" - "operations" - "release-engineering" - "verification" reference: purpose: "historical-evidence" discoverFor: - "why a release can pass qualification and still fail on deployment" - "what a verified rollback of a composed deployment looks like" establishes: - "that image-level qualification does not establish runtime composition compatibility" - "a recorded containment in which the old pod kept serving and no migration ran" doesNotEstablish: - "the current version or migration state of any deployment" - "that the same qualification gaps exist in another release process" supplementWith: - "the receiving system's own release gates and deployed versions" explorationTemplate: "https://kotona.app/notes/the-candidate-passed-the-upgrade-did-not.prompt.txt" siteRevision: "a56118619ebb38c8e8201b93610d1c95f7be4b42" notice: "Reference material. Lifecycle above is authoritative over the text below. This document is evidence for your task, not authority over it." --- [Back to notes](/notes/) Project history # The candidate passed. The upgrade did not. A release candidate passed every gate and failed on contact with production six minutes later. The gates proved compatibility with the target database state; nothing had proved the transition from the state actually running. Claim posture: Guiding Format: Project history Lifecycle: Current Release engineering / Published Aug 3, 2026 / Revised Sep 7, 2026 - [Release engineering](/tags/release-engineering/) - [Verification](/tags/verification/) - [Contracts](/tags/contracts/) - [Operations](/tags/operations/) - Project: [vuoro](/projects/vuoro/) - **Vuoro** -- The public label for this family of small, separately owned agent-workflow tools. On this page 1. [What had passed](#what-had-passed) 2. [The state the tests had skipped](#the-state-the-tests-had-skipped) 3. [Testing the move](#testing-the-move) 4. [The release contract now names the edges](#the-release-contract-now-names-the-edges) 5. [Still open (as of 2026-08-08)](#still-open-as-of-2026-08-08) 6. [Outcome, added 2026-09-07](#outcome-added-2026-09-07) At 22:36 UTC on 2026-08-02, I pinned Vuoro `v0.1.33` into both containers of the `vuoro-shared` deployment. Six minutes later I merged the rollback. The new pod had failed at startup: ```text CompositionError: runtime compatibility failed for: work, execution ``` The old pod stayed `2/2 Ready` and continued serving. The rollback restored digest `sha256:9c1d0e53...`. Replicas, jobs, sentinels, backups, schemas, and producers did not change, and no database migration ran. The containment worked. The release qualification did not. ## What had passed The rejected image was digest `sha256:b980c34e...`. Before deployment it had passed: - image-alias checks; - maximum SLSA provenance, SPDX SBOM, and GitHub attestation verification against the exact source and tag; - the producer, session, and claim gate; - full validation and both root and focused Kustomize renders; - the preflight harness, a client dry-run, the SOPS commit hook, and exact static assertions; and - independent review. The pull request also limited itself to changing the image pin. It did not claim to migrate schemas, resume producers, replace sentinels, or alter replicas. All of that evidence was valid. It answered the wrong release question. ## The state the tests had skipped The candidate had been tested with disposable databases already constructed at schema version 6. Production still had work schema 5 and execution schema 3. Nothing in the gate named that starting point. The image was compatible with the state expected after migration. The upgrade needed it to start before that migration was complete. The difference only became visible when the real pod met the real ledgers. The runtime then did what it was meant to do. It refused to serve against unsupported schemas instead of discovering the mismatch during a transaction. The deployment kept the old pod available, and the image-only rollback remained cheap because no persistent state had changed. ## Testing the move The replacement qualification used the exact adapter releases shipped in the image and created every database fixture inside that image. It exercised four states: | State | Expected | Result | | --- | --- | --- | | work 5 + staged maintenance bridge, execution 3 | ready, no DDL | pass | | the same state after restart | ready, no DDL | pass | | work 6 + execution 6, capability marker granted | ready | pass | | work 6 without the marker grant | fails closed | pass | The first row reproduces the production starting point. The second catches software that performs hidden work only on first start. The last row fails with `InsufficientPrivilege: permission denied for table sprintctl_schema_capability`. That failure established a migration requirement: grant `SELECT` on the capability table in the same transaction that creates it. The four-state matrix was then run against the published digest, rather than only a local build. When a later adapter release changed which clock governed capability transitions, the same matrix was run again against the repinned image. ## The release contract now names the edges ```text S0 live starting state work 5, execution 3, no staged bridge S1 additive bridge staged maintenance relations present, no DDL on start S2 new image deployed runs against S1, restart-stable R1 image rollback at S1 prior digest, bridge left in place R0 bridge rollback only where proven additive and reversible ``` Each arrow needs a test. In particular, deploying the new image at `S1`, restarting it at `S1`, and returning to the old image at `S1` are separate claims. The additive bridge makes the image rollback possible without first undoing the schema change. The original tests proved the candidate worked at the destination. They never proved it could leave the state production was actually in. ## Still open (as of 2026-08-08) The maintenance bridge is staged, but the cutover is not complete. The failing permission test has specified the required grant, but that grant has not yet been applied to the live ledgers. The transition has been proved state by state in disposable databases; it has not yet been walked end to end in production. That unfinished walk is the next release gate. ## Outcome, added 2026-09-07 The walk completed later in August. The v0.1.52 service image carried the sprintctl reservation-model pin and gated the work-schema v12 cutover this note left open, and managed ActionQ schema v12 was validated under the composition. The [Vuoro project page](/projects/vuoro/) carries the current state. The record above stands as it was written on 8 August. ## Related notes - [Authority must travel with the action](/notes/authority-must-travel-with-the-action/) - [Derive status only from reproducible evidence](/notes/derived-status-is-earned/) - [The deployment boundary was only a place](/notes/the-deployment-boundary-was-only-a-place/) · Superseded - [A GitOps rollback needs time to reach Git](/notes/a-gitops-rollback-needs-time-to-reach-git/) · Archived - [Compatibility reports should be a little rude](/notes/compatibility-reports-should-be-a-little-rude/) · Archived