One TCF Canada backend deployment had already finished.

Railway reported:

status = SUCCESS

Backend readiness was healthy too:

status = ok
database = reachable

Even the CLI deployment message contained the commit SHA I expected.

By the usual signals, the release could continue. I stopped the pipeline anyway. I did not deploy the eval worker or publish the Frontend.

Railway had returned this:

metadata.commitHash = null

I could prove that some version was healthy. I could not prove that it was the version that had just passed CI.

Green CI does not identify production

Many pipelines finish with four checks:

  • CI passed.
  • The provider reports SUCCESS.
  • The health endpoint returns 200.
  • The website opens.

All four matter, but none answers the question I needed: which commit is running in production?

Suppose CI passes, then main receives another commit. If the deploy script checks out the latest main again, the result can be:

CI verified A
Production deployed B

Both systems show green. Commit B never passed that CI run.

What exact-SHA means

I require the whole release chain to agree on one SHA:

CI SHA
  = Deployment source SHA
  = Railway API SHA
  = Railway Worker SHA
  = Cloudflare marker SHA
  = Release receipt SHA

The SpeakUp deployment workflow does not guess the latest main at publish time. It takes the workflow_run.head_sha that triggered the verified CI run, checks out that exact SHA, and deploys it.

The commit that passed CI is the commit that gets deployed.

A health check proves health, not identity

This was the TCF Canada state:

Railway status       = SUCCESS
Backend /ready       = ok
Database             = reachable
CLI message          = expected SHA present
metadata.commitHash  = null

The first four fields showed that the service could run. They did not replace the missing version identity.

The verifier rejected the deployment and stopped at:

provider-verification blocked

I did not redeploy the Backend, deploy the eval worker early, accept the CLI message as a substitute for commit metadata, or call the release complete because health looked good.

The release did not finish, but the pipeline did its job: it stopped when the evidence was incomplete.

API and Worker can run in parallel, but not every time

Railway API and Worker builds are expensive, so parallel deployment looks like the obvious optimization.

The first move from an embedded worker to an external worker cannot use that order:

Adjust capacity
   ↓
Worker verified
   ↓
Switch API to external mode

If the API switches first and the Worker is not ready, background work may have nowhere safe to go.

Once the system is already stable on the external topology, later replacements can run in parallel:

API build    ─┐
              ├→ Paired readback
Worker build ─┘

Both identities must verify before Cloudflare publication continues. A failed side cannot produce a successful release receipt.

Unchanged runtime should not rebuild every service

I later split version identity into two digests:

Runtime digest
- Backend source
- Frontend source
- Schema
- Public runtime configuration

Control digest
- GitHub Actions workflow
- Publisher scripts
- Deployment verifier
- Deployment CLI

If only the Publisher or verifier changes, the runtime digest stays the same. Railway and Cloudflare do not need another runtime build.

The pipeline still rechecks the Railway API, Worker, database readiness, Cloudflare marker, Frontend asset hash, production URLs, and release receipt.

One control-only run completed strict readback in 1 minute 32 seconds without republishing the runtime.

That is not a 13-minute deployment reduced to one minute. The accurate claim is narrower: with no runtime change, the pipeline revalidated the existing deployment in 1 minute 32 seconds.

Why I keep a release receipt

Release evidence used to be spread across GitHub Actions, Railway, Cloudflare, a health endpoint, database readiness, and the Frontend marker.

A release receipt binds those facts together:

source SHA
runtime digest
control digest
API deployment
Worker deployment
database readiness
frontend marker
asset hashes
publication result
verification timestamps

Only after both provider identities, both health states, and the production readback agree does the receipt record:

paired_verified

It gives the next person concrete answers: which commit was released, which services rebuilt, which runtime was reused, which checks ran, and which boundaries were still unverified.

The release states I use now

I no longer reduce every release to done or not done. I track:

Implemented
Tested
Merged
CI green
Deploying
Provider verified
Live readback verified
Customer flow verified

The distinctions matter:

CI green        ≠ deployed
Merged          ≠ live
Railway success ≠ exact-SHA verified
Health 200      ≠ database ready
Website opens   ≠ correct Frontend assets
Deploy success  ≠ customer flow verified

What I kept from this incident

The first article was about speed and cost. This one is about release correctness.

A reliable pipeline avoids rebuilding work that did not change, but it does not trade identity checks for speed.

Its most useful behavior is knowing when the available evidence is not enough to continue.

Back to the first article: One Documentation Change, One Full CI Run: How I Burned 2,000 Minutes

I also turned the evidence rules behind these articles into a reusable public resource: Evidence-first CI/CD writing skill