What is actually running?

09.09.20263 min readAtalaia team

During an incident, someone always asks the same question: what version is actually running? The answer usually starts with a tag, such as api:1.8.2, followed by a long pause while people work out whether that tag was rebuilt, which commit it came from and whether the fix that merged yesterday is in it.

It does not have to be a pause. With a few conventions in the build, any running container can be traced back to the commit, the build run and the pull request that produced it.

Tags describe intent, not reality

A tag is a pointer. It can be pushed again, overwritten by a rebuild or shared by several images over time. Two clusters both running api:1.8.2 may be running different code. A digest, the sha256: hash of the image manifest, names one exact image and nothing else.

Kubernetes records the resolved digest even when the manifest used a tag, so you can always see what was really pulled:

kubectl get pods -n prod \
  -o custom-columns=POD:.metadata.name,IMAGE:.status.containerStatuses[0].imageID

The trace, step by step

1. PodRead the image digest from the container status.
2. ImageRead the OCI labels or provenance attached to that digest.
3. CommitThe revision label gives the exact commit in the source repository.
4. BuildThe commit leads to the CI run that built and pushed the image.
5. Pull requestThe commit leads to the reviewed change that merged it.

Each link is cheap to add. The trail breaks only when one of them is missing.

The same trace works in the other direction, which is often the more useful question. When a fix merges, you want to know which environments are running it yet. Starting from the commit, find the digest the build produced, then search the clusters for pods running that digest. If the answer is none, the fix has not shipped, whatever the release notes say. Teams that can answer both directions quickly tend to find that incident calls get shorter and rollbacks get less frightening.

Put the source in the image

The OCI image spec defines standard annotation keys for exactly this. Set them at build time:

docker build \
  --label org.opencontainers.image.source=https://github.com/acme/api \
  --label org.opencontainers.image.revision=$(git rev-parse HEAD) \
  --label org.opencontainers.image.created=$(date -u +%Y-%m-%dT%H:%M:%SZ) \
  -t ghcr.io/acme/api:1.8.2 .

In GitHub Actions, the docker/metadata-action generates these labels for you. Reading them back from any digest takes one command:

crane config ghcr.io/acme/api@sha256:<digest> | jq '.config.Labels'

Labels are easy, but anyone who can build an image can set them. Treat them as a helpful pointer, and use signed provenance when the answer has to stand up to scrutiny.

Make the trail trustworthy

Build provenance is a signed statement from the CI platform saying which repository, commit and workflow produced a digest. Unlike a label, it cannot be written by the build steps themselves, so it holds even if someone tries to pass off an image built elsewhere.

gh attestation verify oci://ghcr.io/acme/api@sha256:<digest> --repo acme/api

# from the commit, find the build run and the pull request
gh run list --repo acme/api --commit <commit-sha>
gh api repos/acme/api/commits/<commit-sha>/pulls --jq '.[].html_url'

What breaks the trail

• Images built on laptops. No CI run, no provenance, often no labels. Block pushes to production registries from anything but CI.
• Retagging between environments. Promoting by rebuilding creates a new digest. Promote the same digest instead.
• Squash merges without links. The commit on main must still point to its pull request. Most hosts handle this, but check your merge settings.
• Base image drift. Your commit did not change, but the base did. Pin base images by digest and let the update bot raise the change.

Check yours in 10 minutes

1. Pick one production service and list its running image digests with the kubectl command above.
2. Run crane config on one digest. Is there a revision label?
3. Follow that commit to a build run and a pull request. Time how long it takes.
4. Check whether your deploy manifests use tags or digests.
5. Check whether anyone outside CI can push to the production registry.

If the trace takes more than a few minutes, or stops at step two, you know which link to add first.

Where this sits on the line

In the tower (in development):