Container Image Signing & Verification

Overview

Every container image produced by a logiccloud submodule is cosign-signed and SBOM-attested at push time, and every staging or production deploy verifies those signatures before Helm installs or upgrades the release.

Version 1 of this scheme is a CI deploy-gate: signatures are checked in the deployment pipeline immediately before HelmDeploy. There is no cluster-side admission control yet — an image that fails verification is blocked at the pipeline, not at the Kubernetes API server. Admission control is anticipated for v2 (see Future evolution).

Key model & rotation

Signing is key-based cosign. The toolchain is pinned for reproducibility:

Tool	Version	Notes
cosign	2.4.1	signing, attestation, verify
syft	1.18.1	SBOM generation
SBOM	—	CycloneDX JSON

The private key and its passphrase live in the Azure DevOps variable group logiccloud-signing, as the secrets logiccloud-cosign-private-key and logiccloud-cosign-passphrase.

The public key is committed at ci/security/cosign.pub in the shared ci repository and distributed to all consumers via the @ci repo ref — there are no per-service copies to keep in sync.

Rotation

1. Re-run ci/security/cosign-bootstrap.sh on a release-engineer workstation to generate a fresh key pair.
2. Update the logiccloud-signing variable group with the new private key and passphrase.
3. Commit the new ci/security/cosign.pub.

Old images intentionally fail verification until they are re-signed with the new key — this is the expected behaviour of a key rotation, not a regression.

The env-var interface (COSIGN_PRIVATE_KEY / COSIGN_PASSWORD) is kept stable to allow a future swap to keyless OIDC without touching the calling pipelines.

Signing flow (per service)

ci/steps/registry/publish-container.yaml invokes ci/steps/security/sign-and-attest.yaml after the Docker push. That step:

1. Resolves the immutable image digest (cosign triangulate → crane digest → docker inspect fallback).
2. Generates a CycloneDX SBOM via syft.
3. Runs cosign sign and cosign attest --type cyclonedx against the digest (never a mutable tag).

The step soft-skips — it warns and exits 0 — when COSIGN_PRIVATE_KEY is empty, so feature and PR builds that have no access to the signing secrets stay green.

Verification flow (per env)

logiccloud/make-pipeline.js emits a VerifyImageSignatures step (ci/steps/security/verify-cosign.yaml@ci) before every HelmDeploy@0. It verifies every image listed in the environment's *-versions.yaml against ci/security/cosign.pub.

Environment tier	Envs	Behaviour
Dev / staging	staging2	softFail: true during the rollout window
Production	app-eu2, kaiserworks, consol, elektror, smartpackaging	softFail: false — hard-fail from day one

A failed verification blocks the deploy with exit code 2.

3rd-party mirror signing scope

3rd-party images mirrored into the dev and staging ACRs are cosign-signed at mirror time by logiccloud/clusters/{dev,staging}/terraform/scripts/mirror-image.sh. The signing key is passed via the acr-mirror.tf local-exec provisioner, sourced from the logiccloud-vault Key Vault.

Production-cluster mirror signing is deferred to a v2 Feature.

Docker Hub mirror

The go-logiccloud-control Docker Hub publish stage uses an explicit cosign copy to carry the signatures and SBOM attestations across to docker.io. Cross-registry signature artifacts do not travel automatically, so this explicit copy is required.

Emergency bypass

Emergency use only

The verify template honours a COSIGN_VERIFY_DISABLED=true pipeline variable that skips signature verification. It exists only for the narrow window of an emergency hotfix.

Using it leaves an audit trail in the build log and must be reverted immediately after the hotfix. Never leave verification disabled as a standing configuration.

Future evolution

Two evolutions are anticipated for v2, neither of which changes the stable env-var wrapper interface:

• Keyless OIDC signing via Azure Workload Identity Federation (replacing the long-lived private key).
• Cluster-side admission control (Kyverno, Connaisseur, or the Sigstore Policy Controller) to enforce signatures at the Kubernetes API server rather than only in CI.

References

• ADR-0049-1: Container image signing model and key lifecycle — LC-A-1467742107
• Parent Feature LC-2387 — LC-2387