Chapter 25 — CI/CD Pipeline Design and Quality Gates for Sovereign Cloud

Summary

This chapter defines the architecture of CI/CD pipelines purpose-built for sovereign cloud estates, where every stage from source checkout through deployment verification must satisfy jurisdictional residency, provenance, and policy constraints. It examines supply chain security through the SLSA framework, in-toto attestations, and Sigstore signing; multi-stage quality gates that enforce progressively stricter criteria at each promotion boundary; and the composition of SAST, DAST, SCA, container scanning, and IaC scanning into a unified security evidence record. The chapter establishes that the pipeline itself is a sovereign system whose build infrastructure, artefact registries, execution environments, and secrets management must reside within the sovereign boundary. Architects will find practical guidance on agent-enhanced pipeline intelligence, override governance for emergency changes, and pipeline observability grounded in OpenTelemetry instrumentation and DORA metrics.

25.1 Anatomy of a sovereign CI/CD pipeline

A CI/CD pipeline in its most elementary form is a sequence of automated stages that carry a code change from a developer’s workstation into a running production environment. The stages are familiar to any practitioner: source checkout, build, test, scan, gate, stage, deploy, verify. What makes a pipeline sovereign is not the addition of new stages—though some stages do expand—but the set of constraints that every stage must satisfy, the provenance guarantees that must be maintained end to end, and the architectural decisions about where the pipeline itself executes and where its artefacts reside.

The source stage begins with a commit to a version-controlled repository. In sovereign estates, the repository hosting platform is itself a sovereignty consideration. Source code for workloads destined for a regulated sovereign zone should reside in a repository whose data-at-rest storage respects the zone’s residency requirements. GitHub Enterprise Server, GitLab Self-Managed, or Gitea instances deployed within the zone satisfy this constraint; SaaS-hosted repositories may not, depending on the jurisdiction and the provider’s data residency commitments. The source stage also establishes the first link in the provenance chain: the commit hash, the author identity (authenticated through the federated identity stack described in Chapter 13), the branch, and the timestamp together form the origin record from which all downstream attestations derive.

The build stage compiles source into deployable artefacts—container images, binary packages, Helm charts, Terraform modules. For sovereignty, the build environment matters as much as the build output. A container image built on infrastructure outside the sovereign boundary may be technically identical to one built inside it, but the provenance record will show that the build occurred in an uncontrolled jurisdiction, which may be unacceptable under regulatory frameworks that impose requirements on the processing environment for sensitive workloads. Build infrastructure residency is discussed in detail in section 25.5.

The test stage executes the automated verification suites that provide evidence of functional correctness and regression safety. In a sovereign pipeline, the test stage is typically decomposed into multiple sub-stages—unit tests, integration tests, contract tests, end-to-end tests—each contributing results to the quality evidence record. Test results are published to IBM DevOps Insights in JUnit XML format, where they become inputs to quality gate evaluation and DORA metric calculation as described in Chapter 16 [1].

The scan stage applies security and compliance analysis tools to the build artefacts. Static application security testing (SAST), software composition analysis (SCA), container image scanning, infrastructure-as-code scanning, and licence compliance scanning each produce structured findings that are posted to DevOps Insights as quality indicators. The scan stage is where the pipeline intersects most directly with supply chain security: the scans answer questions about what is inside the artefact, whether any of it is known to be vulnerable, and whether any of it carries licence terms incompatible with the deployment context. Section 25.4 examines the scan stage in depth.

The gate stage is the decision point. A quality gate evaluates the accumulated evidence—test results, scan findings, provenance attestations, sovereignty constraint checks—against a policy that defines the conditions for promotion. If the conditions are met, the pipeline proceeds. If they are not, it halts. Quality gates are positioned at every promotion boundary: from build to staging, from staging to pre-production, from pre-production to production. Each boundary may enforce a different policy, with progressively stricter thresholds as the artefact approaches the sovereign production environment. Section 25.3 describes multi-stage gate design.

The staging and deployment stages place the artefact into progressively more production-like environments. In sovereign estates, each environment resides within a defined zone, and the promotion path must respect zone boundaries. An artefact may be tested in a non-sovereign development environment but must pass through a sovereignty gate before entering a sovereign staging environment, at which point it is subject to the full set of sovereign constraints: registry residency, image signing verification, OPA admission control, and network policy enforcement.

The verification stage confirms that the deployed artefact is operating correctly in its target environment. Smoke tests, synthetic transaction monitors, canary analysis, and SLO evaluation against short-window baselines all contribute to post-deployment verification. The verification results are fed back to DevOps Insights as deployment records, closing the loop that enables Concert to calculate change risk scores as described in Chapter 16 [1].

End-to-end sovereign CI/CD pipeline

Tekton, the Kubernetes-native pipeline framework that underpins Red Hat OpenShift Pipelines, is the pipeline execution engine most naturally aligned with sovereign cloud operations [2]. Its architecture runs pipeline tasks as Kubernetes pods, which means that pipeline execution inherits the same isolation, scheduling, and policy enforcement mechanisms that govern application workloads. A Tekton pipeline running on a sovereign OpenShift cluster executes within the sovereign boundary by construction: the compute, storage, and network resources consumed by pipeline tasks are all within the cluster’s jurisdiction. This architectural property is not available from CI/CD platforms that execute pipelines on shared, multi-tenant infrastructure outside the operator’s control.

Executive Perspective

The CI/CD pipeline is the last line of automated defence before software reaches production, and in a sovereign estate it carries an additional burden: it must guarantee that every artefact deployed into a regulated zone was built, scanned, signed and promoted within that zone’s trust boundary. For the CIO, this is a direct translation of supply chain risk into engineering process. The SolarWinds compromise demonstrated that a single weakness in the build chain can propagate to thousands of downstream organisations; sovereign pipeline design ensures that the organisation’s own build infrastructure, artefact registries and signing keys are under its jurisdictional control, not delegated to shared multi-tenant platforms with opaque residency guarantees. The business value is both protective and enabling: protective because quality gates and provenance verification catch vulnerabilities and policy violations before they reach production, and enabling because a well-governed pipeline gives the organisation confidence to deploy more frequently, knowing that each release carries machine-verified evidence of its integrity and compliance.

25.2 Build provenance and supply chain security

The software supply chain has become one of the most consequential attack surfaces in modern computing. The SolarWinds compromise of 2020, in which adversaries injected malicious code into a build system and distributed it to thousands of organisations through a trusted update channel, demonstrated that the integrity of the build process is as important as the integrity of the source code [3]. For sovereign cloud estates, where the consequences of a supply chain compromise may include regulatory breach, loss of operational sovereignty, and exposure of data subject to jurisdictional controls, supply chain security is not an optional enhancement but a foundational requirement.

The Supply-chain Levels for Software Artifacts (SLSA) framework, maintained by the Open Source Security Foundation, provides a graduated maturity model for supply chain integrity [4]. SLSA defines four levels of assurance, each building on the previous.

At SLSA Level 1, the build process must produce provenance metadata—a structured document that records what was built, from which source, by which build system, and at what time. The provenance document is a statement of fact about the build, not a guarantee of integrity; it may be generated by the same build system it describes, which means it is only as trustworthy as the build system itself. Level 1 is nonetheless valuable because it establishes the practice of generating provenance and makes it machine-readable.

At SLSA Level 2, the provenance must be generated by a hosted build service rather than by developer workstations, and it must be signed by the build service’s identity. This provides a degree of tamper resistance: an attacker who compromises a developer’s machine cannot forge provenance that claims the build was produced by the hosted service. The signature creates a verifiable link between the provenance document and the build platform.

At SLSA Level 3, the build platform must provide additional isolation guarantees: builds must run in ephemeral, isolated environments that are provisioned fresh for each build and destroyed afterwards, preventing one build from influencing another. The provenance must be generated by the build platform itself, not by user-controlled build scripts, and must be non-falsifiable by the build’s owner. This level addresses the insider threat: even a developer with commit access to the source repository cannot produce provenance that misrepresents the build’s origin.

At SLSA Level 4, the build must be reproducible—given the same source inputs, the build process must produce bit-for-bit identical outputs—and the provenance must include a complete, verified description of the build environment. Reproducible builds allow independent parties to verify that a given artefact was produced from a given source, without trusting the original build system at all. This is the highest level of supply chain assurance currently defined.

For sovereign cloud operations, SLSA Level 3 should be treated as the target for production workloads in regulated sovereign zones. The isolation and non-falsifiability properties at Level 3 align directly with regulatory expectations around the integrity of software deployed in critical environments. SLSA Level 4’s reproducibility requirement, while aspirational, is not yet practically achievable for many technology stacks and should be pursued incrementally.

Build attestation is the mechanism by which provenance claims are made concrete and verifiable. An attestation is a signed statement about a software artefact, structured according to the in-toto attestation framework [5]. The statement includes a predicate—the claim being made—and a signature—the cryptographic proof that the claim was made by the stated authority. Common predicates include provenance (where and how the artefact was built), vulnerability scan results (what vulnerabilities were found), and SBOM (what components the artefact contains).

Sigstore and its cosign tool provide the signing infrastructure that makes attestation practical at scale [6]. Traditional code signing requires each signing authority to manage its own key material—generating keys, protecting private keys, distributing public keys, managing revocation. Sigstore replaces this with a keyless signing model based on short-lived certificates issued by a certificate authority (Fulcio) that verifies the signer’s identity through OIDC federation. The signature is recorded in a transparency log (Rekor) that provides a tamper-evident, publicly auditable record of every signing event. The signer does not need to manage long-lived keys, and the verifier can confirm both the identity of the signer and the time of signing by querying the transparency log.

In a sovereign pipeline, cosign is used to sign container images at the build stage. The signing identity is the build system’s OIDC-verified identity—the Tekton service account, for example—and the signature is recorded in the transparency log. At the deployment stage, the Kubernetes admission controller verifies the signature before admitting the image to the cluster. An image that lacks a valid signature from the build system, or whose signature was produced by a build system outside the sovereign boundary, is rejected. This enforcement is implemented through policy-as-code: an OPA policy or a Kyverno policy that evaluates the image’s signature against a list of trusted signing identities and blocks admission if verification fails.

Container image signing with cosign integrates naturally with OpenShift’s image signature verification capabilities and with the broader policy-as-code ecosystem described in Chapter 10 and Chapter 11. The combination of SLSA provenance, in-toto attestations, and Sigstore signing creates a chain of evidence that traces an artefact from source commit to production deployment, with cryptographic verification at each link. For regulated sovereign zones, this chain of evidence is both a technical control and a regulatory asset: it provides the documented, verifiable record of software provenance that supervisory authorities increasingly expect.

Supply chain security stack

25.3 Multi-stage quality gates

The quality gate, as introduced in Chapter 16, is the mechanism by which accumulated evidence is converted into a promotion decision. A single quality gate is useful; a system of gates, positioned at every promotion boundary and composed to enforce progressively stricter criteria, is what transforms a pipeline from a conveyance into a governance mechanism.

The design principle is straightforward: each gate evaluates a specific class of evidence, and the gates at later promotion boundaries require all the evidence demanded by earlier gates plus additional evidence appropriate to the environment being entered. The result is a layered defence in which no single gate failure is catastrophic—earlier gates catch most problems—but the final gates before production admission are comprehensive.

The unit test gate is the first gate, positioned immediately after the build and unit test stages. Its criteria are deliberately lightweight: a minimum test pass rate (typically 100 per cent for unit tests, since a failing unit test indicates a regression that should never be promoted), a minimum code coverage threshold (the specific percentage varies by organisation, but 80 per cent is a common baseline), and the presence of test results in the expected format. The unit test gate exists to catch obvious regressions early, not to provide comprehensive quality assurance.

The integration test gate is positioned after the integration and contract test stages. Its criteria extend the unit test gate’s requirements: all unit test criteria must still be satisfied, plus integration test results must be present and must meet a minimum pass rate. Integration tests exercise the interactions between components and are more likely to reveal problems that unit tests miss—incorrect API contracts, incompatible data formats, race conditions in asynchronous workflows. The integration test gate may also require that contract test results from consumer-driven contract testing frameworks (such as Pact) are present and passing, ensuring that the component under test is compatible with its consumers’ expectations.

The security scan gate is positioned after the security scanning stage and evaluates vulnerability findings from SAST, SCA, container scanning, and IaC scanning tools. Its criteria are expressed in terms of finding severity and count: no critical vulnerabilities, no more than a configurable number of high-severity vulnerabilities, and all findings must have been triaged (acknowledged, accepted with justification, or remediated). The security scan gate is where supply chain security controls intersect with quality governance: SCA findings that identify vulnerable dependencies, container scan findings that identify vulnerable base image layers, and IaC scan findings that identify insecure infrastructure patterns are all evaluated at this gate.

The compliance gate evaluates regulatory and organisational compliance evidence. Its criteria include the presence and completeness of an SBOM in CycloneDX or SPDX format [7][8], licence compliance scan results confirming that no component carries licence terms incompatible with the deployment context, and evidence that the artefact has been built in accordance with the organisation’s secure development lifecycle requirements. The compliance gate may also evaluate audit trail completeness: whether the pipeline has produced a continuous, unbroken chain of evidence from source commit to the current stage.

The sovereignty gate is specific to sovereign pipelines and has no equivalent in conventional CI/CD design. It evaluates whether the artefact satisfies the constraints of the sovereign zone it is being promoted into. Its criteria include: the artefact was built on infrastructure within the sovereign boundary (verified through build provenance attestation), the artefact’s container image was pulled from a registry within the sovereign boundary, the artefact’s signatures were produced by a signing identity authorised for the target zone, and the artefact’s deployment manifest targets only resources within the permitted zone. The sovereignty gate is the pipeline’s counterpart to the OPA admission control that operates at the cluster level; it catches sovereignty violations before the deployment is attempted, providing earlier feedback and avoiding the operational disruption of a failed admission.

Gate composition is the mechanism by which these individual gates are assembled into a promotion policy. A promotion from development to staging might require passing the unit test gate and the security scan gate. A promotion from staging to pre-production might require all of those plus the integration test gate and the compliance gate. A promotion from pre-production to sovereign production requires all gates, including the sovereignty gate. The composition is expressed as a policy document—a DevOps Insights policy definition or an OPA policy—that lists the gates required at each boundary and the minimum criteria for each.

Override governance addresses the reality that quality gates must occasionally be bypassed. An emergency security patch, a critical production fix, or a regulatory-mandated change may need to reach production before all normal quality criteria can be satisfied. The question is not whether overrides should be permitted—they must be, because the alternative is a pipeline that blocks critical fixes—but how they are governed.

Override governance requires four elements. First, an override must be requested by a named individual with explicit authority to request overrides, not by the pipeline automation itself. Second, the override must be approved by a second individual with override approval authority—separation of duties prevents a single actor from both requesting and approving. Third, the override must be recorded in the quality evidence trail with the identity of the requester, the identity of the approver, the justification, and the specific gate criteria that were bypassed. Fourth, the override must trigger a time-bounded remediation obligation: the bypassed criteria must be satisfied within a defined period (typically 72 hours for critical fixes), and the pipeline must track and enforce this obligation. DevOps Insights records override events as quality indicators, making them visible to Concert’s change risk scoring and to audit review [1].

Multi-stage quality gate architecture

25.4 Security scanning in the sovereign pipeline

Security scanning in a sovereign pipeline serves two purposes simultaneously: it protects the estate from vulnerabilities and malicious code, and it generates the evidence of due diligence that regulatory frameworks require. The scan stage is not a single tool but a composition of specialised scanners, each addressing a different dimension of security risk.

Static Application Security Testing (SAST) analyses source code or compiled bytecode for security vulnerabilities without executing the application. SAST tools identify common vulnerability patterns—SQL injection, cross-site scripting, insecure deserialisation, hard-coded credentials—by analysing code structure, data flow, and control flow. SAST is most effective when integrated early in the pipeline, ideally as a pre-commit hook or a pull request check, so that developers receive feedback before their code is merged. In the pipeline itself, SAST runs after the source checkout stage and produces findings in SARIF (Static Analysis Results Interchange Format), which DevOps Insights can ingest as quality indicators.

Dynamic Application Security Testing (DAST) analyses running applications by sending crafted requests and observing the responses. Where SAST analyses code, DAST analyses behaviour: it discovers vulnerabilities that are only exploitable at runtime, such as authentication bypass, session management flaws, and server misconfiguration. DAST runs later in the pipeline, after the artefact has been deployed to a test environment, because it requires a running instance to test against. In sovereign pipelines, the DAST environment must itself reside within the sovereign boundary if the test data includes regulated information—a constraint that is easy to overlook when DAST is treated as a generic pipeline stage rather than a sovereignty-aware one.

Software Composition Analysis (SCA) examines the third-party libraries, frameworks, and components included in the artefact. SCA tools maintain databases of known vulnerabilities and licence terms for open-source components, and they compare the artefact’s dependency tree against those databases. The output is twofold: a vulnerability report listing components with known CVEs, and a licence report listing the licence terms associated with each component. Both are essential for sovereign pipelines. The vulnerability report feeds the security scan gate; the licence report feeds the compliance gate. SCA tools also generate SBOMs—or validate SBOMs generated by the build system—ensuring that the artefact’s component inventory is accurate and complete.

Container image scanning is a specialised form of SCA that analyses the layers of a container image. A container image typically includes an operating system base layer, system libraries, runtime environments, and application code. Each layer may contain vulnerable packages. Container scanners decompose the image into its layers, enumerate the packages in each layer, and check them against vulnerability databases. For sovereign pipelines, container scanning has an additional function: verifying that the base image originates from an approved registry within the sovereign boundary and that the image’s signature chain is intact. An image built from a base image pulled from an untrusted registry outside the sovereign zone is a supply chain risk regardless of whether the image itself contains known vulnerabilities.

Infrastructure-as-Code (IaC) scanning applies security analysis to Terraform, CloudFormation, Kubernetes manifests, Helm charts, and other infrastructure definitions. Tools such as Checkov and tfsec parse IaC files and evaluate them against libraries of security rules [9]. Common findings include overly permissive security groups, unencrypted storage resources, missing logging configurations, and IAM policies that violate least-privilege principles. IaC scanning is particularly important in sovereign pipelines because infrastructure misconfigurations can undermine sovereignty controls: a security group that allows unrestricted inbound traffic, a storage bucket without server-side encryption, or an IAM role that trusts principals from outside the sovereign zone are all sovereignty violations that IaC scanning can detect before the infrastructure is provisioned.

The integration of scan results with DevOps Insights is what elevates scanning from a development activity to an operational governance mechanism. Each scanner produces structured findings—vulnerability identifiers, severity ratings, affected components, remediation guidance—that are posted to DevOps Insights as quality indicators. DevOps Insights aggregates these findings across scanners, deduplicates where multiple scanners identify the same issue, and presents a unified security posture for each build. Quality gates evaluate the aggregated findings against the promotion policy: if the total critical and high findings exceed the threshold, the gate blocks promotion. Concert consumes the aggregated scan results through the DevOps Insights REST API and incorporates them into its change risk scoring, so that a deployment carrying a high number of unresolved findings receives an elevated risk score even if the quality gate was overridden [1].

Vulnerability management workflows close the loop between scanning and remediation. A finding that is identified by a scanner but never acted upon provides no security value. The vulnerability management workflow defines how findings move through a lifecycle: from initial identification, through triage (where an engineer assesses severity and exploitability in the specific deployment context), to remediation (where the vulnerable component is updated or replaced), or to acceptance (where the finding is documented as an accepted risk with justification, approval, and a review date). DevOps Insights tracks findings across builds, so that a vulnerability accepted in one build can be surfaced if it persists across subsequent builds without being remediated. Concert’s risk model treats persistent, unresolved findings as an escalating risk factor: a finding that was first identified thirty days ago and has been carried through ten subsequent builds receives a higher risk contribution than a finding identified in the current build for the first time.

Emerging regulatory mandates for pipeline evidence. Two regulatory developments are reshaping what quality gates must enforce and what evidence pipelines must produce (see Chapter 3 for the broader regulatory context). The EU Cyber Resilience Act (CRA), which entered into force in 2024 with phased compliance deadlines extending to 2027, requires manufacturers of products with digital elements to maintain a software bill of materials, implement coordinated vulnerability disclosure processes, and provide security updates throughout the product’s support lifetime [13]. These obligations map directly onto the CI/CD pipeline: SBOM generation at build time (via CycloneDX or SPDX), SCA scanning against known vulnerability databases, and a governed disclosure workflow integrated with the vulnerability management lifecycle described above become not merely best practice but legal requirements for in-scope products. For organisations operating in pharmaceutical and life sciences, the US FDA’s 21 CFR Part 11 imposes system validation requirements on software used in regulated processes, including electronic records and electronic signatures [14]. A CI/CD pipeline delivering software into a GxP-regulated environment must produce validation evidence — test execution records, environment configuration snapshots, approval chains — that satisfies Part 11’s expectations for auditability, traceability and tamper-evidence. Quality gates in these pipelines must therefore include validation documentation checks alongside the security and compliance gates described above.

Security scanning composition

25.5 Sovereignty-specific pipeline constraints

The sections above describe how a pipeline’s outputs—artefacts, test results, scan findings, provenance attestations—are governed for sovereignty. This section addresses a more fundamental question: how the pipeline itself is governed. A pipeline that enforces sovereignty constraints on its outputs but executes on infrastructure outside the sovereign boundary is a contradiction. The pipeline is a processing environment; the artefacts it handles—source code, build intermediates, test data, scan results—are data assets. If those assets are subject to sovereignty requirements, the environment that processes them must satisfy the same requirements.

Build infrastructure residency is the first constraint. The compute resources on which builds execute must reside within the sovereign boundary. For Tekton pipelines on OpenShift, this is achieved by running the pipeline on a cluster within the sovereign zone—a natural consequence of Tekton’s architecture, which executes tasks as pods on the cluster where the pipeline is defined [2]. For other CI/CD platforms, achieving build residency may require deploying self-hosted runners (GitHub Actions), self-managed executors (GitLab CI), or dedicated build agents (Jenkins) on infrastructure within the zone. The key requirement is that source code, build intermediates, and build outputs never leave the sovereign boundary during the build process. This includes ephemeral data: build caches, temporary files, and intermediate compilation artefacts must all reside on storage within the zone, even if they are deleted after the build completes.

Artefact registry placement is the second constraint. Container images, Helm charts, and other deployable artefacts must be stored in registries within the sovereign zone. A sovereign pipeline should push its build outputs to a registry instance—such as an OpenShift integrated registry, a self-hosted Harbor instance, or a cloud-provider registry in the approved region—that satisfies the zone’s residency requirements. Cross-zone registry mirroring, where artefacts are replicated from a central registry to zone-local registries, is permissible if the mirroring mechanism satisfies the same constraints: the data path must be encrypted, the source and destination registries must both be within approved zones, and the mirroring should be initiated from the destination zone (pull-based) rather than pushed from the source, to ensure the destination zone controls its own data ingestion.

Cross-zone promotion controls govern how artefacts move between zones with different sovereignty classifications. A common pattern is a development zone with relaxed sovereignty requirements and a production zone with strict requirements. Artefacts built in the development zone may be promoted to the production zone, but the promotion must pass through the sovereignty gate described in section 25.3. In some regulatory contexts, promotion from a non-sovereign zone to a sovereign zone may require the artefact to be rebuilt within the sovereign zone from verified source, rather than simply copied. This rebuild-on-entry pattern provides the strongest assurance that the artefact’s provenance is entirely within the sovereign boundary, at the cost of additional build time and infrastructure.

Pipeline execution environment isolation ensures that pipelines for sovereign workloads do not share infrastructure with pipelines for non-sovereign workloads in ways that could compromise integrity or confidentiality. Tekton’s pod-based execution model provides task-level isolation through Kubernetes namespace separation, pod security standards, and network policies. Each sovereign zone should have its own pipeline namespace, with network policies that prevent pipeline pods from communicating with resources outside the zone. Pipeline service accounts should have narrowly scoped permissions—sufficient to pull source, push artefacts, and post results to DevOps Insights, but nothing more. The principle of least privilege, described in Chapter 13 for operational access, applies with equal force to pipeline service accounts.

Secrets management in the pipeline requires particular attention. Pipelines consume secrets—registry credentials, signing keys, API tokens for DevOps Insights, cloud provider credentials for deployment—and those secrets must be managed with the same rigour applied to application secrets. HashiCorp Vault or IBM Secrets Manager should provide secrets to pipeline tasks through short-lived, dynamically issued credentials rather than long-lived static secrets stored in pipeline configuration. Tekton’s integration with external secret stores, through the Kubernetes Secrets Store CSI driver or through custom task steps that query Vault, enables this pattern without requiring secrets to be stored in the pipeline definition itself.

The cumulative effect of these constraints is that the pipeline becomes a sovereign system in its own right—not merely a tool that produces sovereign outputs, but an infrastructure component that is itself subject to sovereignty governance. This is an important conceptual shift. Organisations that treat the pipeline as exempt from sovereignty constraints—because it is “just tooling” or because “the code is not sensitive”—create a gap in their sovereignty posture that is difficult to defend under regulatory scrutiny. The pipeline has access to source code, credentials, artefacts, and deployment targets; if any of those are sovereign assets, the pipeline that handles them is a sovereign processing environment.

Sovereign pipeline infrastructure

25.6 Agent-enhanced pipeline intelligence

The emergence of agentic operations, described in Chapter 14 and Chapter 15, introduces a new class of participant in the CI/CD pipeline: AI agents that observe, analyse, and act upon pipeline events within defined authority boundaries. Agent-enhanced pipeline intelligence does not replace the pipeline’s deterministic quality gates—those remain the authoritative decision points—but augments the pipeline with capabilities that deterministic automation cannot provide.

Pipeline failure analysis is the most immediately valuable application. When a pipeline fails—a test suite produces unexpected failures, a security scan identifies a new vulnerability, a deployment verification detects a regression—the failure must be diagnosed before it can be remediated. In traditional pipelines, diagnosis is a manual activity: an engineer reviews the build logs, identifies the failing test or scan finding, traces the cause to a code change, and determines the appropriate fix. An agent with access to the build logs, the test results, the code diff, and the historical record of similar failures can perform this analysis in seconds, producing a structured diagnosis that identifies the probable cause, cites the relevant code change, and suggests remediation options. The engineer reviews the agent’s analysis rather than performing the analysis from scratch—a pattern that reduces mean time to remediation without removing human judgement from the loop.

Build outcome prediction uses historical pipeline data to estimate the probability that a given change will pass the pipeline successfully. An agent trained on the historical record of builds—which changes passed, which failed, what characteristics (size, scope, affected components, author, time of day) correlated with failure—can flag high-risk changes before the pipeline is invoked. This prediction does not replace the pipeline; a change flagged as high-risk still runs through all stages and gates. But the prediction enables prioritisation: if a team has multiple changes queued for the pipeline, the agent can recommend an ordering that maximises throughput by running low-risk changes first and reserving capacity for the more intensive testing that high-risk changes may require.

Test selection optimisation addresses one of the most persistent cost and time challenges in CI/CD: the growth of test suites. As applications mature, test suites grow, and the time required to execute the full suite grows with them. At some point, the test suite becomes a bottleneck: developers wait twenty, thirty, sixty minutes for pipeline feedback, which degrades the tight feedback loop that CI/CD is meant to provide. An agent that analyses the code diff and the test dependency graph can identify which tests are relevant to the change and recommend a focused test subset that provides high confidence at a fraction of the execution time. The full suite is still executed—but asynchronously, in a parallel pipeline stage, so that the developer receives fast feedback from the focused subset while the comprehensive suite runs in the background.

Guardrails on pipeline agents are essential and non-negotiable. An agent operating in the CI/CD context has access to source code, build systems, artefact registries, and deployment infrastructure—the same systems that supply chain attacks target. Agent authority must be bounded by explicit policy.

At a minimum, pipeline agents must operate under the following constraints. They may read build logs, test results, scan findings, and pipeline configuration, but they must not modify pipeline definitions or quality gate policies—those are governed by the same code review process that governs all infrastructure-as-code changes. They may suggest code changes, but they must not commit changes directly to protected branches; suggestions must be submitted as pull requests subject to human review. They may query DevOps Insights and Concert for historical data, but they must not modify quality gate evaluations or override gate decisions. They may initiate pre-approved diagnostic actions—rerunning a flaky test, requesting additional scan detail—but they must not initiate deployments, rollbacks, or infrastructure changes without explicit human approval.

These guardrails are expressed as policy-as-code and enforced by the platform, not by the agent’s own self-restraint. The agent’s service account permissions, Kubernetes RBAC bindings, and API token scopes are all configured to reflect the permitted authority boundary. An agent that attempts an action outside its authority is blocked by the platform before the action is executed, producing an audit record of the attempt. This is the same pattern of bounded autonomy described in Chapter 15 for Concert’s agentic workflows, applied to the CI/CD domain.

Agent-enhanced pipeline intelligence

25.7 Pipeline observability and continuous improvement

A CI/CD pipeline is itself a system—a distributed, event-driven system with its own performance characteristics, failure modes, and degradation patterns. Treating the pipeline as a system to be observed and continuously improved, rather than as a tool that either works or does not, is a mark of operational maturity that directly affects the delivery performance metrics described in Chapter 16.

Pipeline telemetry is the foundation of pipeline observability. Every pipeline execution produces a rich set of signals: execution duration for each stage and task, resource consumption (CPU, memory, network I/O), queue wait times, cache hit rates, and outcome (pass, fail, timeout). These signals should be collected, stored, and analysed with the same discipline applied to application telemetry. OpenTelemetry provides a natural framework for pipeline instrumentation: Tekton tasks can be instrumented to emit spans that represent each stage of the pipeline, creating a distributed trace that shows the flow of a build from source checkout to deployment verification [10]. Pipeline traces can be visualised in the same observability tools—Instana, Jaeger, Grafana Tempo—that the organisation uses for application traces, providing a unified view of both application and delivery system health.

Build time analytics identify bottlenecks and degradation trends in pipeline performance. The DORA metric “lead time for changes” is influenced directly by pipeline duration: a pipeline that takes ten minutes provides fundamentally different delivery economics than one that takes sixty. Build time analytics decompose total pipeline duration into stage-level contributions, identifying which stages dominate execution time and how those durations trend over time. A gradual increase in integration test stage duration, for example, may indicate test suite growth that has not been accompanied by test infrastructure scaling—a problem that, left unaddressed, will eventually degrade lead time to the point where it affects deployment frequency.

Flaky test detection addresses one of the most insidious pipeline reliability problems. A flaky test is one that produces inconsistent results—passing on one execution and failing on the next—without any change to the code under test. Flaky tests erode trust in the pipeline: if engineers learn that a failing test is “probably just flaky,” they begin ignoring test failures, which defeats the purpose of the test gate. Detection requires statistical analysis of test results across builds: a test that has failed in three of the last twenty executions, with no correlated code change, is a candidate for flaky classification. Once classified, flaky tests should be quarantined—moved to a separate, non-blocking test suite—until their root cause is identified and resolved. DevOps Insights, which accumulates test results across builds, provides the historical data needed for flaky test detection; an agent can automate the statistical analysis and propose quarantine actions for human approval.

Pipeline-as-code maturity metrics assess the health of the pipeline’s own codebase. Pipeline definitions—Tekton pipeline and task YAML, Jenkinsfiles, GitHub Actions workflow files—are code, and they should be subject to the same quality standards as application code. Maturity metrics include: version control coverage (what percentage of pipelines are defined in code rather than configured through a UI), test coverage for custom pipeline tasks (are reusable tasks tested before being consumed by production pipelines), documentation completeness (do pipeline definitions include descriptive annotations), and dependency freshness (are pipeline task images and tool versions current, or do they lag behind available updates by months or years).

Feeding pipeline health into Concert’s operational model connects pipeline observability to the broader operational intelligence framework. Concert’s entity model, described in Chapter 14, can include pipeline instances as entities with their own health indicators, dependency relationships, and risk profiles [11]. A pipeline that is experiencing elevated failure rates, increased execution times, or growing flaky test counts is a risk to the delivery capability of the services it supports. Concert can surface this risk in its unified view, correlating pipeline health degradation with potential impacts on deployment frequency, lead time, and—ultimately—change failure rate for the services that depend on the pipeline.

This feedback loop—from pipeline telemetry, through analytics, to Concert’s risk model, and back to engineering decisions about pipeline investment—is what transforms pipeline operations from a reactive, break-fix activity into a continuously improving practice. The DORA research consistently finds that investment in delivery infrastructure—faster builds, more reliable tests, better pipeline tooling—is correlated with improvements in all four DORA metrics [12]. Pipeline observability provides the evidence base for those investments, and Concert provides the framework for prioritising them against competing demands.

Pipeline observability architecture

Key Takeaways

Bridge to Chapter 26 — The AI-Augmented Developer Partner

The preceding sections have described how CI/CD pipelines for sovereign cloud estates are designed, governed, and continuously improved. The pipeline is the mechanism by which code becomes running software, and the quality gates within it are the mechanism by which an organisation ensures that only software meeting its standards—functional, security, compliance, and sovereignty—reaches production.

But the pipeline operates on code that has already been written. It evaluates and gates; it does not create. The next frontier in sovereign software delivery is the application of AI to the authoring process itself: code generation, code completion, code explanation, and code transformation powered by large language models trained on vast corpora of programming knowledge. IBM watsonx Code Assistant represents this capability within the IBM sovereign stack, providing AI-assisted coding that operates within the same sovereignty, governance, and policy constraints that govern the rest of the delivery system.

Chapter 26 examines how watsonx Code Assistant integrates with the sovereign development workflow, how its outputs are governed by the same quality gates and supply chain controls described in this chapter, and how the combination of AI-assisted authoring and sovereign pipeline governance creates a development experience that is both more productive and more controlled than either capability alone.

References

[1] IBM Corporation, “IBM DevOps Insights documentation,” IBM Cloud Docs, 2024. [Online]. Available: https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-di_working

[2] Tekton Authors, “Tekton Pipelines documentation,” The Linux Foundation, 2024. [Online]. Available: https://tekton.dev/docs/pipelines/

[3] K. Thompson, “SolarWinds attack explained: and why it was so hard to detect,” CSO Online, 2021. [Online]. Available: https://www.csoonline.com/article/3601508/solarwinds-supply-chain-attack-explained.html

[4] Open Source Security Foundation, “Supply-chain Levels for Software Artifacts (SLSA),” 2023. [Online]. Available: https://slsa.dev/spec/v1.0/

[5] in-toto Project, “in-toto: A framework to secure the integrity of software supply chains,” 2023. [Online]. Available: https://in-toto.io/

[6] Sigstore Authors, “Sigstore: A new standard for signing, verifying and protecting software,” The Linux Foundation, 2023. [Online]. Available: https://www.sigstore.dev/

[7] CycloneDX Project, “CycloneDX Specification,” OWASP Foundation, 2023. [Online]. Available: https://cyclonedx.org/specification/overview/

[8] Linux Foundation, “SPDX Specification Version 2.3,” The Linux Foundation, 2022. [Online]. Available: https://spdx.github.io/spdx-spec/v2.3/

[9] Bridgecrew, “Checkov: Policy-as-code for everyone,” Palo Alto Networks, 2024. [Online]. Available: https://www.checkov.io/

[10] OpenTelemetry Authors, “OpenTelemetry Specification,” Cloud Native Computing Foundation, 2024. [Online]. Available: https://opentelemetry.io/docs/specs/otel/

[11] IBM Corporation, “IBM Concert documentation,” IBM, 2024. [Online]. Available: https://www.ibm.com/docs/en/concert

[12] N. Forsgren, J. Humble and G. Kim, Accelerate: The Science of Lean Software and DevOps. Portland, OR: IT Revolution Press, 2018. [Online]. Available: https://itrevolution.com/product/accelerate/

[13] European Parliament and Council of the European Union, “Regulation (EU) 2024/2847 on horizontal cybersecurity requirements for products with digital elements (Cyber Resilience Act — CRA),” Official Journal of the European Union, vol. L, Nov. 2024. [Online]. Available: https://eur-lex.europa.eu/eli/reg/2024/2847

[14] U.S. Food and Drug Administration, “21 CFR Part 11: Electronic Records; Electronic Signatures,” Code of Federal Regulations, Title 21, Part 11, U.S. Government Publishing Office, Washington, DC, 1997 (as amended). [Online]. Available: https://www.ecfr.gov/current/title-21/chapter-I/subchapter-A/part-11

Sovereign Cloud Operations: AI-Driven Management of Sovereign Estates © 2026 by Alan Hamilton is licensed under CC BY-SA 4.0