Chapter 18 — Multi-Agent Orchestration Patterns

Summary

This chapter establishes the architectural foundations for multi-agent collaboration in sovereign cloud operations, addressing why a single general-purpose agent is insufficient at enterprise scale and how specialised agents—planners, executors, reviewers, guardrails, and synthesisers—must be composed into governed pipelines. It examines orchestration topologies (centralised supervisor, peer-to-peer mesh, and hierarchical), communication mechanisms based on durable message queues, and shared-context models grounded in Concert’s topology and vector-database retrieval. The chapter catalogues the principal failure modes of multi-agent systems—cascading hallucination, scope creep, coordination deadlock, conflicting actions, and audit gaps—and pairs each with specific architectural controls. Architects will find detailed guidance on layered guardrails enforced through OPA policy-as-code, blast-radius limits, zone-aware credentials, and human-in-the-loop checkpoints, along with testing and validation practices for agent pipelines before and after production deployment.

Part VI of this book turns from the individual components of the sovereign operations control plane to the question of how multiple AI agents work together. Earlier chapters established what Concert does as the operational brain, what watsonx Orchestrate does as the conversational interface, and how the four planes—Observability, Automation & Orchestration, Agentic Intelligence, and Governance & Audit—relate to one another. This chapter and those that follow examine the patterns through which agents collaborate to handle operational tasks that no single agent can manage alone, and how those collaborative patterns can be designed to respect the sovereignty, auditability and resilience requirements that have been the book’s constant concern.

18.1 Why a single agent is not enough

There is an appealing simplicity to the idea of a single, general-purpose operational agent: one system that can observe the estate, interpret what it sees, plan a response, execute the necessary changes, validate the outcome, and update the relevant records. If such an agent existed and worked reliably, the orchestration problem discussed in this chapter would not arise. The evidence from both practical deployments and the research literature suggests that this idealised agent is not reachable, at least not with current technology and not at enterprise scale.

The first constraint is the context window. Every language model at the heart of an agent processes information within a finite working memory. In a large enterprise estate, the volume of relevant context—topology graphs, metric histories, change logs, policy documents, runbook libraries, ticket histories, regulatory mappings—far exceeds what any current model can hold simultaneously. An agent asked to diagnose a complex, multi-service incident across three cloud providers while respecting the sovereignty constraints of two distinct regulatory zones must, at some point, choose what to include and what to discard. The more it must do, the more it must discard, and the more its reasoning degrades [1].

The second constraint is specialisation versus generality. A model or agent tuned to excel at network topology analysis will have different characteristics from one tuned to interpret compliance policy or to write Terraform plans. Attempting to build a single agent that is equally adept at all three typically produces one that is mediocre at each. Just as an organisation does not employ a single engineer who is equally expert in network engineering, regulatory affairs and infrastructure-as-code, it should not expect a single agent to substitute for specialists in all these domains.

The third constraint is the single point of failure. An architecture that routes all operational intelligence through one agent creates a critical dependency. If that agent encounters a model hallucination on a sensitive action, if its tool-calling pipeline breaks, or if the upstream model it relies on becomes unavailable, every operational task in flight is affected simultaneously. Distributing intelligence across multiple agents, each with a bounded scope, reduces the blast radius of any individual failure.

The fourth constraint, and perhaps the most significant for sovereign operations, is accountability opacity. When a single agent takes a complex sequence of actions—diagnosis, then planning, then execution, then validation—the chain of reasoning is held within one opaque process. A regulator or internal auditor asking “who decided to apply this change, on what basis, and who authorised it?” faces the challenge of reconstructing reasoning from model internals rather than from a structured, attributed record of decisions made by identifiable actors. Multi-agent systems, when designed correctly, decompose that chain into discrete steps, each attributed to a specific agent with a specific role, with structured hand-offs between them that can be logged and inspected.

The move to multi-agent systems is therefore not a matter of engineering fashion. It is an architectural response to real constraints—cognitive, technical and regulatory—that a single-agent model cannot overcome. The analogy to human team structures is instructive. Large enterprises do not assign all operational responsibility to one senior engineer; they deploy teams with specialised roles, coordination mechanisms, review processes, and clear accountability at each step. Multi-agent orchestration is the computational equivalent of that team structure, with the added requirement that the coordination record must be complete, durable and queryable in ways that even the best-run human teams rarely achieve.

Sovereign operations, in particular, benefits from this decomposition. The auditability requirements of DORA, NIS2 and equivalent frameworks demand that organisations can reconstruct the precise chain of events during any significant incident or automated change. A multi-agent architecture that assigns each role to a distinct, identifiable agent, that routes decisions through reviewer agents before action is taken, and that logs every inter-agent communication provides a natural substrate for that reconstruction. It does not guarantee good governance, but it makes it possible to achieve in a way that a monolithic agent does not.

18.2 Agent roles in sovereign operations

Before examining how multiple agents are arranged into topologies and how they communicate, it is worth establishing a clear taxonomy of the roles they play. These roles are not vendor categories or product labels; they are functional archetypes that appear, in various forms, across research literature and practical deployments alike [2][3]. In the context of this book’s architecture, IBM Concert and watsonx Orchestrate instantiate these archetypes in specific ways.

Planner agents receive a high-level objective—”investigate the latency spike on the payments service,” “prepare a failover plan for the Frankfurt sovereign zone,” “identify all services with misconfigured egress policies”—and decompose it into a sequence of sub-tasks. A planner must understand the structure of the problem domain: which executor agents are available, what each can do, and in what order sub-tasks should be performed. It must also respect sovereignty constraints: if an objective spans two sovereign zones, the planner must not generate a plan that would route sensitive data from one zone’s agents to the other. In Concert’s architecture, the application topology and resilience model provide the planner with a structured understanding of the estate—which services exist, how they depend on each other, which zone they belong to—so that planning is grounded in an accurate world model rather than in assumptions [4].

Executor agents carry out specific, bounded actions. An executor might query Instana for the metric history of a service, invoke an Ansible playbook to rotate a certificate, call the ServiceNow API to open a change record, or apply a Terraform plan through a pipeline. The defining characteristic of an executor is that it acts within a clearly defined, narrow scope. It does not decide what to do or whether to do it; that responsibility lies with the planner and the reviewer. The executor’s job is to translate an instruction into a concrete operation and return a structured result. Watsonx Orchestrate embodies this principle in its domain agent model, where individual agents are configured to call specific tools—Jira, Slack, GitHub, Kubernetes, Terraform—within defined permissions [5].

Reviewer agents occupy the critical middle position between a planner’s proposal and an executor’s action. Before an executor is permitted to act, the reviewer evaluates the proposed action against applicable policies, checks whether the action falls within the executor’s authorised scope, assesses the expected impact, and either approves the action, rejects it with an explanation, or escalates to a human. In a sovereign operations context, the reviewer plays a particularly important role in enforcing zone boundaries: it can intercept a proposed action that would, for example, write diagnostic data to a logging endpoint outside the permitted jurisdiction, and block it before any harm is done. Reviewer agents are implemented in practice through policy-as-code engines—Open Policy Agent is the most widely used [6]—that evaluate structured action representations against rule sets derived from the organisation’s governance policies.

Guardrail agents are distinct from reviewer agents in that they operate continuously rather than at a specific approval checkpoint. Where a reviewer evaluates a discrete proposed action, a guardrail agent monitors the behaviour of the agent system as a whole over time, watching for patterns that suggest a boundary is being exceeded: an executor accumulating permissions it was not originally granted, a planner generating sub-tasks at a rate that suggests a runaway loop, an agent communicating with an endpoint that lies outside its expected network boundaries. The guardrail agent’s response may be to log and alert, to rate-limit, or, in the case of a serious violation, to pause an entire agent pipeline and request human review. This continuous monitoring role maps naturally to the governance and audit plane described in Chapter 4: guardrail agents are the runtime enforcement arm of the policies that watsonx.governance maintains at the lifecycle level.

Synthesiser agents perform the aggregation function that becomes necessary whenever multiple executor agents have worked in parallel on different aspects of a task. After several executors have gathered metrics from different environments, queried different runbooks, and checked different policy databases, their outputs must be assembled into a coherent, human-readable summary that accurately represents the overall situation without losing the attribution of each finding to its source. A synthesiser agent takes these structured outputs, resolves any contradictions or redundancies, and produces a summary—typically in natural language—that a human operator can act on. Critically, the synthesiser must preserve provenance: the summary should make clear which assertions came from which executor agents, so that a reviewer or auditor can trace any specific claim back to its origin.

In Concert’s operational model, these roles are not implemented as five separate products but as distinct, composable behaviours within a coherent workflow framework. Concert itself, with its topology and resilience intelligence, provides the world model that planner agents consult. The Orchestrate agent platform provides the execution and communication substrate through which executor, reviewer and synthesiser agents are instantiated and coordinated. Guardrail functions are distributed across both platforms and through the watsonx.governance integration, which supplies the policy definitions that reviewer and guardrail agents evaluate at runtime.

Agent role taxonomy in a sovereign operations pipeline

18.3 Orchestration topologies

Having defined the roles agents play, the next question is how they are arranged relative to each other. The research and practitioner literature describes several principal topologies for multi-agent systems, each with characteristic trade-offs [2][7]. For sovereign operations, the choice of topology is not purely a matter of technical preference; it is also constrained by the sovereignty requirements that have been a constant theme throughout this book.

Centralised supervisor

In the centralised supervisor topology, a single planner or supervisor agent sits at the top of the hierarchy. It receives objectives, decomposes them into tasks, dispatches those tasks to executor agents, collects results, and coordinates follow-on steps. Reviewer and synthesiser agents may sit either at the supervisor level—reviewing all proposed actions before any executor acts—or at the level of individual task sequences.

The centralised supervisor is easy to reason about and straightforward to audit. Every task passes through a single coordination point, which means the full task graph is visible in one place, the sequence of decisions is clear, and the log of inter-agent communications can be assembled at a single point of aggregation. For operations teams accustomed to incident commanders and change advisory boards, the mental model maps naturally onto existing practice.

The principal drawback is the coordinator bottleneck. When the supervisor agent is making all decisions sequentially—waiting for each executor to complete before dispatching the next—parallelism is limited. In complex incidents involving multiple independent sub-systems, where parallel investigation would significantly reduce resolution time, a strictly centralised supervisor may be too slow. Additionally, if the supervisor agent itself encounters a failure or produces incorrect reasoning, every downstream task is affected.

For many sovereign operations use cases, particularly those that involve a well-defined sequence of steps—detect, diagnose, plan, review, execute, validate—the centralised supervisor is the appropriate starting point. Its simplicity pays dividends in auditability, and its single coordination point makes it relatively straightforward to enforce zone-boundary constraints.

Peer-to-peer mesh

In the peer-to-peer mesh topology, agents communicate directly with each other rather than through a central coordinator. An executor that completes one step in a workflow may directly invoke the next agent in sequence, passing its output as input. Agents may discover each other through a service registry, negotiate task assignments, and collaborate without a designated supervisor.

The mesh topology is well suited to parallel, loosely coupled tasks where the overall workflow does not have a strict total order. When multiple independent investigations must proceed simultaneously—checking network performance in one zone while reviewing configuration drift in another, concurrently examining change history and querying the CMDB—a mesh can dispatch these tasks in parallel without waiting for a central coordinator to sequence them.

The significant disadvantage for sovereign operations is auditability. In a mesh, the coordination record is distributed across all the point-to-point communications between agents. Reconstructing a complete picture of what happened, in what order, on whose instruction, requires assembling logs from every agent that participated in the workflow. Without deliberate design—specifically, the use of a shared, durable message bus discussed in Section 18.4—the audit trail can be fragmented, incomplete or manipulable. For regulated environments where a complete, tamper-evident record of automated actions is a compliance requirement, the mesh topology demands additional investment in observability infrastructure to achieve what the centralised supervisor provides by construction.

A further concern in sovereign operations is that direct agent-to-agent communication in a mesh may, if not carefully governed, inadvertently carry data across sovereign zone boundaries. If Agent A in Zone 1 directly calls Agent B in Zone 2, passing full task context including telemetry and logs, it may violate the data-residency requirement that applies to Zone 1’s data. This is a subtler risk than it might appear: the agents may be doing the right things operationally while breaking a governance rule implicitly through the content of their communications.

Hierarchical

The hierarchical topology generalises the centralised supervisor to multiple levels. Rather than one supervisor managing all executors, supervisors at each level manage a sub-team of executors or lower-level supervisors. A top-level supervisor manages two or three domain supervisors; each domain supervisor manages the executors relevant to its domain: network operations, identity and access, application performance, compliance.

This topology scales well for complex, multi-domain operational tasks, which are precisely the tasks that dominate sovereign cloud operations in a large enterprise. An incident affecting a regulated financial service running across two cloud providers may require simultaneous investigation of network performance, application-level errors, identity and access policies, and regulatory reporting obligations. These domains are sufficiently distinct that separate domain supervisors, each expert in their domain’s tools and policies, can plan and coordinate more accurately than a single general supervisor could.

The hierarchical topology also aligns naturally with the organisational reality of large operations teams, where different sub-teams hold domain expertise. It can be designed so that domain supervisors never need to share raw data with each other—they share structured findings—which makes it easier to enforce zone boundaries at the interface between domains.

The primary challenge is coordination complexity. With multiple supervisors, the design must specify clearly who reports to whom, how conflicts between domain supervisors are resolved, and how a consistent picture of the situation is assembled at the top level without requiring every agent to wait on every other.

Topology choice and sovereign zone constraints

In a sovereign operations architecture, the choice between these topologies must be made in the light of a constraint that does not appear in general-purpose multi-agent research: agents acting in different sovereign zones must not pass raw data across zone boundaries. The orchestration topology must respect the same boundaries as the data topology.

This constraint has practical consequences. In a centralised supervisor topology, the supervisor must be placed in a zone that is allowed to receive summary or structured data from all the zones it coordinates—typically a control plane zone that is itself subject to appropriate sovereignty controls. Executor agents in individual sovereign zones may return structured results (policy evaluation outcomes, metric summaries, change record identifiers) without returning raw telemetry, log content or personal data. The supervisor sees the structured findings; the raw data never leaves its zone.

In a hierarchical topology, the zone boundary rules apply at each level. Domain supervisors in Zone 1 coordinate executors in Zone 1 and return zone-compliant structured results to the top-level supervisor. The top-level supervisor is never exposed to Zone 1’s raw data. This architecture mirrors the principle established in Chapter 6 on the zero-copy integration substrate: federated access rather than centralised copy.

The peer-to-peer mesh, because it routes data through unstructured point-to-point communications, is the topology least naturally aligned with zone boundary enforcement, and requires the most additional investment in data classification, communication inspection, and routing controls to make it safe for sovereign operations.

18.4 Communication mechanisms

The topology defines the structural relationships between agents. The communication mechanism defines how they exchange information. The choice of communication mechanism has significant consequences for performance, resilience, and—critically for sovereign operations—auditability.

Tool calls via function calling APIs are the most immediate form of inter-agent communication in LLM-based systems. An orchestrating agent generates a structured function call—specifying the tool name, input parameters and expected output format—that is evaluated by an execution layer and whose result is returned to the agent’s context window. This mechanism is synchronous: the calling agent waits for the result before proceeding. It is natural for tasks that have a clear dependency ordering and where latency is acceptable. Watsonx Orchestrate uses this mechanism extensively: domain agents expose tools that can be called by orchestrating agents, and the results feed back into the coordinating agent’s reasoning [5].

The advantages of tool calls are simplicity and tight coupling: the orchestrating agent knows immediately whether an action succeeded or failed and can adjust its plan accordingly. The disadvantages are the same tight coupling from a resilience perspective. If the called tool is temporarily unavailable, the calling agent blocks. If the calling agent’s context window fills before a long-running tool completes, the conversation state may be lost. Tool calls are also inherently difficult to audit independently: the record of what was called and what was returned exists in the model’s context, not in a durable, independent store, unless the execution layer explicitly persists it.

Message queues decouple agents temporally and spatially. An agent publishes a message describing a task or a result; a downstream agent or tool consumes it when ready. The queue persists the message independently of either agent’s availability. Apache Kafka, which IBM ships as IBM Event Streams, is the canonical implementation for enterprise-scale, durable, high-throughput messaging [8]. In a multi-agent system built on Kafka, every inter-agent communication is a durable record—a message written to a topic with a specified retention policy. The full history of agent communications can be reconstructed by replaying the message log, which is precisely the kind of tamper-evident, chronologically ordered record that audit and compliance functions require.

For sovereign operations, Kafka-based inter-agent communication offers a further advantage: messages can be routed through topic partitions that are co-located with the sovereign zones they serve. A message carrying Zone 1 operational data is written to a topic whose partitions reside within Zone 1’s infrastructure. Downstream consumer agents in the same zone read from those partitions; agents in other zones are not consumers of those topics. The queue becomes a natural enforcement point for zone-boundary data governance.

Shared state stores—Redis for low-latency, in-memory state; etcd for distributed consensus; purpose-built operational databases—allow multiple agents to read and write common state without direct communication. A planner agent writes a task plan to a shared store; executor agents read their respective tasks from the same store and write results back; a synthesiser agent reads all results and assembles the summary. This pattern avoids the bottleneck of point-to-point communication but introduces the concurrency challenges discussed in Section 18.5.

Structured APIs between agent services—REST, gRPC, GraphQL—sit between synchronous tool calls and asynchronous queues in terms of coupling. They are appropriate when agents are deployed as long-running services rather than as ephemeral conversational processes, and when the interaction model requires request-response semantics but also benefits from versioned, documented interfaces.

In practice, a sophisticated multi-agent system for sovereign operations will use a combination of these mechanisms. Tool calls handle the tight, synchronous interactions between an orchestrating agent and its immediate tool set. Kafka provides the durable, auditable backbone for longer-horizon inter-agent workflows. Shared state stores maintain the working memory of an in-flight task. Structured APIs connect agent services that have stable, service-like characteristics.

The general principle for sovereign operations is that any inter-agent communication that constitutes an operational decision—a proposed action, an approval, a rejection, an escalation—should be written to a durable, attributed, queryable store before it takes effect. This is the communication-layer equivalent of the attribution requirement discussed throughout Part V. IBM Event Streams, as the managed Kafka service within the IBM sovereign stack, provides this backbone in a form that is operationally manageable and compatible with the zone-boundary controls the platform enforces elsewhere.

18.5 Shared context and memory

For a team of agents to work coherently on a complex operational task, they must share situational awareness. Each agent cannot begin from scratch; it must be able to access a common understanding of the estate, the current incident, the constraints that apply, and the actions that have already been taken. How that shared context is constructed, maintained and kept consistent is one of the more technically demanding aspects of multi-agent design.

Concert’s topology and signal model as a shared world model is the foundation. As described in Chapter 14, Concert continuously discovers services, their dependencies, their configurations, their telemetry histories, and their resilience posture. This model is not a static snapshot; it is continuously updated as the estate changes. For a multi-agent system, Concert’s model serves as the authoritative description of the world that all agents consult. A planner agent deciding how to investigate an incident reads the topology to understand which services are likely affected. An executor agent about to apply a configuration change queries the model to verify that the target service is in a state that makes the change safe. A reviewer agent checks the model to confirm that the proposed change respects zone boundaries. By grounding all agents in the same topology model, the system avoids the dissonance that arises when different agents hold inconsistent beliefs about the estate [4].

Vector databases extend shared memory into the historical and semantic dimension. Operational history—past incidents, their diagnoses, the actions taken, the outcomes observed—accumulated over months and years is too voluminous and too unstructured for agents to search efficiently through exact-match queries. Vector databases index this history as high-dimensional embeddings, allowing agents to retrieve semantically similar past cases given a description of the current situation. This is the pattern described by Lewis et al. as Retrieval-Augmented Generation: rather than relying solely on knowledge encoded in model weights, agents retrieve relevant documents or records from an external index and incorporate them into their reasoning context [1]. In an operational setting, this means an executor agent investigating a memory leak can retrieve the three most similar incidents from the past year, read how they were diagnosed and resolved, and use that context to guide its investigation.

Structured working memory for in-flight tasks is the third layer. When an orchestration workflow is in progress—a planner has decomposed an objective, executors are working on their sub-tasks, a reviewer has approved some actions and is evaluating others—the current state of all of this must be held somewhere that all participating agents can read and write. This is the shared state store role described in Section 18.4. In practice, a Redis instance or an etcd cluster holds the task graph, the status of each sub-task, the outputs of completed executors, and the decisions of reviewer agents. Every agent that touches this working memory writes a timestamped, attributed record: which agent read what, which agent wrote what, at what time and in response to which event.

The challenge of context consistency arises when multiple agents read and write shared state concurrently. If two executor agents simultaneously discover that a service configuration needs updating—one based on network telemetry, the other based on application metrics—and both attempt to write their proposed update to shared state, the result may be a conflict. In the worst case, the second write silently overwrites the first, and the combined view becomes inconsistent.

Three patterns address this challenge, each with different trade-offs. Optimistic locking allows multiple agents to read the current state freely but requires each write to assert the version it is updating. If the version in the store has changed since the agent last read it—because another agent has written in the meantime—the write is rejected and the agent must re-read and recalculate. This pattern is simple and low-overhead in low-contention scenarios. Event sourcing avoids the problem of conflicting writes to a shared mutable state by appending all changes as immutable events to a log. The current state is derived by replaying the event log from any given point. No write is ever lost; conflicts are resolved by the logic that interprets the log. This pattern aligns naturally with Kafka-based inter-agent communication: if all state changes are events on a Kafka topic, the event log is the state store. Conflict-free Replicated Data Types (CRDTs) provide data structures—counters, sets, maps—that are designed to merge concurrent writes without conflicts, at the cost of some restrictions on the kinds of operations they support. CRDTs are most useful when agents need to accumulate findings concurrently without coordination: a set of affected services, a collection of policy violations, a union of relevant runbook identifiers.

For sovereign operations, there is a governance implication that cuts across all three patterns: every write to shared state must be attributed to an agent identity. Whether the persistence mechanism is optimistic locking on a Redis key, an event appended to a Kafka topic, or a CRDT merge operation, the record must include who wrote it, at what time, under what authority, and as part of what workflow. This is not merely good engineering practice; it is a requirement of the accountability chain that regulators and auditors expect. An action that is correctly executed but whose provenance cannot be established is, from a governance perspective, almost as problematic as one that was not executed at all.

18.6 Failure modes of multi-agent systems

Multi-agent systems introduce failure modes that do not appear in simpler, single-agent or script-based automation. Understanding these failure modes is a prerequisite for designing systems that are both effective and safe. For each mode, there is a corresponding architectural control.

Cascading hallucination occurs when an executor agent produces an incorrect output—a fabricated metric value, a misidentified service name, an invented configuration parameter—and a downstream agent treats that output as ground truth without independent verification. The downstream agent incorporates the hallucinated value into its reasoning, produces further reasoning based on a false premise, and may ultimately take or propose an action that is wrong in a way that is difficult to diagnose because each step in the chain appeared locally reasonable. The research literature on LLM hallucination documents this phenomenon in individual model outputs [9]; in a multi-agent chain, the error compounds at each step.

The primary architectural control is the reviewer agent, which independently verifies proposed actions against authoritative sources—the Concert topology model, policy-as-code rules, external APIs—rather than relying solely on the executor’s claim. Secondary controls include idempotent executors, discussed below, and structured output schemas that constrain what executors can claim: if an executor’s output must include a verifiable resource identifier from the Concert CMDB, a hallucinated resource name will fail schema validation before it propagates.

Scope creep occurs when an agent interprets its mandate more broadly than intended and takes actions beyond its authorisation. This may happen gradually—a single “sensible” extension of scope at each step—or abruptly, if an agent’s reasoning leads it to conclude that achieving its objective requires actions it was not explicitly authorised to take. Scope creep is particularly concerning in sovereign operations because unauthorised actions may cross zone boundaries, affect resources that belong to a different regulatory context, or trigger changes to shared infrastructure that other teams depend on.

The control for scope creep is the combination of a blast radius limit and explicit authorisation scope. Each agent is issued credentials that have been scoped to the specific resources, zones and operations it is permitted to affect. An executor agent assigned to investigate a payment service in Zone 2 holds credentials that are valid only for reading telemetry in Zone 2. Even if the agent reasons that it should also examine Zone 1 because the payment service has a dependency there, its credentials will not permit that action. Policy-as-code evaluation at the action level enforces this at runtime, as described in Section 18.8.

Coordination deadlock occurs when two agents each wait for the other to complete before proceeding, and neither can proceed without the other. In operational terms, Agent A may be waiting for Agent B to release a lock on a configuration resource before it can read the current state, while Agent B is waiting for Agent A to complete a dependency check before it will release that lock. Neither makes progress; the task blocks indefinitely.

The architectural control is a combination of timeout policies and explicit deadlock detection. Every inter-agent communication carries a deadline: if a response is not received within the specified duration, the waiting agent treats the interaction as failed and escalates. Deadlock detection in the orchestration layer monitors the dependency graph of in-flight tasks and identifies cycles—patterns where Agent A is waiting on B, which is waiting on C, which is waiting on A. When a cycle is detected, the orchestration layer breaks it by cancelling the lowest-priority task in the cycle and notifying the relevant agents to retry with an adjusted approach. IBM Event Streams’ consumer group management provides the plumbing for this: each agent’s progress is tracked through Kafka consumer offsets, making stalls visible.

Conflicting actions occur when two agents simultaneously make mutually incompatible changes to the same system. One agent, following its planner’s instruction, scales up a service’s replica count to address latency. A second agent, following a different planner’s instruction in a parallel workflow, scales the same service down to address a cost optimisation objective. Both changes succeed individually; the result is an oscillation between two states that serves neither objective.

The control is a combination of shared state coordination (so that both agents know the other’s intent before acting) and an executor-level idempotency requirement. Executors are designed so that their actions are declared in terms of desired state rather than incremental delta: “ensure replica count is N” rather than “increment replica count by one.” This allows the orchestration layer to detect that two executors are attempting to set incompatible desired states for the same resource, reject the second instruction, and escalate to the planner or human for resolution. In practice, this is implemented through the state-locking mechanisms described in Section 18.5, combined with a resource registry that records which agent holds the lock on which resource.

Audit gap is the failure mode most directly relevant to the governance requirements of this book. An audit gap occurs when an action is taken—infrastructure is changed, a configuration is modified, a service is restarted—but the attribution chain from that action back to the human who authorised the original objective is broken or missing. This may happen if an agent takes an action that was not explicitly part of its tasked workflow but was inferred as necessary; if a communication between agents uses a channel that is not durably logged; or if the working-memory state is overwritten without preserving the history.

The control for audit gaps is the mandatory attribution logging requirement that must be built into every layer of the multi-agent architecture. Every action taken by an executor must be associated with: the identity of the executor agent; the identity of the planner or supervisor that dispatched the task; the identity of the reviewer agent that approved it; the ticket or objective identifier that initiated the workflow; and the identity of the human operator who raised the original request. This chain must be written to a durable, append-only log before the action is executed, not after. IBM Event Streams, with its configurable retention and replication, provides the infrastructure for this log in a form that is auditable, immutable and accessible to governance tooling.

18.7 Multi-vendor agent collision

The failure modes catalogued in the preceding section are internal to the agentic operations layer. In practice, however, a sovereign cloud estate hosts multiple independent sources of automated action, many of which predate the agentic layer and operate with no awareness of it. A realistic multi-cloud environment typically runs cloud-native auto-scalers (AWS Auto Scaling Groups, Azure VMSS, Google Cloud managed instance groups), Kubernetes HPA and VPA controllers, a resource optimisation platform such as IBM Turbonomic, and the agentic operations layer described in this chapter. Each system is individually well-designed. The problem is that they can issue contradictory instructions on the same resource within the same time window, and the resulting oscillation is worse than either system acting alone.

This is not a hypothetical concern. Any organisation that has deployed Turbonomic alongside a Kubernetes HPA on the same workload has encountered it: the HPA scales pods based on CPU utilisation while Turbonomic right-sizes the same pods based on a broader set of resource signals. Without explicit coordination, the two systems fight each other in alternating cycles, and the problem grows more acute as the agentic layer introduces additional actors capable of proposing infrastructure changes.

Resource locking and lease patterns

Three architectural patterns address the collision problem at the prevention layer, before conflicting actions reach the resource.

Advisory locks with time-limited leases. Before any agent—whether a Turbonomic action, an Orchestrate executor, or a Kubernetes operator—modifies a resource, it must acquire a lease from a coordination service. The lease is a time-bounded exclusive claim, stored in a distributed consensus system such as etcd, a DynamoDB table with conditional writes, or a purpose-built coordination service exposed through the control plane. It records the acquiring agent’s identity, the intended action category, and an expiry time. Other agents check the lease before proceeding; if a valid lease exists, the requesting agent defers or escalates for conflict resolution. The time-limited expiry prevents a crashed agent from holding a lock indefinitely.

Priority hierarchy. Not all automated actions carry equal urgency or authority. The recommended hierarchy, from highest to lowest, is: safety and incident-response agents (responding to active outages or security events); human-initiated actions; sovereignty enforcement agents (correcting zone-boundary or compliance violations); and optimisation agents (cost right-sizing, performance tuning, capacity rebalancing). When a higher-priority agent requests a lease that a lower-priority agent currently holds, the lower-priority lease is pre-empted: the lower-priority agent receives a cancellation signal, rolls back if necessary, and yields the resource. Conflicts between agents at the same priority level are escalated to a human operator rather than resolved by arbitrary tie-breaking.

Single writer principle. For any given resource and action category, exactly one system should be authoritative. If Turbonomic owns scaling decisions for a workload, the Kubernetes HPA should be disabled or configured in a passive, metrics-only mode. The principle does not mean that only one system exists in the estate; it means that for each specific resource, the question “who is allowed to change this?” has a single, unambiguous answer recorded in the resource registry. The agentic operations layer consults this registry before proposing any action, and the OPA policy evaluation described in Section 18.8 enforces it.

Conflict detection and resolution

Prevention cannot eliminate all collisions, particularly when legacy automation operates outside the lease framework. Concert’s topology model, which continuously ingests state changes from across the estate, can detect conflicting recommendations: when two systems have proposed contradictory actions on the same resource within a configurable time window, Concert flags a collision event. The supervisory agent resolves the conflict automatically if the priority hierarchy is unambiguous, or escalates to a human operator otherwise. Every collision event is logged with full attribution—which agents were involved, what each proposed, and how the conflict was resolved. Recurring collisions on a specific resource class signal that the single-writer assignment needs to be revisited.

The “disable native” versus “integrate native” decision

Architects must confront a genuine trade-off. The first option is to disable native auto-scaling entirely and delegate all scaling decisions to Turbonomic or the agentic layer. This eliminates collision by construction. The cost is that cloud-native auto-scalers are deeply integrated with their platform’s health-checking and availability-zone logic; disabling them forfeits that responsiveness and requires the replacement system to replicate platform-specific awareness.

The second option is to retain native auto-scaling with guardrails—configuring auto-scalers to operate within a bounded range (minimum and maximum instance counts set by Turbonomic or policy), while the optimisation layer adjusts those bounds periodically rather than issuing individual scale actions. The native auto-scaler acts within its authorised range; the agentic layer modifies the range itself, on a slower cadence. Conflicts can still arise at the boundary, but they are less frequent and more predictable than unconstrained dual-control.

For sovereign operations, the second approach is generally preferable. It preserves platform-specific resilience logic and confines the agentic layer to setting policy boundaries rather than issuing individual commands—a separation that is easier to audit and less likely to produce surprising behaviour during incidents.

Sovereign zone scoping of agent authority

The collision domain can be further partitioned by aligning agent authority with sovereign zone boundaries. An agent authorised to act in Zone A must not hold leases on resources in Zone B, enforced through the zone-aware credential mechanism described in Section 18.8. Collision detection and resolution thus become zone-local problems, and a misbehaving agent in one zone cannot interfere with resources in another. For multi-jurisdictional organisations, this partitioning is not merely a performance optimisation—it is a governance necessity.

18.8 Guardrails architecture

Guardrails are the runtime enforcement mechanisms that ensure agent behaviour remains within authorised bounds, regardless of what the agent’s model-based reasoning might otherwise produce. They are the complement to governance policies expressed at the lifecycle level by watsonx.governance: where governance defines what is permitted over time, guardrails enforce it moment by moment.

Policy-as-code evaluation before action is the foundational guardrail. Every proposed action from an executor agent—before it is passed to the tool that will implement it—is submitted to a policy evaluation engine. Open Policy Agent (OPA), developed under the Cloud Native Computing Foundation, provides the industry-standard implementation of this pattern [6]. OPA accepts a structured description of the proposed action (the tool being called, the resource being affected, the agent’s claimed identity, the sovereign zone in which the resource resides, and the category of the change) and evaluates it against a set of Rego policies that encode the organisation’s governance rules. If the policies permit the action, execution proceeds. If they deny it, the action is blocked and the reason is logged. If they require additional approval, the action is held pending escalation.

For sovereign operations, the OPA policy library must encode zone-boundary rules (no executor agent may affect a resource in Zone X if it was authorised in Zone Y), data classification rules (an agent that has access to data classified as personally identifiable must not write outputs to stores outside the jurisdiction that governs that data), and change category rules (configuration changes to production systems in regulated zones require reviewer agent approval; changes in sandbox zones may proceed without explicit review).

Blast radius limits provide a quantitative constraint on the scope of any single agent invocation. The policy is simple in principle: no agent may affect more than N resources per workflow invocation without triggering an explicit human approval checkpoint. The value of N is calibrated per resource category and per zone. In a regulated production environment, N may be one: every resource change requires individual approval. In a development environment with loose sovereignty constraints, N may be fifty. The point is that a runaway agent—one stuck in a loop or operating on incorrect reasoning—will hit the blast radius limit before it can cause widespread damage, and will be stopped pending human review.

Zone-aware credentials are the access control expression of zone-boundary enforcement. Each executor agent is provisioned with credentials scoped to the specific zones, resource types and operations it is authorised for in the current workflow. These credentials are issued by the identity and access management infrastructure described in Chapter 13, and they expire at the end of the workflow or after a configurable time-to-live, whichever is shorter. An agent that attempts to call a tool against a resource outside its credential scope will receive an access denial from the target system—not just a soft policy warning, but a hard technical refusal. The agent cannot override this with more persuasive reasoning; the credential is simply invalid for the attempted operation.

Rate limiting and circuit breakers address the failure mode of runaway automation: an agent executing actions faster than the estate can safely absorb, or repeatedly retrying a failing action. Rate limiting caps the number of actions an agent can take per unit of time. Circuit breakers, borrowed from the distributed systems pattern [10], monitor the error rate of an agent’s tool calls and, when the error rate exceeds a threshold, interrupt the circuit—stopping further calls to that tool until it has been assessed and cleared. In the context of multi-agent systems, circuit breakers may be applied not just to individual tool endpoints but to entire agent pipelines: if the reviewer agent is detecting an unusually high rejection rate for a particular executor’s proposed actions, the circuit breaker can pause that executor’s operation and request human diagnosis.

Human-in-the-loop checkpoints define the categories of action for which automated execution, regardless of confidence level, is not permitted. This is not a failure of automation; it is a deliberate design choice reflecting the principle that certain decisions involve stakes, accountability or regulatory significance that no agent can fully bear. In sovereign operations, the categories that typically require human confirmation include: failover of a regulated workload between jurisdictions; change to a zone boundary definition or sovereign perimeter; modification of a cryptographic key management policy; any action affecting more than a defined percentage of production capacity simultaneously; and any action whose policy evaluation returns an explicit “escalate to human” result rather than a permit or deny.

The integration of watsonx.governance with the orchestration layer is what makes these guardrails operational at enterprise scale. Watsonx.governance maintains the inventory of approved agents, their authorised roles and capability bounds, the version-controlled policy sets that OPA evaluates, and the audit log of all policy evaluations and their outcomes. It provides the governance lifecycle management—policy authoring, review, approval and publication—that ensures guardrail rules remain current as the organisation’s regulatory context evolves. The operational guardrails (OPA, rate limiters, circuit breakers, credential expiry) enforce the rules in real time; watsonx.governance provides the authoritative source of what those rules should be.

Guardrails architecture for a multi-agent sovereign operations pipeline

18.9 Testing and validating multi-agent systems

Testing a multi-agent system is substantially more demanding than testing individual agents or individual automation scripts. Unit-level testing of a single agent—verifying that it produces the correct tool call given a specific input prompt, or that its output parser correctly extracts a service name from a model response—is necessary but not sufficient. The characteristic failure modes described in Section 18.6 are emergent properties of agent interaction: they arise from the combination of agents and communication mechanisms, not from any individual component in isolation.

Integration testing requires assembling the full agent pipeline—planner, executors, reviewer, synthesiser, guardrail agents—against a representative test environment and evaluating its end-to-end behaviour on a range of scenarios. The test environment must include a representative Concert topology model (which can be populated from a copy of production topology metadata, stripped of sensitive data), realistic metric and log streams (which can be generated synthetically or replayed from recorded production traffic), and a policy-as-code rule set that mirrors production governance. The Kafka topics that carry inter-agent communications in production should be replicated in the test environment, so that the integration test captures the full message flow and not just the final output.

Chaos injection—deliberately introducing failures into the test environment—is essential for validating the failure mode controls described in Section 18.6. A chaos test might inject a hallucinated metric value into an executor’s tool response, to verify that the reviewer agent correctly rejects the downstream action proposal. It might introduce a network partition between two executor agents, to verify that the timeout and circuit breaker mechanisms fire as expected. It might lock a shared state key to simulate contention, to verify that the optimistic locking retry logic and the deadlock detection mechanism behave correctly. Chaos engineering, well established in the resilience testing of distributed systems [10], applies directly to multi-agent systems where the “distributed” components are agents rather than microservices.

Shadow mode deployment is the technique that provides the highest confidence before granting write permissions to a production agent pipeline. In shadow mode, the full multi-agent system runs against production data and production context—reading live telemetry from Instana, consulting the live Concert topology model, evaluating live policy from watsonx.governance—but all proposed actions are logged and evaluated rather than executed. The shadow pipeline produces a record of what it would have done, in what order, on what basis, and subject to what guardrail decisions. Human operators review this record against what they actually did during the same period. Discrepancies are investigated: did the agent propose an action that humans did not take, and if so, would the action have been beneficial? Did the agent fail to propose an action that humans took, indicating a gap in its reasoning? Shadow mode is demanding in terms of human review time, but it provides a level of behavioural evidence that no amount of synthetic testing can match [11].

Evaluating agent output quality requires metrics that go beyond traditional software correctness measures. Four metrics are particularly useful for sovereign operations contexts. The task completion rate measures the fraction of objectives presented to the agent pipeline that result in a complete, verified resolution rather than an escalation or abandonment. The policy violation rate measures the fraction of proposed actions that are blocked by the OPA guardrail, which serves as an indicator of how well the planner’s decomposition respects governance constraints—a high violation rate suggests the planner is generating plans that the governance rules will not permit, indicating a misalignment between the planner’s training and the organisation’s policy. The escalation rate measures the fraction of tasks that require human intervention beyond the initial objective-setting, which indicates whether the agent pipeline is calibrated correctly relative to the organisation’s automation maturity. The attribution completeness rate measures the fraction of executed actions for which a complete attribution chain—agent identity, approver, objective, human initiator—can be reconstructed from the audit log, which is the direct governance metric for audit gap prevention.

Human review sampling complements automated metrics. A random sample of completed agent workflows—perhaps five per cent in the early stages of deployment, declining as confidence grows—is reviewed by senior operations engineers who assess whether the agent’s reasoning was sound, whether it missed relevant context, and whether its action selection was appropriate. This review serves both as a quality signal and as a feedback source: engineers who identify patterns of suboptimal reasoning contribute to the refinement of planning prompts, the expansion of the runbook library that executors consult, and the tuning of guardrail thresholds.

The feedback loop from production behaviour to agent improvement is the mechanism that converts operational experience into capability improvement over time. Every escalation that a human resolves is a case study: what was the context, what did the agent propose, why was it insufficient, and what did the human do differently? This case study, structured and stored in the vector database that executor agents query for historical precedents, enriches the retrieval corpus for future incidents. Every policy violation that an OPA rule blocks is a signal that the planner’s decomposition logic may need adjustment. Every completed task that receives a positive assessment from a human reviewer is a positive signal that the associated reasoning pattern is worth reinforcing. None of this feedback requires retraining the underlying model; it operates at the level of the retrieval corpus, the policy rules, and the workflow configuration—artefacts that the operations team can update without a model release cycle.

Key Takeaways

Bridge to Chapter 19 — ITSM and Multi-Agent Workflows

This chapter has established the architectural principles governing how multiple AI agents collaborate in a sovereign operations context. It has defined a functional taxonomy of agent roles, analysed the trade-offs between orchestration topologies in the light of sovereign zone constraints, described the communication mechanisms that provide durable and auditable inter-agent coordination, explained how shared context and memory can be maintained consistently across concurrent agents, catalogued the failure modes unique to multi-agent systems and their corresponding controls, detailed the layered guardrails architecture that enforces governance at runtime, and outlined the testing and validation practices necessary to build confidence in agent pipeline behaviour before and after production deployment.

The patterns described here are not context-free. They are designed to operate within the four-planes architecture established in Chapter 4, grounded in the Concert topology model introduced in Chapter 14, expressed through the Orchestrate agent platform described in Chapter 17, and governed by the watsonx.governance lifecycle management discussed in Chapter 22 and Chapter 23. They are also designed to compose: the patterns for ITSM integration, knowledge-augmented operations, and autonomous self-healing explored in subsequent chapters all depend on the multi-agent orchestration substrate described here.

Chapter 19 moves from the general patterns of multi-agent orchestration to their application in a specific, high-value operational domain: IT Service Management. It examines how Concert, Orchestrate and a multi-agent pipeline interact with ITSM platforms such as ServiceNow to handle the full incident lifecycle—from automated detection and triage, through multi-agent investigation and remediation planning, to change execution, validation and ticket closure—in a way that preserves the governance and auditability requirements that regulated enterprises must satisfy.

References

[1] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-T. Yih, T. Rocktäschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in Advances in Neural Information Processing Systems, vol. 33, H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H. Lin, Eds. Curran Associates, Inc., 2020, pp. 9459–9474.

[2] J. S. Park, J. C. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” in Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ‘23), New York, NY, USA: ACM, 2023, pp. 1–22. doi: 10.1145/3586183.3606763.

[3] H. Chase, N. Siddiqui, A. Bhosle, et al., LangGraph: A library for building stateful, multi-actor applications with LLMs, LangChain, Inc., 2024. [Online]. Available: https://langchain-ai.github.io/langgraph/

[4] IBM, “IBM Concert: Introduction and overview,” IBM Documentation, 2024. [Online]. Available: https://www.ibm.com/docs/en/concert

[5] IBM, “IBM watsonx Orchestrate: Agent and tool configuration,” IBM Documentation, 2024. [Online]. Available: https://www.ibm.com/docs/en/watsonx/watson-orchestrate

[6] Open Policy Agent (OPA) Project, “OPA documentation,” Cloud Native Computing Foundation, 2024. [Online]. Available: https://www.openpolicyagent.org/docs/latest/

[7] IBM Research, “Multi-agent systems for IT operations automation,” IBM Research Technical Report, IBM Research Division, 2023. [Online]. Available: https://research.ibm.com

[8] Apache Software Foundation, Apache Kafka documentation, version 3.7, 2024. [Online]. Available: https://kafka.apache.org/documentation/

[9] Y. Bai et al., “Constitutional AI: Harmlessness from AI feedback,” Anthropic Technical Report, Anthropic, San Francisco, CA, 2022. [Online]. Available: https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback

[10] M. T. Nygard, Release It! Design and Deploy Production-Ready Software, 2nd ed. Raleigh, NC: Pragmatic Bookshelf, 2018.

[11] National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, Gaithersburg, MD: U.S. Department of Commerce, Jan. 2023. [Online]. Available: https://doi.org/10.6028/NIST.AI.100-1

Sovereign Cloud Operations: AI-Driven Management of Sovereign Estates © 2026 by Alan Hamilton is licensed under CC BY-SA 4.0