Chapter 28 — Recommendation-Driven Operations

Summary

This chapter describes the shift from reactive alerting to proactive, recommendation-driven operations in which AI agents identify situations meriting attention, formulate structured proposals, and present them to operators with evidence, confidence assessments, and policy context. It defines the seven structural elements of a well-formed recommendation — observation, analysis, proposed action, expected outcome, confidence score, risk assessment, and policy compliance status — and examines the generation patterns that produce them, including historical incident matching, anomaly detection with causal reasoning, and predictive analytics. The chapter addresses confidence scoring, uncertainty quantification, and the calibration practices required to sustain operator trust, as well as recommendation routing based on service ownership, skill requirements, and regulatory exposure. Sovereign constraints are treated throughout: models must execute within zone boundaries, cross-zone actions must be detected and governed, and every recommendation must satisfy transparency and audit requirements regardless of its disposition.

The previous chapter examined how conversational interfaces lower the barrier between operators and the operational intelligence embedded in agentic platforms. A chat-first experience makes it possible for an engineer to ask a question in natural language and receive an answer drawn from topology, telemetry, knowledge bases and policy. But there is a fundamental limitation to a question-and-answer model: it is reactive. The operator must know that something warrants investigation before they ask. Recommendation-driven operations inverts this relationship. Rather than waiting for a human to pose a question, the system identifies situations that merit attention, formulates structured recommendations, and presents them to the appropriate operator with the evidence, confidence assessment and policy context required for an informed decision.

This chapter examines how operational recommendations are generated, structured, scored, routed and governed within a sovereign operations architecture. It traces the evolution from simple alerting to intelligent recommendation, describes the anatomy of a well-formed recommendation, explores the AI patterns that produce recommendations, addresses the critical question of confidence and uncertainty, and considers how sovereign zone constraints shape what recommendation systems are permitted to propose.

28.1 From alerts to recommendations

The history of operational awareness can be read as a sequence of attempts to close the gap between a system exhibiting a problem and a human understanding what to do about it. Each generation of tooling has narrowed that gap, but none before the current one has attempted to bridge it entirely.

The first generation was monitoring: instruments that measured system state and displayed it on dashboards. CPU utilisation, memory consumption, disk usage, network throughput—each metric was sampled at regular intervals and plotted against time. The value of monitoring was real but its limitation was equally real: it required a human to watch. An operator staring at a dashboard at three in the morning might notice a trend that suggested trouble, but if no one was watching, the trend passed unremarked until the system failed. Monitoring answered the question “what is happening?” but only for those who were already looking [1].

The second generation was alerting: the application of thresholds to monitored signals, with automatic notification when a threshold was breached. If CPU utilisation exceeded ninety per cent for five consecutive minutes, page the on-call engineer. If the error rate crossed two per cent, send a notification to the operations channel. Alerting removed the requirement for continuous human attention but introduced a new problem: the threshold had to be set correctly in advance, the signal had to be the right one, and the notification had to reach the right person. Too many alerts and the on-call engineer learned to ignore them—a phenomenon so pervasive in the industry that “alert fatigue” has become a standard term in the operational lexicon [2]. Moogsoft’s research across enterprise operations teams found that large organisations routinely generate millions of alerts per month, of which the vast majority are either false positives, duplicates, or symptoms of problems already known. The cognitive load imposed by this volume is not merely unpleasant; it is operationally dangerous. When every alert looks the same, the one that represents a genuine, novel, urgent problem is lost in the noise [2].

The third generation was correlation: platforms that consumed raw alert streams and applied grouping logic to reduce them to a smaller number of actionable situations. IBM Instana’s Smart Alerts, Moogsoft’s correlation engine, and similar capabilities in other AIOps platforms all share this intent: if twenty alerts fire within a two-minute window and all relate to services that share a common dependency, those twenty alerts are more usefully presented as a single correlated situation—”the database cluster in zone EU-West is degraded”—than as twenty individual notifications [3]. Correlation represented a genuine advance. It reduced the volume of notifications, provided topological context, and gave operators a starting point for investigation rather than a collection of disconnected data points. But correlation still left the operator with the question “what should I do about it?” The correlated situation told them that a problem existed and approximately where it was located. It did not tell them what was causing it, what should be done, what the expected outcome of the proposed action would be, or whether the action was safe to take given the current regulatory and policy context.

The fourth generation—the subject of this chapter—is recommendation. A recommendation system does not merely detect that something is wrong and group related signals. It analyses the situation, draws on historical experience, topological context, policy constraints and predictive models, and formulates a specific, structured proposal for action. The proposal includes not only the what—”scale the payments database read replicas from three to five”—but also the why, the expected outcome, the confidence level, the risk assessment, and the policy compliance status. The operator receiving the recommendation is not starting an investigation from zero. They are evaluating a pre-assembled case, with evidence, and making a decision about whether to accept, modify or reject it.

The four generations of operational awareness

This evolution matters for sovereign operations specifically because the complexity of the decision space has expanded beyond what reactive approaches can reliably handle. An operator in a sovereign multi-cloud environment is not merely asking “is the service healthy?” They are asking whether telemetry can be examined across zone boundaries, whether a proposed remediation would move data across jurisdictions, whether the remediation is permitted under the applicable regulatory framework, and whether the change aligns with the organisation’s risk appetite for the affected service tier. These questions, asked under time pressure during an incident, are precisely the conditions under which human judgement is most vulnerable to error. A recommendation system that has already evaluated these dimensions and presents its analysis alongside the proposed action transforms the operator’s task from “work out what to do under pressure” to “evaluate whether this well-reasoned proposal is correct.”

The cognitive load problem deserves explicit treatment because it is the operational reality that makes recommendation systems necessary rather than merely convenient. Research on cognitive load in high-stakes operational environments—originally conducted in aviation and nuclear power but increasingly applied to software operations—demonstrates that human decision-making degrades predictably under conditions of information overload, time pressure and fatigue [4]. The degradation is not random; it follows specific patterns: operators fixate on the first plausible explanation, neglect contradictory evidence, and underweight low-probability high-consequence scenarios. These are precisely the failure modes that post-incident reviews identify most frequently as contributors to extended outages. A recommendation system that presents a structured analysis, including alternative hypotheses and their relative likelihoods, directly counteracts these cognitive failure patterns by externalising the analytical work that the operator’s overloaded cognition would otherwise perform badly.

28.2 Anatomy of an operational recommendation

Not all recommendations are equal. An informal suggestion—”you might want to look at the database”—is qualitatively different from a structured recommendation that an operator can evaluate, an auditor can review, and a policy engine can validate. The difference lies in structure. A well-formed operational recommendation contains a set of elements that together provide the basis for an informed, accountable decision. Without this structure, recommendations are opinions; with it, they are governance artefacts.

The observation is the factual basis: what the system has detected that prompted the recommendation. “The p99 latency for the payments-api service has exceeded 800ms for the past twelve minutes, coinciding with a 340 per cent increase in database connection pool wait times on payments-db-primary.” The observation is grounded in telemetry and must be verifiable—an operator who doubts the recommendation should be able to navigate to the underlying metrics and confirm that the observation is accurate. In a Concert-based architecture, the observation draws from Instana’s continuous telemetry and Concert’s topology-aware correlation, and the recommendation carries references to the specific metric streams and time windows from which the observation was derived [3][5].

The analysis is the interpretive layer: the reasoning that connects the observation to a causal hypothesis. “Historical analysis of this service shows that connection pool exhaustion on payments-db-primary has preceded p99 latency breaches on six previous occasions in the past ninety days. In five of those six cases, the root cause was identified as long-running analytical queries contending with transactional workload during peak processing windows. The current event coincides with the batch reconciliation window, which is consistent with this pattern.” The analysis demonstrates that the recommendation is not a reflex—it is grounded in evidence and reasoning. It also exposes the reasoning to scrutiny: an operator who knows that the batch reconciliation schedule was changed last week can immediately assess whether the historical pattern still applies.

The proposed action is the specific operational step being recommended. “Terminate long-running analytical queries on payments-db-primary that have exceeded 120 seconds of execution time, and redirect the batch reconciliation workload to the read replica payments-db-analytics for the remainder of the current processing window.” The action must be specific enough to be executable—or, in an automated environment, specific enough to be translated into an API call or a runbook invocation without ambiguity. Vague recommendations (“consider investigating the database”) transfer cognitive load back to the operator and forfeit the primary value of the recommendation system.

The expected outcome describes what should happen if the recommendation is accepted and the action is taken successfully. “Connection pool wait times on payments-db-primary should return to baseline (below 15ms) within three minutes of query termination. p99 latency for payments-api should subsequently recover to below 200ms within five minutes.” The expected outcome serves two purposes. First, it allows the operator to evaluate the recommendation’s plausibility: if the expected outcome seems disproportionate to the proposed action, that is a signal to investigate further before accepting. Second, it provides a measurable basis for post-action verification: if the action is taken and the expected outcome does not materialise, the recommendation was either wrong in its analysis or incomplete in its proposed action, and further investigation is warranted.

The confidence score expresses the system’s assessment of how likely the recommendation is to be correct. Section 28.4 examines confidence scoring in detail; here, it suffices to note that the confidence score is not a single number but a composite that reflects confidence in the observation (are the underlying metrics reliable?), confidence in the analysis (does the causal reasoning hold?), and confidence in the expected outcome (will the proposed action produce the predicted result?). A recommendation might have high observational confidence, moderate analytical confidence, and low outcome confidence—a profile that tells the operator something importantly different from a single aggregate score.

The risk assessment evaluates the potential negative consequences of taking the proposed action. “Terminating long-running queries will abort the current batch reconciliation run. The reconciliation will need to be restarted after the peak processing window closes, adding approximately ninety minutes to the reconciliation cycle. No data loss will result because the reconciliation process is idempotent.” The risk assessment also considers the consequences of not acting: “If the current trajectory continues without intervention, the connection pool is projected to reach full exhaustion within eight minutes, at which point the payments-api service will begin returning 503 errors to upstream consumers.”

The policy compliance status indicates whether the proposed action is permitted under the applicable operational and regulatory policies. “The proposed action is within the scope of pre-approved standard change model SC-DB-007 (query termination on production databases). No cross-zone data movement is involved. The action does not require CAB approval under the current change management policy.” This element is distinctive to enterprise and sovereign operations contexts: in environments where actions have regulatory consequences, knowing that a recommended action is policy-compliant before deciding whether to accept it is not optional metadata—it is a prerequisite for the decision. In the architecture described in this book, policy compliance evaluation is performed by the Open Policy Agent policy engine, with policies defined in Rego and maintained as code [6].

Anatomy of a well-formed operational recommendation

The structural discipline imposed by these seven elements serves three audiences simultaneously. For the operator, the structure provides the information required to make a decision quickly and confidently, without having to assemble the case independently. For the auditor, the structure provides a documented record of what the system recommended, on what basis, and with what confidence—regardless of whether the recommendation was accepted, modified or rejected. For the machine learning feedback loop, the structure provides a labelled dataset of recommendations, their structural components, and their outcomes, enabling continuous improvement of the recommendation system itself.

28.3 Recommendation generation patterns

The generation of operational recommendations is not a single technique but a family of complementary patterns, each suited to a different category of operational situation. A mature recommendation system employs several of these patterns simultaneously and synthesises their outputs into unified recommendations. The patterns can be understood along a spectrum from reactive to predictive.

Pattern matching against historical incidents is the most direct generation mechanism. When a current situation resembles a previous incident—similar telemetry signatures, similar topology context, similar temporal characteristics—the resolution of the previous incident becomes a candidate recommendation for the current one. The knowledge-augmented operations architecture described in Chapter 20 provides the retrieval mechanism: the current situation is encoded as a query against the vector database of post-incident reviews and resolution records, and the most semantically similar historical cases are retrieved. The recommendation agent then evaluates whether the historical resolution is applicable to the current context, accounting for differences in system configuration, scale, or environment that may have changed since the historical incident occurred.

This pattern is powerful when the organisation has accumulated a rich incident history and has invested in the post-incident review process. Its limitation is that it can only recommend actions that have been taken before. Novel failure modes—those without historical precedent in the organisation’s experience—will not be addressed by pattern matching alone. The pattern is also sensitive to the quality of the historical record: if post-incident reviews are sparse, poorly structured, or inconsistently created, the retrieval results will be correspondingly unreliable. The knowledge base quality metrics described in Chapter 20, Section 20.7, directly affect the reliability of pattern-matched recommendations.

Anomaly detection with causal reasoning operates at a different level of abstraction. Rather than matching the current situation to a specific historical incident, this pattern identifies deviations from established baselines and reasons about their probable causes using the topology model. Instana’s anomaly detection identifies when a metric deviates significantly from its learned baseline—not merely crossing a static threshold but exhibiting behaviour that is statistically unusual given the time of day, day of week, and recent history [3]. Concert’s topology model then provides the causal reasoning layer: knowing that service A depends on service B, which depends on database C, the system can reason that an anomaly in service A’s latency, coinciding with an anomaly in database C’s connection pool, is more likely to be caused by the database issue than by a problem in service A itself [5].

The causal reasoning is not infallible—correlation in time and topology does not guarantee causation—but it is substantially more reliable than the unaided human intuition that it replaces, particularly in complex topologies where a single root cause may manifest as anomalies across dozens of dependent services. The recommendation generated from this pattern takes the form: “Anomalous behaviour has been detected in [component]. Causal analysis, based on topology dependencies and temporal correlation, suggests [root cause hypothesis]. The recommended action is [proposed remediation targeting the hypothesised root cause].”

Predictive analytics generate recommendations before problems materialise, addressing the proactive dimension that reactive patterns cannot reach. Three predictive domains are particularly well-established in operational practice.

Capacity prediction uses time-series forecasting to project resource consumption trends and identify when capacity thresholds will be breached. “At the current growth rate, the storage volume for the customer-data service in sovereign zone EU-Central will reach ninety per cent utilisation within fourteen days. The recommended action is to provision additional storage capacity or implement data lifecycle policies to reduce consumption.” The value of capacity recommendations is that they convert an eventual crisis into a planned change—a shift-left operation in the truest sense—giving the organisation time to execute the change through normal change management processes rather than under emergency conditions [7].

Performance degradation prediction uses learned models of normal performance behaviour to detect gradual degradation trends that have not yet breached alert thresholds. A service whose p99 latency has been increasing by three per cent per week for the past six weeks has not triggered any threshold-based alert, but the trend, if it continues, will breach the SLO within a month. The recommendation system identifies these slow-burn trends and proposes investigation before they become incidents.

Failure prediction uses patterns learned from historical failures to identify conditions that have preceded failures in the past. If a particular combination of metrics—elevated garbage collection pause times, increasing thread pool utilisation, and rising error rates on a specific API endpoint—has preceded an out-of-memory crash on three previous occasions, the recommendation system recognises the pattern forming and proposes pre-emptive action. This is a more sophisticated variant of pattern matching: rather than matching against the incident itself, it matches against the precursor conditions that preceded the incident.

Best-practice comparison against the knowledge base generates recommendations by comparing the current configuration or operational posture of a service against documented best practices, vendor recommendations, or organisational standards. This pattern is less about responding to an immediate situation and more about continuous posture improvement. “The TLS certificate for the payments-gateway service is configured with a 4096-bit RSA key. The organisation’s cryptographic standards document (CS-2024-003) specifies that new certificates should use ECDSA P-384 keys. The recommended action is to rotate the certificate to an ECDSA P-384 key at the next scheduled rotation window.” These recommendations are generated by periodically comparing the actual state of the estate—as discovered by Concert’s continuous topology and configuration scanning—against the corpus of standards and best practices indexed in the operational knowledge base [5][8].

Four recommendation generation patterns arranged by operational timeline

In practice, a single operational situation may trigger recommendations from multiple patterns simultaneously. A database performance anomaly might generate a pattern-matched recommendation based on a similar historical incident, a causal reasoning recommendation based on topology analysis, and a best-practice recommendation noting that the database’s connection pool configuration differs from the vendor-recommended settings. The recommendation system must synthesise these into a coherent set rather than presenting them as competing proposals. Concert’s orchestration layer handles this synthesis by ranking recommendations based on their relevance, confidence and urgency scores, and by identifying redundancies—cases where two recommendations from different patterns are proposing essentially the same action from different analytical starting points [5].

28.4 Confidence scoring and uncertainty quantification

A recommendation without a confidence assessment is an assertion, and assertions are a poor basis for operational decisions. The operator needs to know not only what the system recommends but how confident it is in that recommendation, and—critically—what the sources of uncertainty are. Confidence scoring in operational recommendation systems is therefore not an ornamental feature but a structural requirement for safe human-agent collaboration.

Confidence in an operational recommendation is not a monolithic quantity. It decomposes into at least three dimensions, each of which can vary independently.

Observational confidence reflects the reliability and completeness of the telemetry on which the recommendation is based. If the recommendation rests on metrics from a well-instrumented service with continuous, high-resolution telemetry and no recent gaps, observational confidence is high. If the recommendation rests partly on metrics from a service whose instrumentation was recently changed, whose telemetry has intermittent gaps, or whose metrics are sampled at low resolution, observational confidence is correspondingly lower. Observational confidence can be assessed programmatically: the system knows the instrumentation coverage of each service, the recency and resolution of the telemetry, and whether any data quality issues have been flagged.

Analytical confidence reflects the strength of the causal reasoning that connects the observation to the proposed action. A recommendation based on a causal chain that has been validated in five previous incidents, with the same topology and the same failure signature, carries high analytical confidence. A recommendation based on a novel correlation—a pattern the system has not seen before but which is topologically plausible—carries lower analytical confidence. Analytical confidence is harder to quantify than observational confidence because it depends on the richness of the historical record and the complexity of the causal chain. Systems that have processed thousands of incidents across a well-documented estate will produce higher analytical confidence scores than systems operating with sparse history against a poorly documented topology.

Outcome confidence reflects the system’s assessment of how likely the proposed action is to produce the predicted result. This dimension incorporates the uncertainty inherent in any prediction about complex system behaviour. Even when the observation is solid and the analysis is sound, the outcome of a proposed action may be uncertain because the system’s behaviour is non-linear, because the action has side effects that the model does not fully capture, or because the environment has changed in ways that invalidate historical outcome data. Outcome confidence is typically the lowest of the three dimensions, because it is the most forward-looking and therefore the most exposed to the unpredictability of complex systems.

The composite confidence score presented to the operator should expose these dimensions rather than collapsing them into a single number. A recommendation with high observational confidence, high analytical confidence, and low outcome confidence tells the operator: “We are quite sure about what is happening and why, but less certain about what will happen if you take this action.” That profile calls for a different decision process—perhaps accepting the analysis but taking the action incrementally, with monitoring—than a recommendation with uniformly high confidence across all three dimensions.

Calibration is the property that makes confidence scores trustworthy over time. A well-calibrated confidence score means that, of all recommendations issued with seventy per cent confidence, approximately seventy per cent turned out to be correct. Calibration is not automatic; it must be measured and maintained through a feedback loop. Each recommendation that is accepted and executed generates an outcome that can be compared to the predicted outcome. Over time, these comparisons reveal whether the system’s confidence scores are well-calibrated, overconfident, or underconfident. Platt scaling and isotonic regression are standard techniques for recalibrating confidence scores based on observed outcomes [9], and they can be applied periodically—monthly or quarterly—using the accumulated recommendation-outcome dataset.

The operational consequences of miscalibration are asymmetric, and this asymmetry must inform the system’s design. False positives—recommendations that propose action when none is needed—waste operator time and erode trust in the recommendation system. If operators learn that most recommendations are unnecessary, they will begin ignoring all recommendations, including the ones that matter. This is alert fatigue transposed to the recommendation layer, and it is the primary risk of an overactive recommendation system. False negatives—situations where the system fails to generate a recommendation when one was warranted—are less immediately visible but potentially more consequential: a missed recommendation during a developing incident can result in delayed response and extended impact.

The cost asymmetry between false positives and false negatives varies by context. For a tier-one regulated financial service, the cost of a missed recommendation (a false negative that leads to an SLO breach or a regulatory notification) substantially exceeds the cost of an unnecessary recommendation (a false positive that wastes fifteen minutes of operator time). For a development environment, the cost relationship may be reversed: constant unnecessary recommendations are disruptive, while a missed recommendation in a non-production environment is rarely consequential. The recommendation system’s confidence thresholds—the minimum confidence required to generate a recommendation rather than simply logging the observation for retrospective analysis—should therefore be service-tier-dependent, with lower thresholds (more sensitive, more false positives) for critical services and higher thresholds (more specific, fewer false positives) for lower-tier services.

The decision of when to recommend versus when to escalate is a function of both confidence and risk. A high-confidence, low-risk recommendation can be presented to the operator as a routine suggestion. A high-confidence, high-risk recommendation—one where the proposed action is likely correct but the consequences of being wrong are severe—should be escalated to a senior operator or a change advisory review. A low-confidence observation, regardless of risk, should be logged and made available for investigation rather than presented as a recommendation, because the analytical basis is insufficient to support a specific proposed action. This decision matrix maps naturally to the human-in-the-loop guardrails described in Chapter 18: the recommendation system’s confidence and risk assessment feed directly into the guardrail framework’s escalation logic.

Confidence-risk decision matrix

28.5 Recommendation routing and prioritisation

A recommendation that reaches the wrong person, or reaches the right person at the wrong time, or arrives buried in a queue of lower-priority items, fails to achieve its purpose regardless of its analytical quality. Routing and prioritisation are therefore not administrative afterthoughts but integral components of the recommendation system’s effectiveness.

Routing determines who receives a recommendation. The routing decision draws on three inputs. First, the service ownership model maintained in Concert’s topology identifies the team and individual responsible for each service. A recommendation concerning the payments-api service is routed to the payments engineering team; a recommendation concerning network latency between availability zones is routed to the network engineering team. The ownership model must be current—a routing decision based on stale ownership data sends recommendations into a void where no one acts on them. Concert’s continuous discovery helps maintain this currency by associating services with their deployment metadata, which typically includes team ownership labels [5].

Second, the on-call schedule determines which member of the owning team is currently available to act. Routing a recommendation to a team’s general channel during business hours is appropriate for non-urgent posture recommendations. Routing a recommendation about an actively developing incident to the on-call engineer’s personal notification channel is appropriate for urgent situations. The recommendation system integrates with the organisation’s on-call management tooling—PagerDuty, Opsgenie, or equivalent—to resolve the current on-call assignment for a given service [10].

Third, the skill and authority requirements of the recommendation determine whether the initial recipient can act on it or whether it requires routing to a specialist or a more senior decision-maker. A recommendation to scale a Kubernetes deployment can be acted upon by any member of the platform engineering team with appropriate RBAC permissions. A recommendation to modify a database schema requires routing to the database engineering team specifically. A recommendation that involves cross-zone coordination—a failover from one sovereign zone to another, for instance—requires routing to an operator with cross-zone authority, which may be a senior architect or a designated cross-zone operations lead.

Prioritisation determines the order in which recommendations are presented when multiple recommendations are active simultaneously. During a major incident, the recommendation system may generate recommendations from multiple patterns across multiple affected services. Presenting them in arbitrary order forces the operator to triage the recommendations themselves—a cognitive load transfer that the system should handle.

Prioritisation is driven by three factors. Business impact is the primary ranking criterion: a recommendation concerning a tier-one customer-facing service takes precedence over a recommendation concerning a tier-three internal tool, regardless of technical severity. Business impact is assessed using Concert’s service model, which maps technical services to business capabilities and assigns criticality tiers based on organisational classification [5].

SLO proximity provides a temporal urgency dimension. A recommendation becomes more urgent as the affected service approaches its SLO boundary. If the payments-api service has a 99.9 per cent availability SLO and the current incident has consumed forty per cent of the monthly error budget, the recommendation to remediate is more urgent than if only five per cent of the error budget has been consumed. SLO-based prioritisation aligns recommendation urgency with business consequence rather than with arbitrary severity classifications [11].

Regulatory exposure adds a compliance dimension to prioritisation. A recommendation concerning a service that processes personal data subject to GDPR, or a service that falls within the scope of DORA’s critical function requirements, carries regulatory exposure that elevates its priority. The recommendation system assesses regulatory exposure by consulting the service’s classification metadata in Concert’s topology: services tagged with regulatory scope attributes (GDPR, DORA, NIS2, sector-specific regulations) receive a priority uplift that ensures they are addressed before services without regulatory exposure [12].

The integration with ITSM workflows described in Chapter 19 is the mechanism through which prioritised recommendations enter the organisation’s operational process. A recommendation that is accepted by an operator triggers the creation of the appropriate ITSM artefact—an incident record if the recommendation responds to an active problem, a change request if the recommendation proposes a configuration change, or a problem record if the recommendation identifies a systemic issue that warrants root cause investigation. The recommendation’s structural elements—observation, analysis, proposed action, expected outcome, confidence score, risk assessment, policy compliance status—are written into the ITSM record, providing a documented basis for the action that is richer and more structured than a manually created record would typically contain.

Recommendation routing and prioritisation flow

28.6 Human-agent collaboration on recommendations

The value of a recommendation system is realised only when operators engage with its output constructively—neither accepting recommendations blindly nor dismissing them reflexively. The design of the recommendation review workflow directly influences which of these patterns emerges, and getting the design right is as much a human factors challenge as a technical one.

The recommendation review workflow begins when a recommendation is delivered to an operator. The operator’s first task is to evaluate the observation: is the factual basis accurate? In well-designed systems, the observation is presented with links to the underlying telemetry, allowing the operator to verify the data with minimal effort. If the observation is confirmed, the operator evaluates the analysis: does the causal reasoning make sense given their knowledge of the system? The analysis section exposes the reasoning chain, including the historical precedents and the topology relationships on which the causal hypothesis is based. An experienced operator may recognise factors that the system’s analysis has not accounted for—a recent configuration change not yet reflected in the topology model, a known issue with a dependency that the system has not correlated, or domain knowledge about the service’s behaviour under specific conditions that is not captured in the knowledge base.

The operator then has three options. Acceptance means the operator agrees with the recommendation and authorises the proposed action. In a fully integrated architecture, acceptance triggers the execution workflow: if the action falls within a pre-approved standard change model, the recommendation system—through watsonx Orchestrate’s agent execution capabilities—can carry out the action automatically, creating the appropriate change record in ServiceNow and monitoring the expected outcome [6][13]. If the action requires explicit change approval, acceptance creates a change request pre-populated with the recommendation’s evidence and routes it through the normal approval workflow.

Modification means the operator agrees with the analysis but proposes a different action, or agrees with the proposed action but wants to adjust its scope or timing. “The analysis is correct—the database connection pool is the bottleneck—but rather than terminating queries, I want to increase the connection pool size first and terminate queries only if that does not resolve the issue within ten minutes.” Modification is captured as a structured annotation on the recommendation record: the original recommendation is preserved, the operator’s modification is recorded alongside it, and the modified action becomes the basis for execution. This creates a rich training signal for the recommendation system: modifications that recur across multiple similar situations indicate a systematic gap in the system’s reasoning that can be addressed through model retraining or knowledge base updates.

Rejection means the operator disagrees with the recommendation and does not authorise any action based on it. Rejection should require a reason—not as a bureaucratic burden but as a feedback mechanism. “Rejected: the batch reconciliation schedule was changed last Tuesday and now runs two hours earlier. The analysis is based on an outdated schedule assumption.” Rejection reasons are the most valuable feedback the recommendation system can receive, because they identify specific gaps in the system’s knowledge or reasoning that, once corrected, prevent the same erroneous recommendation from being generated in future.

The feedback loop that connects operator decisions back to recommendation quality improvement is the mechanism through which the system becomes more trustworthy over time. Each recommendation outcome—accepted and successful, accepted and unsuccessful, modified, or rejected—contributes to a labelled dataset that can be used for three purposes. First, confidence recalibration: the observed success rate of recommendations at each confidence level is compared to the stated confidence, and the calibration is adjusted if they diverge. Second, pattern refinement: recommendations that are consistently modified in the same way indicate that the recommendation generation pattern is systematically missing a consideration that can be incorporated. Third, knowledge base gap identification: recommendations that are rejected because the system lacked specific knowledge identify documents or data sources that should be added to the operational knowledge base.

Building trust in a recommendation system is a gradual process that follows a predictable trajectory. In the early stages, operators will verify every recommendation independently—treating recommendations as suggestions to be validated rather than proposals to be acted upon. This is appropriate and should be encouraged: blind trust in a system that has not yet demonstrated its reliability is more dangerous than scepticism. As the system demonstrates accuracy—as operators observe that recommendations are correct more often than not, that confidence scores are well-calibrated, and that the system’s analysis matches their own expert judgement—trust increases and operators begin to rely on recommendations more heavily, investing their limited attention in verifying the recommendations that fall outside the system’s demonstrated competence rather than verifying everything.

This trust trajectory has implications for system deployment. A recommendation system should not be launched at full scope across all services simultaneously. A phased rollout—starting with a small number of well-understood services where the recommendation system’s accuracy can be validated quickly and visibly—allows trust to build on demonstrated performance rather than on claims. Each phase of the rollout extends coverage to additional services, and each extension carries with it the credibility established in previous phases. Concert’s service-tier model provides a natural structure for this phased rollout: starting with tier-two services (important but not the most critical), validating accuracy, and then extending to tier-one services once confidence is established [5].

28.7 Sovereign constraints on recommendation systems

Recommendation-driven operations in a sovereign architecture must respect the same zone boundaries, data residency requirements, and regulatory constraints that govern every other component of the operations control plane. These constraints shape the recommendation system in ways that go beyond access control and into the fundamental question of what a recommendation system is permitted to propose.

Data residency for recommendation models is the first and most fundamental constraint. The models that generate recommendations—whether they are statistical models trained on historical incident data, embedding models used for pattern matching against the knowledge base, or large language models used for causal reasoning—process operational data that is subject to residency requirements. Telemetry from a sovereign zone, incident histories from that zone, and the configuration data of services operating in that zone are all sovereign data. The recommendation model that processes this data must execute within the same sovereign zone, or within a zone whose residency posture is compatible with the data it processes. This precludes, for most sovereign deployments, the use of externally hosted AI services for recommendation generation against sovereign operational data. The IBM watsonx.ai platform, deployed within the sovereign zone’s infrastructure, provides the model execution environment that satisfies this constraint [8].

The training data for recommendation models raises a related but distinct question. A model trained on incident data from multiple sovereign zones would, in principle, benefit from the broader pattern base. But the training process itself involves processing data from each zone, which may violate residency constraints if training occurs outside those zones. Two architectural patterns address this tension. Federated learning trains local models within each zone and aggregates model updates (not data) at a coordination point, producing a model that benefits from cross-zone experience without cross-zone data movement. Transfer learning trains a base model on non-sovereign data (public incident databases, vendor documentation, general operational knowledge) and then fine-tunes it within each sovereign zone on that zone’s specific data. Both patterns preserve data residency while allowing the recommendation system to benefit from broader experience than a single zone’s history would provide.

Cross-zone recommendation constraints ensure that the recommendation system does not propose actions that would violate sovereign boundaries. A recommendation to failover a service from sovereign zone EU-Central to sovereign zone APAC-South must be recognised as a cross-zone action and subjected to the cross-zone governance framework described in earlier chapters—even if the failover would be technically beneficial and the recommendation system has high confidence in its efficacy. The recommendation system enforces this by consulting the service’s zone classification in Concert’s topology before generating any recommendation that involves resource provisioning, data movement, or traffic routing. If the proposed action would cross a zone boundary, the recommendation is either suppressed (if cross-zone actions are categorically prohibited for that service), escalated to a cross-zone authority (if cross-zone actions are permitted with explicit approval), or annotated with a cross-zone compliance warning (if the operator needs to be aware of the cross-zone implications even though the action is within policy).

Zone-specific recommendation policies allow each sovereign zone to define constraints on what recommendations are permissible within that zone. A sovereign zone operated under the jurisdiction of a national data protection authority with specific requirements for automated decision-making may require that all recommendations involving personal data processing include an explicit data protection impact assessment reference. A zone operated within the scope of a financial regulator may require that recommendations affecting regulated services include a regulatory classification of the proposed change. These zone-specific policies are expressed as Rego rules in the OPA policy engine and evaluated as part of the recommendation generation pipeline—before the recommendation is presented to the operator, not after [6].

Recommendation transparency is a regulatory requirement that is gaining prominence across jurisdictions. The EU’s AI Act, which applies to AI systems that make or materially influence decisions in high-risk domains, establishes transparency obligations for AI systems that include the requirement to inform users that they are interacting with an AI system and to provide meaningful information about the logic involved [14]. Applied to operational recommendation systems, this means that recommendations must be accompanied by explanations that are comprehensible to the operator—not merely raw model outputs but structured reasoning that connects observations to analysis to proposed actions. The recommendation anatomy described in Section 28.2 is, in part, a response to this transparency requirement: each structural element of the recommendation contributes to the explainability of the system’s output.

The audit trail for recommendations is a governance requirement that applies regardless of whether a recommendation is accepted, modified or rejected. Every recommendation generated by the system must be retained, with its full structural content, for the audit retention period applicable to the affected service’s regulatory scope. Accepted recommendations must carry a record of who accepted them, when, and the outcome of the action taken. Modified recommendations must carry a record of the modification and the operator’s rationale. Rejected recommendations must carry a record of the rejection and the reason. This audit trail is the basis for demonstrating to regulators that the recommendation system is operating within its authorised scope, that human oversight is effective, and that the system’s accuracy is monitored and maintained. In a DORA-regulated context, this audit trail forms part of the ICT risk management documentation that the regulation requires financial entities to maintain [12].

Sovereign constraints on the recommendation pipeline

Key Takeaways

Bridge to Chapter 29 — Autonomous Remediation and Self-Healing

This chapter has examined how AI agents generate, structure, score, route and govern operational recommendations—transforming the operator’s role from reactive investigator to informed decision-maker evaluating pre-assembled analytical cases. The recommendations described here still require a human decision: the operator accepts, modifies or rejects each proposal, and the system learns from the outcome.

Chapter 29 takes the next step along the autonomy spectrum. Where recommendation-driven operations presents proposals for human evaluation, autonomous self-healing systems execute remediation actions without waiting for human approval—within carefully bounded scopes, under policy constraints, and with continuous monitoring of outcomes. The confidence scoring, risk assessment, and policy compliance mechanisms described in this chapter are the prerequisites for that autonomy: a system cannot be trusted to act independently until it has demonstrated, through a sustained track record of accurate recommendations reviewed and validated by humans, that its judgement is reliable within its authorised scope. The journey from recommendation to autonomy is not a technology upgrade; it is a trust gradient, earned through demonstrated competence and governed by the same sovereign constraints that apply at every level of the architecture.

References

[1] R. Beyer, C. Jones, J. Petoff, and N. R. Murphy, Site Reliability Engineering: How Google Runs Production Systems. Sebastopol, CA: O’Reilly Media, 2016. [Online]. Available: https://sre.google/sre-book/table-of-contents/

[2] Moogsoft, “Alert fatigue in IT operations: Causes, consequences and solutions,” Moogsoft White Paper, 2023. [Online]. Available: https://www.moogsoft.com/resources/

[3] IBM, “IBM Instana Observability: Smart Alerts and anomaly detection,” IBM Documentation, 2024. [Online]. Available: https://www.ibm.com/docs/en/instana-observability

[4] D. D. Woods and E. Hollnagel, Joint Cognitive Systems: Patterns in Cognitive Systems Engineering. Boca Raton, FL: CRC Press, 2006. [Online]. Available: https://doi.org/10.1201/9781420005684

[5] IBM, “IBM Concert: Application topology and operational intelligence,” IBM Documentation, 2024. [Online]. Available: https://www.ibm.com/docs/en/concert

[6] Open Policy Agent, “OPA documentation: Policy language (Rego) and integration patterns,” The Open Policy Agent Project, 2024. [Online]. Available: https://www.openpolicyagent.org/docs/latest/

[7] N. R. Herbst, S. Kounev, and R. Reussner, “Elasticity in cloud computing: What it is, and what it is not,” in Proc. 10th Int. Conf. Autonomic Computing (ICAC), San Jose, CA, USA, 2013, pp. 23–27. [Online]. Available: https://doi.org/10.1109/ICAC.2013.6604433

[8] IBM, “IBM watsonx.ai: Foundation models and AI development studio,” IBM Documentation, 2024. [Online]. Available: https://www.ibm.com/docs/en/watsonx/saas

[9] J. Platt, “Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods,” in Advances in Large Margin Classifiers, A. Smola, P. Bartlett, B. Schölkopf, and D. Schuurmans, Eds. Cambridge, MA: MIT Press, 2000, pp. 61–74. [Online]. Available: https://www.researchgate.net/publication/2594015_Probabilistic_Outputs_for_Support_Vector_Machines_and_Comparisons_to_Regularized_Likelihood_Methods

[10] PagerDuty, “PagerDuty operations cloud: Incident response and on-call management,” PagerDuty Documentation, 2024. [Online]. Available: https://support.pagerduty.com/docs

[11] A. Hidalgo, Implementing Service Level Objectives: A Practical Guide to SLIs, SLOs, and Error Budgets. Sebastopol, CA: O’Reilly Media, 2020. [Online]. Available: https://www.oreilly.com/library/view/implementing-service-level/9781492076803/

[12] European Parliament and Council, “Regulation (EU) 2022/2554 of the European Parliament and of the Council on digital operational resilience for the financial sector (DORA),” Official Journal of the European Union, L 333, pp. 1–79, 27 Dec. 2022. [Online]. Available: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32022R2554

[13] IBM, “IBM watsonx Orchestrate: Agent and workflow automation,” IBM Documentation, 2024. [Online]. Available: https://www.ibm.com/docs/en/watsonx/watson-orchestrate

[14] European Parliament and Council, “Regulation (EU) 2024/1689 of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (AI Act),” Official Journal of the European Union, L series, 12 Jul. 2024. [Online]. Available: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689

Sovereign Cloud Operations: AI-Driven Management of Sovereign Estates © 2026 by Alan Hamilton is licensed under CC BY-SA 4.0