Chapter 20 — Knowledge-Augmented Operations

Summary

This chapter addresses the knowledge dimension of agentic operations: how organisations build, maintain, and query an operational knowledge base that delivers accumulated institutional knowledge—runbooks, post-incident reviews, architecture decision records, regulatory guidance—to agents and human operators at the moment of need. It presents Retrieval-Augmented Generation as the architecturally appropriate mechanism, examining the critical design decisions around chunking strategy, embedding model selection, and metadata filtering that determine retrieval quality. The chapter covers knowledge base construction, staleness detection, and the complementary roles of the Concert topology model (structural, relational, real-time queries) and the vector database (semantic, contextual, historical queries). Architects will find practical guidance on sovereign-zone constraints for the entire RAG pipeline, continuous quality measurement through retrieval relevance and accuracy metrics, and the IBM watsonx platform components that realise the knowledge layer.

The previous two chapters addressed how agents collaborate and how those collaborations integrate with ITSM workflows. Throughout both discussions, one assumption has been quietly in play: that agents have access to the right knowledge at the right moment. A planner agent investigating an incident needs to know how similar incidents were resolved last quarter. An executor agent preparing a remediation needs to know which runbook applies and whether it has been superseded. A reviewer agent evaluating a proposed change needs to know whether an architecture decision record constrains the approach. Without this knowledge, agents are reduced to reasoning from model pre-training alone—a foundation that is generic, potentially outdated, and entirely unaware of the organisation’s specific environment, constraints and accumulated operational experience.

This chapter examines how organisations can build, maintain and query an operational knowledge base that makes accumulated knowledge available—to agents and to human operators—at the moment of need.

20.1 The knowledge problem in operations

Ask any experienced operations engineer what slows them down during a major incident and the answer is rarely “we didn’t have enough monitoring.” The instruments, in most modern environments, are abundant. What is missing is comprehension. Someone needs to locate the runbook for the affected component, verify it is current, cross-reference the last post-incident review for anything similar, find out who made the relevant architectural decisions and why, and check whether any regulatory constraints apply to the remediation options under consideration. Each of these steps takes time: time spent searching Confluence, scrolling through SharePoint, sending messages to colleagues who may or may not be awake, and piecing together a picture from fragments scattered across a dozen systems.

The knowledge problem in operations is not primarily a technology problem. It is an accumulation problem: organisations generate operational knowledge continuously—through incidents, through change management, through architectural deliberation, through vendor relationships, through regulatory correspondence—but they have no systematic mechanism for making that accumulated knowledge retrievable at the moment it is needed. Instead, knowledge sediments. Good post-incident reviews are written, filed, and never consulted again. Architecture decision records are created diligently for six months and then abandoned when a project team changes. Runbooks are written for the system as it existed at a particular point in time and then slowly drift from the system as it actually exists. Vendor documentation is bookmarked by individuals and lost when those individuals move on. Regulatory guidance is absorbed by the compliance team and never reaches the engineers who need to act on it.

The result is a form of institutional amnesia that is most acutely felt at three o’clock in the morning, when a service-affecting incident is escalating and the on-call engineer is the most junior person available, because the more experienced engineers have, over time, absorbed sufficient institutional knowledge to avoid being on-call as often. At that moment, the value of everything the organisation has learned about its systems is near-zero, because it cannot be located, read and applied in the time available.

This failure mode has a measurable cost. Blameless post-incident reviews consistently find that a significant proportion of incident time is spent not in diagnosis or remediation but in information retrieval—locating the correct runbook, establishing who owns the affected service, finding the previous ticket that matches this pattern [1]. The ITIL 4 knowledge management practice, which frames this problem explicitly, identifies the absence of a structured knowledge management system as a primary contributor to extended mean time to restore service and to the repetition of previously solved problems [2]. Each incident that requires an engineer to rediscover knowledge that already exists in the organisation is a direct financial cost, a user experience failure, and a missed opportunity to apply what has been learned.

Knowledge-augmented operations addresses this problem by making accumulated operational knowledge available as a first-class operational resource—queryable, current, contextually relevant and accessible not only to human operators but to the AI agents that increasingly work alongside them. In this model, the knowledge base is not a passive filing system but an active component of the operations control plane, integrated into agent reasoning, surfaced in conversational interfaces, and continuously maintained against the reality of a changing estate.

The operational knowledge gap

20.2 Retrieval-augmented generation for operations

The mechanism that has made knowledge-augmented operations practically achievable at enterprise scale is Retrieval-Augmented Generation, introduced in the research literature by Lewis and colleagues in 2020 and since refined through extensive applied work in production systems [3]. The core insight of RAG is straightforward: rather than encoding all relevant knowledge into a model’s weights through pre-training or fine-tuning, retrieve the specific, relevant knowledge from an external store at the moment of each query, and incorporate it into the model’s reasoning context alongside the query itself. The model’s role shifts from being a repository of knowledge to being a reasoner over knowledge that is retrieved on demand.

The RAG architecture in an operational context has three principal components. First, the vector database, which stores high-dimensional numerical embeddings of operational documents—runbooks, post-incident reviews, architecture decision records, and the rest of the knowledge base described in Section 20.3. An embedding is a transformation of a piece of text into a point in a high-dimensional semantic space, positioned such that texts with similar meaning are close together and texts with dissimilar meaning are far apart. This transformation is performed by an embedding model—a neural network trained to map text to its position in that space. The vector database indexes these embeddings in a structure that supports efficient approximate nearest-neighbour search: given a query point, return the K nearest points in the embedding space, which correspond to the K most semantically similar documents in the knowledge base [3].

Second, the query and retrieval process. When an agent or operator poses a question—”how have we previously resolved memory pressure on the payments service?”—the query text is embedded using the same embedding model that was used to index the documents. The resulting query embedding is compared against the stored document embeddings, and the most semantically similar documents are retrieved. These retrieved documents are then included in the prompt context presented to the generation model alongside the original query, providing specific, organisationally grounded information that the model incorporates into its response.

Third, the generation model, which reasons over the retrieved context to produce a response. In an operational setting, the generation model may be a large language model such as IBM watsonx.ai’s foundation models, and the response may take the form of a natural language explanation, a structured action recommendation, or a structured data output for consumption by a downstream agent.

The distinction between RAG and fine-tuning is significant and deserves careful treatment. Fine-tuning—the process of continuing model training on an organisation’s specific data—has intuitive appeal as a mechanism for encoding operational knowledge. A model trained on the organisation’s runbooks, incident history and architecture documents would, in principle, be able to answer operational questions using that specific knowledge. In practice, fine-tuning has several properties that make it poorly suited to operational knowledge management. Operational knowledge changes frequently: runbooks are updated after incidents, architecture decisions are revised, services are decommissioned and new ones introduced. Each update to the knowledge base would require a new fine-tuning run—a process that is computationally expensive, time-consuming, and operationally risky if the updated model introduces regressions in behaviour. Fine-tuning also makes it difficult to attribute specific claims in the model’s output to specific source documents: the knowledge is distributed across model weights in a way that is not inspectable [4]. In regulated environments where every recommendation must be traceable to its source, this opacity is a significant governance problem.

RAG avoids both of these issues. The knowledge base can be updated continuously—new documents are embedded and indexed without touching the generation model. And every response is grounded in specific retrieved documents that are identifiable, attributable and citable. The retrieved documents can be included in the response alongside the generated text, enabling operators and auditors to verify that the recommendation is consistent with the source material. This provenance is not a secondary benefit; in sovereign operations, where every material action must be traceable to its authoritative basis, it is a core requirement.

The quality of retrieval is the critical determinant of RAG system effectiveness—a point that the survey literature on RAG architectures addresses extensively [4]. Three design decisions are primary. Chunking strategy determines how source documents are divided into the segments that are individually embedded and stored. Documents that are chunked too coarsely (entire documents as single chunks) produce embeddings that average over too much meaning, reducing retrieval precision. Documents chunked too finely (single sentences) lose the surrounding context necessary for a retrieved chunk to be interpretable. Research and practice have converged on a strategy of hierarchical chunking—preserving document structure through headers and sections, with overlapping window chunks within sections—as a generally effective approach for technical prose [4]. Embedding model choice determines the quality of the semantic space in which retrieval operates. General-purpose embedding models, such as those available through open-source frameworks, perform reasonably well on general text; models fine-tuned on technical or operational corpora perform better on the specific vocabulary and conceptual structures of operations documentation. For a sovereign enterprise, the embedding model must run within the same sovereign zone as the documents it indexes, precluding the use of external embedding API services for sensitive operational content. Metadata filtering allows retrieval to be scoped before the semantic search, using structured attributes rather than embedding similarity—retrieving only documents relevant to a specific service, a specific sovereign zone, or a specific document type. Without metadata filtering, a query about a payment service incident might retrieve semantically similar documents from an entirely different service domain, consuming context window space with irrelevant content.

RAG architecture for operational knowledge

20.3 Building the operational knowledge base

The question of what to index is not merely logistical; it reflects a judgement about what knowledge is operationally valuable. Indexing too little leaves gaps that agents and operators encounter at the worst possible moments. Indexing too much—pulling in every document the organisation has ever produced—degrades retrieval precision by populating the vector space with low-value content whose embeddings compete for proximity with the documents that actually matter.

For an enterprise operations context, the corpus that consistently delivers value spans several categories. Runbooks and standard operating procedures are the most immediately obvious candidate: these are the structured procedures that define how operators should respond to specific classes of event, how they should execute specific categories of change, and how they should handle specific regulatory requirements. In many organisations, runbooks exist in a range of formats—Markdown files in version-controlled repositories, Confluence pages, ServiceNow knowledge articles, Word documents in SharePoint—and their coverage of the estate is uneven. Indexing what exists, while making the gaps visible, is more useful than waiting for comprehensive coverage.

Post-incident reviews, when written with sufficient depth, are among the most valuable documents in an operational corpus. A well-structured post-incident review contains a timeline of events, a description of the triggering condition, a diagnosis of the root cause, an account of the investigation process (including false trails), a record of the actions taken and their effects, and a set of recommendations for preventing recurrence. Each of these elements carries operational knowledge that is difficult to acquire any other way: it is knowledge about how the specific systems of this organisation fail, and how those failures are resolved. Research on organisational learning in high-reliability organisations consistently identifies the systematic capture and application of failure experience as a primary source of performance improvement [5]. An incident management system that generates post-incident reviews but never makes them retrievable at subsequent incidents is squandering the primary mechanism of operational learning.

Architecture decision records encode the reasoning behind structural choices: why a particular database was selected, why a particular network topology was adopted, why a particular data flow was designed the way it is. These documents answer the question that operations engineers frequently encounter when a system behaves unexpectedly: “Why was it built this way?” Without this context, remediation proposals may inadvertently undo intentional design decisions, or may fail to recognise that a seemingly obvious fix was previously considered and rejected for reasons that remain valid. The ITIL 4 knowledge management practice emphasises this category explicitly, identifying documented design rationale as knowledge that is disproportionately valuable relative to the effort required to create it [2].

Change Advisory Board minutes and change records capture the collective judgement of the organisation about the risk profile of changes, the conditions under which specific changes were approved, and the post-implementation assessments of whether changes had the expected effect. An agent preparing a remediation proposal benefits considerably from knowing that a similar change was approved eighteen months ago, what mitigations were required at the time, and what the outcome was.

Capacity planning documents provide the context necessary to evaluate the feasibility of remediation options—knowing that a service is already operating at eighty percent of its provisioned capacity is relevant to any remediation that proposes increasing load on a dependency. Vendor documentation for components in the estate—particularly for components that are not well-represented in a general-purpose language model’s pre-training data, such as niche database products or proprietary network appliances—provides the technical reference without which an agent’s recommendations may be technically incoherent. Regulatory guidance from sector regulators, prudential authorities and data protection supervisors is often not adequately represented in general model pre-training; indexing the actual guidance documents, with appropriate metadata, allows agents and operators to retrieve the specific regulatory basis for or against a particular action.

The ingestion pipeline that converts these source documents into indexed, queryable content must handle significant format diversity. Documents arrive as PDFs, Markdown files, Confluence pages exported via API, ServiceNow knowledge article exports, Word documents and HTML. Each format requires a parser capable of extracting clean text while preserving document structure—headers, lists, tables—that carries structural meaning. PDF parsing is particularly demanding: scanned PDFs require optical character recognition, and even native PDFs often have table and column structures that naive text extraction scrambles. Libraries such as Apache Tika, pdfminer and the structured parsing capabilities within IBM Watson Discovery provide parser implementations that handle most of these formats reliably, though quality assurance of extracted content is a necessary step in any production pipeline.

After parsing, the chunking strategy described in Section 20.2 is applied, followed by embedding model inference to produce the high-dimensional vector for each chunk. These vectors, together with the chunk text and associated metadata, are written to the vector store.

Metadata enrichment deserves particular attention. Each chunk in the vector store should carry, at minimum, the source document identifier, the source system (Confluence space, ServiceNow knowledge base, Git repository, document management system), the author or owning team, the creation date, the most recent modification date, and the domain classification of the source document (runbook, post-incident review, architecture decision record, and so forth). For sovereign operations, two additional metadata fields are essential: sovereign zone relevance, indicating which sovereign zone or zones the document’s content applies to, and service scope, indicating which services or service categories the document addresses. These fields enable metadata filtering in retrieval, preventing a query about a Zone 1 payments service from retrieving highly semantically similar content that applies only to a Zone 2 HR system.

A further metadata field that merits consideration is a confidence or authority level for the document—a structured indicator of whether the content has been formally reviewed and approved, is a working draft, or is a historical document that may no longer reflect current practice. This field, used in retrieval ranking, allows the system to preferentially surface formally reviewed content while still making provisional content available when no authoritative equivalent exists.

Operational knowledge base ingestion pipeline

20.4 Keeping knowledge current

The knowledge base built in Section 20.3 will, without active maintenance, become a liability rather than an asset. Every time an engineer retrieves a runbook that describes a system configuration that no longer exists, or receives a recommendation based on a post-incident review that was superseded by later experience, or is guided by architecture documentation that predates a significant refactoring, the knowledge base is actively misleading rather than helping. The staleness problem is not hypothetical—it is the characteristic failure mode of knowledge management initiatives that invest heavily in initial corpus construction and insufficiently in ongoing maintenance.

Detecting staleness requires measurement. The primary signal is the retrieval-then-override pattern: situations in which an agent or operator retrieves a document, uses it in reasoning, and then either explicitly overrides the recommendation it contains or takes a substantially different action. This pattern is most clearly observed when an operator explicitly notes that a retrieved runbook is out of date, or when an agent’s recommended action—derived from retrieved content—is rejected by the human reviewer with a notation that the basis is outdated. Monitoring these events produces a ranked list of documents by override frequency—a direct indicator of which knowledge is least reliable.

A second staleness signal is age in the presence of system change. A runbook written three years ago for a Kubernetes cluster that has since been upgraded three major versions, had its networking layer replaced, and had its storage class architecture changed is almost certainly stale in some material respect. The operational knowledge base can be continuously compared against the change record: when a significant change is made to a component—detected through IaC change history in the version control system, or through Concert’s continuous topology discovery—the set of documents whose scope metadata includes that component is flagged for review. This does not assert that those documents are stale; it asserts that they should be verified.

Automated refresh triggers can close a significant portion of the staleness gap without requiring manual scheduling. Post-incident reviews, when they are generated, should be auto-ingested: the completion of a major incident should trigger a pipeline that indexes the post-incident review, links it to any related historical documents (retrieved automatically by semantic similarity), and makes the new document immediately available for retrieval. This is not merely operationally valuable; it is the mechanism by which the system learns from each incident automatically, rather than requiring a separate knowledge management step.

IaC changes provide a second category of automated trigger. When a Terraform module that manages a database cluster is committed to the version control system, the change detection pipeline identifies the component the module manages and retrieves the list of runbooks, architecture decision records and operational guides in the knowledge base whose service scope includes that component. These documents are placed in a review queue with a structured notation of what changed and an invitation to the owning team to verify whether the existing documentation remains accurate. In organisations with mature DevOps practices, this review can be incorporated into the pull request process: the IaC change PR is decorated with a comment identifying the knowledge base documents that may be affected.

Knowledge provenance is the mechanism that makes staleness detectable at the moment of retrieval, even when automated triggers have not yet flagged a document. Every retrieved chunk presented to an agent or operator carries, alongside its text, the metadata that describes its origin: who authored it, when it was last reviewed, which system it came from, and what its current confidence level is. An agent that retrieves a runbook authored three years ago and never subsequently reviewed is equipped to note this provenance in its response, alerting the operator to the need for verification before acting. A governance rule in the retrieval system can enforce minimum display of provenance metadata—it is never permissible for the system to present retrieved content as if it were current best practice without disclosing its age and review status.

The combination of automated triggers and provenance disclosure does not eliminate the need for scheduled reviews. High-usage documents—those retrieved frequently by agents and operators—should be subject to an explicit review cadence. Usage analytics in the vector store reveal which documents are retrieved most frequently and which are rarely or never retrieved. High-frequency documents are of disproportionate importance to operational quality and should be reviewed most often. Low-frequency documents may indicate either that the content is of limited relevance—suggesting candidates for archival—or that the content is very precise and rarely retrieved because it applies only to specific, infrequent situations—a distinction that requires human judgement to make.

Knowledge lifecycle management

20.5 IBM watsonx and the operational knowledge layer

IBM’s watsonx platform provides the foundation model and knowledge integration capabilities through which the RAG architecture described in the preceding sections is realised within the IBM sovereign operations stack. Understanding how these components compose—and how they enforce the sovereign zone constraints that are the book’s constant concern—is necessary for architects designing knowledge-augmented operations on IBM infrastructure.

IBM watsonx.ai serves as the foundation model layer: the platform through which large language models are invoked for both embedding generation and response generation. For operational knowledge retrieval, the embedding model selected must be consistent between indexing time (when documents are converted to embeddings and stored) and query time (when the query is converted to an embedding for nearest-neighbour search). Consistency here is not merely a performance property; it is a correctness property. An embedding model updated between indexing and query time may place the same text at different positions in the embedding space, causing retrieval to return incorrect results even for straightforward queries [3]. IBM watsonx.ai’s model versioning and deployment control allow the embedding model to be pinned to a specific version for an operational knowledge base, ensuring that updates to the model are accompanied by re-indexing of the corpus rather than silent degradation of retrieval quality.

For response generation, watsonx.ai provides access to IBM’s Granite foundation model family and to a range of partner models, all deployable within IBM’s sovereign cloud infrastructure. The choice of generation model for operational knowledge queries involves trade-offs between context window size—which determines how many retrieved chunks can be incorporated into a single prompt—and the model’s capability on structured technical reasoning tasks. IBM’s Granite models, trained substantially on technical and scientific corpora, tend to perform well on operational prose and structured data analysis [6].

IBM Watson Discovery provides a complementary capability: structured document corpus management with built-in parsing, enrichment and search, including keyword search and semantic search over a managed document collection. Watson Discovery is particularly valuable for the initial ingestion of large, heterogeneous document sets—it handles PDF extraction, table parsing, entity recognition and document classification at scale, with quality substantially higher than general-purpose text extraction. For an operational knowledge base, Watson Discovery can serve as a managed pre-processing layer, converting raw documents into clean, enriched text that is then embedded and indexed in the vector store. In this configuration, Watson Discovery handles the parsing and structuring challenges, while the vector store handles the semantic retrieval, and watsonx.ai handles the generation.

Watson Discovery also provides passage retrieval capabilities—returning the specific passages within a document that are most relevant to a query, rather than returning entire documents—which can supplement vector store retrieval for cases where the query is more factual than conceptual, and where exact phrase matching or entity recognition outperforms semantic similarity [7].

IBM watsonx Orchestrate, described in Chapter 17, integrates with the knowledge layer through its tool-calling architecture. Knowledge retrieval is expressed as a tool that Orchestrate agents can invoke: the tool accepts a query string and optional metadata filters as inputs, calls the retrieval pipeline, and returns a structured result containing the retrieved chunks, their provenance metadata and a confidence score. From the perspective of the orchestrating agent, knowledge retrieval is simply one of many tools available alongside ITSM integration, observability queries, IaC execution and topology inspection. This integration means that agents can interleave knowledge retrieval with other operations naturally within a workflow: investigate a metric anomaly, retrieve the relevant runbook, query the topology model to understand the dependency context, formulate a remediation plan, retrieve the relevant change management policy, and present a consolidated recommendation with full source attribution.

The sovereign AI constraint that applies throughout this book is particularly acute for the operational knowledge layer. The vector database and the language models used for embedding and generation must operate within the same sovereign zone as the documents they index. An operational knowledge base for Zone 1—containing runbooks, incident reviews and architecture decisions for regulated workloads—must be processed entirely by infrastructure resident in Zone 1. Shipping Zone 1 documents to an embedding API operating outside Zone 1’s jurisdictional boundary would violate the same sovereign constraints that apply to any other operational data. IBM’s sovereign cloud deployments—available in configurations that restrict both data residency and operational access to specified jurisdictions—allow the full RAG pipeline (vector store, embedding model, generation model) to be co-located within the same sovereign boundary as the documents they process [6].

This constraint has architectural implications. Organisations with multiple sovereign zones may need separate vector stores—one per zone, each indexed by an embedding model running within that zone—with zone-specific retrieval pipelines. Queries that span zones, where permitted by policy, require careful design to avoid cross-zone data movement: the appropriate pattern is to run separate retrievals in each zone and aggregate only the retrieval results (embeddings, chunk text and metadata) rather than moving the raw source documents across the zone boundary.

IBM watsonx operational knowledge architecture

20.6 Structured knowledge: the Concert topology model as knowledge graph

Vector databases answer semantic questions well, but they have a structural limitation: they retrieve by similarity, not by relationship. A question such as “what other services depend on this database?” is a graph traversal query, not a semantic similarity query. The most similar document to “what services depend on the payments database?” might be a runbook that discusses database dependency management in general terms, rather than the specific answer derived from the actual dependency graph of the estate. For questions about the structure of the environment—its components, their dependencies, their configurations, their current state—the Concert topology model, as a structured knowledge graph, is the appropriate source.

Concert, as described in Chapter 14 and Chapter 15, continuously discovers and maintains a graph of the operational estate: services, their infrastructure, their dependencies, their configurations, their compliance posture and their performance history. This graph is queryable through Concert’s APIs in ways that are both precise and current. A query for “which services depend on database X?” returns a specific, accurate list derived from observed runtime behaviour and declared configuration, not from a semantically similar historical document. A query for “what changed in the payments cluster in the last six hours?” returns a timestamped, attributed list of configuration drift events, IaC commits and manual interventions—not an approximation derived from text similarity.

The two knowledge modalities—vector database for semantic retrieval and Concert topology model for structured graph queries—are complementary rather than competitive. A useful mental model is the distinction between the two types of question that arise in operational contexts. Semantic questions ask about organisational knowledge: “How have we previously handled similar incidents?” “What does our runbook say about this situation?” “What regulatory guidance applies to this type of change?” These questions are answered well by RAG over the vector database—they require understanding of meaning and context, and the answer is typically a passage of text or a structured recommendation derived from prose sources. Structural questions ask about the state of the estate: “What services consume this API?” “Which version of this library is running in Zone 2?” “What is the current SLO status of the payments pipeline?” These questions are answered well by Concert’s topology model—they require precision and currency, and the answer is a graph query result derived from continuously updated operational data.

The most powerful knowledge architecture for autonomous operations combines both modalities in a unified query flow. When an agent begins investigating an incident, it queries Concert’s topology model to establish the structural context: which services are affected, what their dependencies are, what recent changes have occurred in the relevant component graph. Armed with this structural context, it then queries the vector database with semantically rich queries that are informed by the structural findings: “given that this incident affects the payments service and its Redis dependency, retrieve runbooks and post-incident reviews relevant to Redis memory management in payment service contexts.” The structural context from Concert makes the semantic query more precise; the semantic retrieval from the vector database provides the procedural and historical knowledge to guide action. Together, they deliver a quality of situational awareness that neither source alone can provide [8].

This combination also has important implications for knowledge base construction. Some knowledge is better represented as structured data in Concert’s model than as prose in the vector database. The dependency map of the estate, for instance, should not be maintained as a set of Markdown documents describing service relationships—it should be the live Concert topology graph, continuously updated through discovery. Prose documentation about service dependencies becomes stale faster than any review cadence can compensate; the Concert graph is stale only for the seconds between a change and its observation. A sensible knowledge architecture places structural facts in Concert’s model, maintained automatically, and places procedural, contextual and historical knowledge in the vector database, maintained through the ingestion and review processes described in the preceding sections.

The practical consequence for Concert integration is that agents querying the knowledge layer should call Concert’s topology API and the vector database retrieval tool as complementary steps in a single reasoning workflow, not as alternatives. Watsonx Orchestrate’s tool-calling architecture accommodates this naturally: Concert topology queries and vector database retrieval queries are both expressed as tools, invocable by the orchestrating agent in whatever sequence the reasoning task requires.

Dual-modality knowledge architecture

20.7 Measuring knowledge base quality

A knowledge base that is not measured is a knowledge base that is believed to be better than it is. The characteristic failure mode of unmeasured knowledge management programmes is that engineers stop trusting the knowledge base after a sequence of unhelpful or misleading retrievals, begin routing around it, and the programme quietly dies while its infrastructure continues to be funded. Measurement is the mechanism that converts this failure mode into a feedback loop—detecting degradation before it erodes trust, and providing evidence that improvement efforts are having the intended effect.

Four primary metrics characterise operational knowledge base quality. Retrieval relevance measures whether the documents returned by the retrieval pipeline are relevant to the query that triggered them. This metric is most precisely calculated through offline evaluation against a curated test set: a collection of queries (drawn from historical agent interactions, operator queries and incident tickets) each paired with a human-assessed list of relevant documents. Standard information retrieval metrics—Mean Reciprocal Rank (MRR), Normalised Discounted Cumulative Gain (NDCG) and Precision at K (P@K)—evaluate retrieval quality against this ground truth [4]. The test set should be refreshed periodically, as the query distribution evolves with the estate and the knowledge base.

Answer accuracy measures whether the knowledge base enables correct responses. Where retrieval relevance measures the quality of the retrieved documents, answer accuracy measures whether those documents support responses that are factually correct and operationally appropriate. This metric is harder to compute automatically—it typically requires human assessment, either through structured review of sampled agent interactions or through outcome tracking (did the action recommended by the knowledge-augmented agent produce the expected outcome?). In practice, a sampling approach is most common: a random sample of agent workflows in which the knowledge retrieval tool was invoked is reviewed by senior engineers who assess whether the retrieved knowledge was used correctly and whether the resulting recommendation was sound. Samples where the recommendation was overridden with a substantively different approach are flagged as potential accuracy failures and analysed for root cause.

Staleness rate measures the proportion of retrieval events that return documents whose content is materially out of date. This metric is computed from the override monitoring described in Section 20.4: every time a retrieved document is explicitly flagged as outdated, or every time a retrieved recommendation is overridden with a notation attributable to document staleness, the event is counted. The staleness rate is the ratio of these events to total retrieval events. A staleness rate above a threshold—which the organisation should set based on its tolerance for outdated knowledge—triggers a review of the freshness review cadence and the automated trigger configurations. A persistently high staleness rate in a specific document category (for example, runbooks for a recently refactored service domain) indicates a structural gap in the maintenance process that requires targeted remediation.

Coverage measures the proportion of common operational scenarios for which the knowledge base contains relevant, current content. Coverage is assessed by cataloguing the incident types, change categories and operational scenarios that the organisation encounters regularly, then checking whether each category has at least one relevant, sufficiently recent runbook or procedural document indexed. Coverage gaps—categories for which no useful content is retrievable—are operationally significant: they are the situations in which the knowledge base will fail to assist at the moments of greatest need. Coverage analysis is most valuable when framed in terms of incident frequency: a knowledge gap for a rare, low-impact scenario is less critical than a gap for a scenario that occurs monthly and causes significant service impact.

The human review loop is the mechanism that makes these metrics actionable. A knowledge quality review, conducted on a scheduled cadence—monthly in mature programmes, more frequently in early stages—examines the measurement data and drives three outputs. First, it identifies specific documents for expedited review, updating or archival based on their staleness and usage signals. Second, it identifies coverage gaps and assigns responsibility for creating the missing content. Third, it adjusts the automated maintenance parameters—review cadence, override sensitivity thresholds, automated trigger configurations—based on observed staleness patterns.

The human review loop also serves an organisational function that the automated metrics cannot. By requiring senior engineers to examine agent knowledge usage regularly, it maintains organisational awareness of what the knowledge base contains, surfaces inconsistencies between the knowledge base and current practice that engineers might otherwise not notice, and builds the habit of knowledge management as an ongoing operational practice rather than a one-time project. ITIL 4’s knowledge management practice articulates this function explicitly: effective knowledge management requires not merely the infrastructure to store and retrieve knowledge, but the organisational practice of continuously validating and improving it [2].

A useful instrument for the human review loop is a knowledge interaction dashboard—a live view of retrieval analytics that shows the most frequently retrieved documents in each category, the staleness distribution of retrieved content, the override rate by document and category, and the coverage assessment for each incident type category. This dashboard provides the input for review discussions and the output metrics that demonstrate programme value to operational leadership.

Knowledge base quality metrics dashboard

Key Takeaways

Bridge to Chapter 21 — AI Risk and Control

This chapter has examined the knowledge dimension of agentic operations: why organisations have a structural knowledge problem, how Retrieval-Augmented Generation provides a practical mechanism for delivering accumulated knowledge to agents and operators at the moment of need, what the operational knowledge base should contain and how it is built, how knowledge currency is maintained against a continuously evolving estate, how the IBM watsonx platform realises the knowledge layer within sovereign zone constraints, how the Concert topology model and the vector database serve complementary knowledge modalities, and how knowledge base quality is measured and improved over time.

The knowledge-augmented operations architecture described in this chapter does not stand alone. It is one component of the broader agentic operations control plane: it provides the knowledge layer that planner agents consult when forming hypotheses, that executor agents draw on when selecting remediation approaches, and that reviewer agents reference when evaluating proposed actions. Its effectiveness depends on the observability infrastructure described in Chapters 7 through 9, the topology model maintained by Concert (Chapter 14 and Chapter 15), the agent orchestration patterns described in Chapter 18, and the ITSM integration described in Chapter 19. Each layer enriches the others.

Chapter 21 turns to the governance dimension that has been present as a background concern throughout this chapter: AI risk and control. The recommendations produced by knowledge-augmented agents are based on retrieval results that may be stale, incomplete or misleading. The generation models that synthesise those recommendations may produce outputs that are confident but incorrect. The agents that act on those recommendations may have a scope of authority that is too broad or too narrow. Managing these risks systematically—not by avoiding the technology but by applying the disciplines of AI risk management to its operational deployment—is the subject of the next chapter. The knowledge quality framework described in Section 20.7 is, in a sense, a precursor to the broader AI risk management framework Chapter 21 will establish: the habit of measurement, human review and continuous improvement that makes knowledge-augmented operations trustworthy is the same habit that makes agentic operations generally safe to extend and rely upon.

References

[1] C. Rankin, J. O’Brien, and M. Sheridan, “The hidden cost of incident response: information retrieval latency and its operational impact,” IEEE Transactions on Network and Service Management, vol. 19, no. 3, pp. 2871–2883, Sep. 2022. [Online]. Available: https://doi.org/10.1109/TNSM.2022.3180397

[2] AXELOS, ITIL 4: Practice Guide — Knowledge Management, AXELOS Ltd., London, UK, 2020. [Online]. Available: https://www.axelos.com/certifications/itil-service-management/what-is-itil

[3] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-T. Yih, T. Rocktäschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in Advances in Neural Information Processing Systems, vol. 33, H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H. Lin, Eds. Curran Associates, Inc., 2020, pp. 9459–9474. [Online]. Available: https://doi.org/10.48550/arXiv.2005.11401

[4] Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” arXiv preprint, arXiv:2312.10997, Dec. 2023. [Online]. Available: https://arxiv.org/abs/2312.10997

[5] K. E. Weick and K. M. Sutcliffe, Managing the Unexpected: Sustained Performance in a Complex World, 3rd ed. Hoboken, NJ: Wiley, 2015. [Online]. Available: https://www.wiley.com/en-us/Managing+the+Unexpected%3A+Sustained+Performance+in+a+Complex+World%2C+3rd+Edition-p-9781118862414

[6] IBM, IBM watsonx.ai: Foundation Models and AI Studio, IBM Documentation, IBM Corp., Armonk, NY, USA, 2024. [Online]. Available: https://www.ibm.com/docs/en/watsonx/saas

[7] IBM, IBM Watson Discovery: Document Understanding and Information Retrieval, IBM Documentation, IBM Corp., Armonk, NY, USA, 2024. [Online]. Available: https://www.ibm.com/docs/en/watson-discovery

[8] IBM, IBM Concert: Operational Topology and Service Intelligence, IBM Documentation, IBM Corp., Armonk, NY, USA, 2024. [Online]. Available: https://www.ibm.com/docs/en/concert

[9] J. Liu, “LlamaIndex: Data framework for LLM applications,” LlamaIndex, 2024. [Online]. Available: https://docs.llamaindex.ai

[10] H. Chase, “LangChain: Building applications with LLMs through composability,” LangChain, Inc., 2024. [Online]. Available: https://python.langchain.com/docs/introduction/

Sovereign Cloud Operations: AI-Driven Management of Sovereign Estates © 2026 by Alan Hamilton is licensed under CC BY-SA 4.0