Evaluating an Agent Knowledge Layer: Retrieval, MCP, and Provenance

Key takeaways

• Test identifiers separately from paraphrased questions: Lexical and dense retrieval can complement each other, but tokenization, filtering and rank fusion affect whether the required passage survives. Compare exact-field lookup and hybrid retrieval against representative queries from your actual workload. Elasticsearch hybrid search
• Model Context Protocol (MCP) provides a shared integration interface: Standardized MCP servers can reduce bespoke connector development across client environments (such as Claude Code and Cursor), though client-side configuration, tool semantics, authentication, and operational ownership remain required for each client.
• Document provenance and rollback separate canonical stores from derived search indexes: Enterprise audit trails demand tracking document revisions, lineage, and retrieval freshness. Systems vary between Git-backed version histories, metadata catalogs, and vector-store snapshot backups. Durable vector databases can function as derived search indexes in this architecture rather than simple transient caches, though not every vector database is inherently a derived index.
• Index synchronization and consistency models dictate stale-answer risk: Vector updates across decoupled systems are frequently eventually consistent, and stale answers also depend on source text, cache state, and context assembly. Storing canonical records in structured stores (such as PostgreSQL) aids reindexing, provided retained source text, configuration, and recovery verification are maintained.
• Deployment isolation depends on the complete data flow: A self-hostable knowledge store, including AKB, is one component of a deployment. Offline operation also requires local inference, embeddings, authentication and operational dependencies. A VPC or local database alone does not establish that boundary.

Retrieval strategy selection: moving beyond standalone vector search

An agent knowledge layer must retrieve exact identifiers, preserve qualifications in tables and documents, and expose the evidence behind an answer. Start by testing those tasks on your own material, then compare how candidate systems handle shared access, source updates, and recovery.

Consider an agent processing a technical query such as: "How do I resolve an ERR_VAL_409 error on billing webhooks?"

• Dense Vector Branch: Evaluates semantic intent to identify documents and runbooks discussing billing webhook error handling.
• Lexical Branch: Searches indexed terms associated with ERR_VAL_409. Inspect the analyzer: punctuation, case normalization or token splitting can change what is matched. An exact-match field may be appropriate when identifier equality is required.
• Rank Fusion: Combines results using Reciprocal Rank Fusion (RRF) or weighted blending to generate a single ranked result list. Note that final $K$ is the returned result count rather than a universal candidate limit; retrieving equal final $K$ result counts across dense, lexical, or hybrid branches does not imply equal branch compute or candidate evaluation cost. Record dense and sparse candidate retrieval depths separately.

To solve this, modern knowledge backends deploy hybrid retrieval pipelines combining lexical (sparse) matching and dense vectors, with optional graph-based context references:

1. Lexical BM25 and Sparse Retrieval: Lexical ranking scores indexed terms, while exact equality lookup may require a separately configured keyword field. According to Elasticsearch documentation, combining exact-term and meaning-based retrieval into a single ranked list helps balance precision with recall.
2. Dense Semantic Embeddings: Captures synonyms, colloquial descriptions, and multi-sentence context where exact terms do not overlap.
3. Fusion and Reranking Algorithms: Systems combine lexical and dense rank lists through techniques like Reciprocal Rank Fusion (RRF) or weighted linear combinations. In primary information retrieval research, Cormack et al. (SIGIR 2009) reported in Table 2 a Mean Average Precision (MAP) of 0.3686 for Reciprocal Rank Fusion compared to 0.3586 for the best individual submitted run on the TREC Robust dataset. This reflects historical research on TREC submitted runs rather than an AKB benchmark, current product ranking, or guaranteed deployment gain; additionally, MAP averages per-query average precision and is distinct from Hit@K, Recall@K, MRR@K and NDCG@K; record the evaluation convention and cutoff. Elasticsearch describes result fusion as one approach to combining lexical and semantic retrieval. Evaluate candidate windows and ranking settings on the actual workload.
4. Document Relationship Graphs: Interconnected enterprise documents, such as design specs citing API contracts or policies referencing appendices, may offer complementary relationship paths. Graph traversal is an optional complementary context mechanism rather than an inherent requirement of hybrid dense/lexical retrieval. When an agent retrieves one section, following its references may supply a missing qualification. Test which relationships the implementation follows and which remain the application's responsibility.

Different platforms expose different controls over this pipeline:

• Elasticsearch: Documents lexical and semantic retrieval with result fusion. Examine how query analyzers, filters and the selected fusion method affect the same test set.
• Dnotitia AKB: Documents hybrid keyword and semantic retrieval with explicit knowledge relationships. Evaluate the chosen vector driver and embedding configuration. Test degraded operation directly; do not infer service uptime from the presence of a lexical branch.
• Pinecone: Documents approaches to dense and sparse retrieval. Select the applicable index/API configuration before comparing results with another system.
• Enterprise context and search products: Products such as Atlan and Glean address organizational context and discovery. Ask how the proposed offering applies source permissions, exposes supporting evidence and handles exact identifiers; do not infer low-level ranking behavior from the category name.

Handling structured hierarchy and tables in enterprise corpora

A retrieval-augmented generation (RAG) pipeline may split documents by a fixed character or token count. In technical manuals, financial reports and policies, a boundary can separate evidence that needs to be read together:

• Table Fragmentation: Splitting a table can separate values from column headers or units, making their interpretation unreliable unless that context is preserved.
• Clause Isolation: Exception clauses (e.g., "This policy applies to all tiers except Enterprise SLA customers") can be cleaved from their parent headings, leading agents to generate contradictory assertions.
• Loss of Structural Lineage: When an agent inspects a third-level subheading (### Remediation Steps), check whether the document title and major section header (## Database Failover) remain available in the chunk text, metadata or retrieval response.

When designing parsing and structural pipelines, teams should scope hierarchy parsing, table extraction, and regression coverage to the document formats and layouts present in their actual workload, rather than assuming every deployment requires every extraction strategy. To prevent structural loss, teams evaluate hierarchical ingestion architectures:

Metadata-governance products and agent knowledge stores organize context in different ways. Atlan describes business definitions, lineage and governance capabilities; confirm the versioning behavior of the relevant asset type. AKB documents documents, tables, files and explicit relations, with canonical text and metadata in PostgreSQL. Evaluate the required hierarchy and recovery behavior on the selected release.

Shared agent backend architecture: Model Context Protocol (MCP)

As development teams adopt multiple AI interfaces, ranging from command-line coding tools like Claude Code to IDE extensions like Cursor and Windsurf, maintaining separate knowledge plugins and context pipelines for each tool introduces maintenance friction and permission sprawl.

The open Model Context Protocol (MCP) provides a standardized client-server protocol for connecting agents to external context and data sources. While an MCP server can reduce redundant connector development across client environments, client configuration, tool semantic definitions, authorization policies, and operational ownership remain required for each client integration.

An MCP implementation connects clients and tools across primary operational roles:

1. Client Tier: IDEs (Cursor), CLI tools (Claude Code), and automated scripts communicate using uniform tool invocations over Streamable HTTP or stdio transports. Note that authentication and credential management differ between local stdio process environments and Streamable HTTP endpoints.
2. Protocol & Service Integration: An MCP server receives incoming requests, handles authorization, and dispatches queries across knowledge tools. Deployments may use a centralized token gateway as an example authentication pattern, though it is an architectural option rather than a protocol-mandated requirement.
3. Storage Tier: The backend orchestrates queries against relational databases, derived vector search indices, and versioned file systems.

For a shared-agent implementation, distinguish the service interface from the client configuration:

• AKB documents MCP access for Claude Code, Cursor and custom agents. Use the documented akb-mcp client proxy or supported HTTP configuration for the selected release, and check the caller's credentials and vault access.
• Elasticsearch and Pinecone document MCP interfaces. Compare the actual exposed operations, authorization requirements and intended client support rather than assuming that every MCP server exposes the same tool set.
• Glean agents as tools illustrate an additional boundary: an MCP tool can invoke a configured agent, rather than directly expose the underlying document store. Inspect the evidence and permissions at both layers.
• Claude Code acts as a client for external MCP services. Project configuration can share connection definitions, but sharing a definition does not grant every developer access to the service.

Test a read and an authorized update from each intended client. Then repeat with a caller who should be denied. Transport compatibility alone does not establish equivalent retrieval behavior or authorization.

Document provenance, auditability, and rollback

When AI agents use enterprise documentation to generate code, draft policy summaries, or answer compliance queries, knowing the provenance of each retrieved passage is essential. Unversioned vector caches create compliance risks: if an internal policy changes, stale embeddings may persist in vector indices, causing agents to quote superseded procedures.

Auditability and recovery depend on two synchronized workflows:

• Document Ingestion Pipeline: Canonical documents stored in primary relational databases or Git repositories feed downstream indexing processes that build derived vector embeddings and lexical text indexes.
• Audit & Rollback Pipeline: Version history records content changes. Separately configured service logs may record operations and access decisions; verify event coverage, retention and restore procedures rather than assuming every action is captured.

To maintain an auditable system, knowledge backends implement varying approaches to versioning and provenance:

Canonical storage versus derived indexes

A resilient architecture separates the canonical record of truth from the derived search index. Durable vector databases can function as derived search indexes in this architecture rather than simple transient caches, though not every vector database is inherently a derived index; stale answers can also arise from un-invalidated prompt caches, source text changes, or context assembly logic. In Dnotitia AKB, canonical documents, metadata, and lexical terms reside in a structured PostgreSQL database with Git-backed version history, allowing engineers to view file-level commit histories. If a derived vector index is corrupted or lost, rebuilding it from canonical storage requires retained source text, pipeline configurations, and post-recovery retrieval verification rather than assuming a database label guarantees automatic restoration.

Semantic provenance and lineage tracking

Record which source version and passage supports an answer. Provenance links retrieved evidence to a specific source revision, but linking evidence does not automatically prove that a generated assertion is logically entailed by that source text. A document timestamp or relationship timestamp can help identify stale evidence, but does not establish that a retrieval engine automatically changes its ranking. During rollback or content withdrawal, applications must recheck current caller access permissions and confirm that withdrawn evidence is removed from active indices and new retrieval and model-context paths, distinguishing withdrawn content no longer served from retained history/backups.

For AKB, Git-backed history makes content changes inspectable. The documented MCP audit facility is optional; coverage depends on which calls traverse it and how it is configured. Hash chaining can support tamper detection, but is not an immutable retention policy. Establish protected external retention and access controls when an audit requirement calls for them. AKB documentation

Client-local history and server records

A coding client's local session history and file checkpoints serve a different purpose from server-side provenance. Confirm where the selected client stores them, who can read them and what survives cleanup. Do not use a successful local rollback as evidence that the shared knowledge store retained the same source version.

Backup and consistency realities

Pinecone documents update consistency separately from security and recovery capabilities. Check the applicable index and offering rather than assuming one backup mechanism or consistency model applies everywhere. For the selected service, measure write-to-read visibility and verify the documented consistency guarantees. Do not generalize one vendor's behavior to every vector index.

Access control, permission scoping, and network isolation

Enterprise knowledge architectures require strict access governance to ensure agents cannot retrieve documents outside their authorized operational boundaries.

Organizations should evaluate isolation across three architectural layers:

1. Caller authorization: Identify how the service authenticates a user or application and decides which records it may access. A user identifier or namespace is an addressing mechanism, not proof that another caller cannot supply the same value.
2. Query and storage boundaries: AKB documents scoped vault access. Test allowed reads and writes, denied cross-vault requests and administrative access in the chosen deployment. For other databases, establish whether filtering is enforced by the engine, injected by trusted application code, or both. AKB documentation
3. Network dependencies: Draw the data flow through parsing, inference, embeddings, storage, authentication, telemetry and backups. AKB documents self-hosting; a disconnected workflow still requires all of these dependencies to operate within the intended network boundary. A private-cloud data plane likewise does not establish offline operation. AKB repository, Pinecone BYOC

Keep security tests separate from content recovery tests. A durable record can still be exposed through an over-broad credential, and a correctly denied request does not prove that backups can be restored.

Acceptance testing and evaluation methodology

Before adopting a knowledge layer, engineering teams must validate retrieval recall, ranking precision, and latency on their own internal corpora rather than relying exclusively on public benchmarks.

A structured evaluation workflow includes:

1. Constructing a representative test corpus

A robust evaluation set should be scoped to the organization's actual workload and document formats rather than assuming every environment contains all file types. It should include representative samples of:

• Alphanumeric queries (error strings, model numbers, SKU identifiers).
• Multi-page documents with deeply nested subheadings.
• Data tables requiring row/column value extraction.
• Cross-referenced policy documents requiring link traversal.

Define the evaluation unit (such as passage chunk, section, or document), authorized ground truth dataset, deduplication rules, and category/sample counts across the corpus.

2. Measuring retrieval metrics across strategies

Evaluate dense, lexical and hybrid retrieval while maintaining the same documents, queries, access filters, and ground truth scope. Record dense and sparse candidate retrieval depths separately, noting that final $K$ is the returned result count rather than a universal candidate limit:

• Hit@K: The fraction of answerable queries for which at least one judged relevant result appears within the top $K$ retrieved candidates.
• Recall@K: Calculated per query as (relevant retrieved in top $K$) / (total judged relevant in ground truth for that query), averaged across all answerable queries.
• Mean Reciprocal Rank (MRR@K): The mean across queries of the reciprocal rank of the first relevant retrieved result (1/\\text{rank}), evaluated as $0$ if no relevant result appears within top $K$.
• NDCG@K: Normalized Discounted Cumulative Gain at $K$, which evaluates ranked result quality using graded relevance judgments normalized against the ideal DCG (IDCG).

Evaluate unanswerable queries (to test rejection/abstention) and unauthorized exposure checks (permission enforcement) as separate verification tests rather than combining them into standard relevance retrieval metrics.

Record the release, embedding model, vector driver, tokenizer, fusion settings, reranker and corpus version. Compare retrieval results separately from generated-answer quality: a relevant passage in the top five does not prove that an answer used it correctly. Report observed results for this configuration rather than importing an unscoped vendor percentage.

3. Measuring update latency and failure modes

Test index behavior during updates:

• Measure how quickly edits to a source document appear in search queries.
• Verify fallback behavior when external embedding APIs encounter rate limits or outages (e.g., verifying whether systems support lexical fallback).
• Inspect audit logging coverage to confirm that retrieval calls and permission denials are recorded.

Knowledge backend comparison

The following matrix separates documented interfaces from the tests needed to choose among them:

For AKB, the core uses Business Source License 1.1. Its conditional production-use grant applies below 100 Named Seats, aggregated across the related entities and deployments defined in the license. Production use at 100 or more aggregated Named Seats, or offering AKB to third parties as a hosted, embedded or rebranded service at any seat count, requires a separate commercial license. The client proxy has separate MIT licensing. Check the LICENSE shipped with the selected release for exact definitions and conditions. AKB LICENSE

Frequently asked questions

Why do vector embeddings fail on exact part numbers or error codes?

Dense retrieval does not guarantee character-level equality. Performance on identifiers depends on the embedding model, tokenization, indexed context and ranking settings; some models may retrieve familiar codes successfully. Lexical ranking can help, but its analyzers may also split or normalize identifiers. Test representative codes and near-miss negatives, and use a configured exact-match field when equality is required.

Can an MCP server be shared across Claude Code and Cursor simultaneously?

Yes, when the server supports the intended clients, transport and access configuration. Connect both clients to the same service, verify the operations each exposes, and test concurrent reads and authorized writes. A shared endpoint does not remove the need for individual credentials, permissions and update-conflict handling.

What is the difference between document-level versioning and passage provenance?

Document-level versioning tracks changes to an entire source file over time (e.g., commit hashes in Git). Passage provenance provides evidence pointers to check a specific chunk, line number, or bounding box within a document revision, though evidence pointers do not automatically prove that every generated assertion is logically entailed. In regulated environments, passage provenance allows human auditors to verify the source text without rereading entire documents.

How does canonical database storage improve vector index recovery?

Vector indexes can experience corruption, schema alterations, or drift across model updates. When systems maintain raw text, metadata, and token indexes in a canonical database (such as PostgreSQL), a derived search index can be reconstructed from those records using the configured processing and embedding pipeline. Canonical recovery requires retained source text, pipeline configuration, and post-recovery verification to confirm acceptable retrieval behavior, rather than relying on a database label alone.

Is hybrid search necessary if our knowledge corpus is small?

Corpus size alone does not decide the retrieval method. Include specific names, configurations and numeric codes in a small evaluation set, then compare lexical, dense and hybrid results. Retain the simplest approach that meets the requirements and rerun the test when the corpus or processing configuration changes.