On-Premise RAG for AI Avatars: A Secure Knowledge Architecture

An on-premise AI avatar should never be able to retrieve more information than the person speaking to it. Private hosting protects the infrastructure boundary, but it does not automatically preserve document permissions inside the retrieval pipeline. A secure design authenticates the user, applies source permissions before any material reaches the language model, cites the evidence used and removes access promptly when a permission changes.

That is the practical standard for retrieval-augmented generation, or RAG, in a bank, government department or regulated enterprise. The useful equation is:

Effective avatar access = authenticated user access ∩ service policy ∩ approved use-case scope.

If any one of those controls says no, the content should not enter the model context. This guide explains how to build that boundary, where on-premise deployment helps, and what an architecture team should test before approving a pilot.

What on-premise RAG actually does

RAG connects a generative model to information outside its original training data. When a user asks a question, the system searches approved knowledge sources, selects relevant passages and gives those passages to the model as context for an answer. An avatar adds speech recognition, text-to-speech, visual rendering and conversational behaviour around that process.

A typical interaction therefore crosses several stages:

1. Authenticate the user and establish the session.
2. Transcribe or interpret the request.
3. Authorise the request against policy and source permissions.
4. Retrieve and rank permitted evidence.
5. Generate an answer grounded in that evidence.
6. Return citations, speak the answer and render the avatar.
7. Record the minimum operational evidence required for audit and support.

Running those components on customer-controlled GPUs can keep prompts, retrieved passages, embeddings and generated responses inside an approved environment. It may also support stricter network and operational controls. But location is only one layer. A local vector database populated through an over-privileged service account can still expose restricted information perfectly efficiently.

A seven-layer secure knowledge architecture

The safest design treats retrieval as an identity and data-governance system, not simply a similarity search feature.

1. Authoritative knowledge sources

Begin with the systems that already own the information: document repositories, policy libraries, case-management platforms, databases, approved websites and operational systems. Record an owner, classification, retention rule and permission model for each source. “The knowledge base” is not a sufficient inventory when different repositories have different legal and operational boundaries.

2. Controlled ingestion

The ingestion service extracts text, divides it into retrievable chunks and creates embeddings or other search representations. It should reject unsupported formats, scan content according to the organisation’s security policy and attach source metadata to every chunk. At minimum, preserve the source identifier, version, owner, classification, effective date and deletion status.

For restricted environments, the embedding model and indexing process may also need to run locally. The restricted-network design guide explains why apparently small external dependencies can undermine an otherwise private architecture.

3. Permission metadata and synchronisation

If content is copied into an index, its access rules must travel with it. That can include users, groups, roles, departments, cases, regions, classifications or explicit deny rules. Permission metadata must remain linked to the authoritative source rather than becoming a one-time snapshot.

Microsoft’s current guidance for document-level access control in Azure AI Search describes both permission-filter metadata and query-time identity enforcement. It also highlights the operational issue buyers should care about: permission changes must be synchronised, and a delay can exist between changing a source permission and updating the searchable index. The same risk applies to any copied index, regardless of product.

4. The private retrieval store

The vector or hybrid search store should be segmented and encrypted according to the organisation’s policy. Separate collections may be appropriate where legal, national or operational boundaries are stronger than ordinary user permissions. Index administrators should not automatically inherit the right to read source content, and backup copies need the same classification as the live index.

5. Query-time identity and security trimming

At query time, the avatar application passes a verified identity or narrowly scoped entitlement to the retrieval service. The service removes unauthorised candidates before they become model context. It should not retrieve broadly and ask the language model to ignore forbidden passages; by then the data has already crossed the control boundary.

Use short-lived credentials and service identities where possible. A shared, high-privilege connector is useful for ingestion only when its output remains correctly permissioned. It is not a substitute for the user’s identity during retrieval.

6. Generation and policy orchestration

The language model receives only authorised evidence plus the instructions required for the use case. Retrieved documents should be treated as untrusted data, not as system instructions. That distinction matters because a compromised or carelessly authored document can contain text designed to override the assistant’s rules.

The OWASP Top 10 for LLM applications identifies prompt injection and sensitive-information disclosure among the main risks. Separate system instructions from retrieved content, restrict tools independently of the model, and require additional policy checks before any consequential action. This also follows the NCSC’s secure-design guidance to apply least privilege, threat modelling and secure defaults to AI systems.

7. Avatar, evidence and handover

The visible avatar should not obscure the evidence chain. For material answers, the interface should offer a readable source, version or effective date rather than relying on a confident voice. It also needs a safe “I do not know” path, a way to clarify identity or scope, and a handover route when a request requires a human decision.

Choose between copied, live and hybrid retrieval

There are three useful patterns. None is universally best.

Copied index with permission metadata

Content is processed into a local search index, including its access rules. This can reduce latency, support disconnected operation and make retrieval capacity easier to control. The trade-off is freshness: document changes, group membership and revocations must propagate reliably. It also creates another governed copy to retain, back up and delete.

Live retrieval from the source

The system queries the authoritative repository using the user’s identity or an approved delegated token. This can preserve current permissions and avoid a large secondary copy. It may add latency, increase dependency on the source system and complicate ranking across multiple repositories.

Hybrid retrieval

Stable, widely permitted material is indexed locally, while highly sensitive or fast-changing information is queried at source. This often gives architecture teams a better control point than forcing every source into one pattern. The routing rule should be explicit and auditable.

Whichever model is selected, define a permission-revocation objective. If a user leaves a role at 10:00, when must their indexed access disappear? What happens if the synchronisation service is unavailable? For sensitive sources, denying access while permission state is uncertain is usually safer than serving a potentially stale entitlement.

RAG improves grounding, not certainty

Retrieval can make answers more current and attributable, but it does not eliminate hallucination. The wrong passage can be retrieved, an obsolete policy can outrank a current one, or the model can overstate what the evidence supports.

A production knowledge architecture therefore needs more than embeddings:

• hybrid lexical and semantic search where exact terms matter;
• version and effective-date controls;
• document-level and passage-level provenance;
• reranking appropriate to the domain;
• conflict handling when two sources disagree;
• an answer threshold below which the avatar declines to speculate; and
• citations that a user can inspect outside the spoken interaction.

For multilingual services, test retrieval separately from speech and generation. A good Arabic voice does not prove that an Arabic query finds the correct bilingual policy passage. The Arabic AI avatar evaluation guide sets out a pipeline-based acceptance method for dialects, code-switching and local deployment.

Control latency without weakening access decisions

A real-time avatar has a longer critical path than a text search: identity, speech recognition, permission filtering, retrieval, reranking, generation, speech synthesis and rendering all contribute to the response. Teams should measure each stage separately before buying more hardware.

Pre-computed embeddings, local inference, parallel processing and carefully scoped caches can help. But a cache must never return an answer created under another user’s broader permissions. Cache keys need to account for the effective permission set, content version and policy state, or caching should occur only below the sensitive boundary.

Capacity decisions should use representative secured queries, not an unfiltered demonstration. Yepic’s GPU capacity-planning framework explains how concurrency, tail latency, reserve and the full media pipeline affect a private deployment.

What Yepic’s enterprise work demonstrates

In the Abu Dhabi Aviation and Oracle enterprise avatar integration, Yepic helped connect a real-time avatar to enterprise information so authorised users could query operational data, receive role- and clearance-aware answers, view charts and hear the result narrated. The delivery also involved development and production environments, API and iframe integration, WebRTC and network testing, browser remediation, cybersecurity support and ongoing maintenance.

That project is relevant evidence for identity-aware enterprise integration and the operational work surrounding a governed avatar. It should not be misrepresented as proof of a universal boxed RAG product or a completed customer-hosted deployment. Yepic’s private and on-premise work is a scoped implementation: source systems, identity, models, GPUs, networking and governance requirements determine the final architecture.

Twelve tests to run before production

1. Different permissions: two users ask the same question and receive appropriately different evidence.
2. Direct extraction: a user explicitly requests restricted material and the retrieval layer blocks it before generation.
3. Revocation: a user is removed from a group and access disappears within the agreed objective.
4. Deletion: a removed source and all of its chunks disappear from search, caches and recoverable copies as policy requires.
5. Conflicting versions: an obsolete policy does not outrank the effective version without warning.
6. Retrieved prompt injection: malicious instructions inside a document do not alter system policy or enable tools.
7. Evidence: the interface supplies the source, version and effective date for a material answer.
8. No authorised result: the avatar declines to answer rather than filling the gap from general model knowledge.
9. Session isolation: transcripts, context and caches do not cross identities or permission sets.
10. Source failure: the service degrades safely when a repository or permission service is unavailable.
11. Multilingual equivalence: equivalent questions in approved languages retrieve equivalently permitted evidence.
12. Audit reconstruction: reviewers can identify the identity, policy decision, source versions and model configuration used without retaining unnecessary customer content.

Connect those tests to the local-first telemetry controls in the on-premise AI observability guide. A private system still needs enough evidence to diagnose a permission failure without routinely exporting conversations or sensitive passages.

Start with a permission map, not a document upload

Before indexing an enterprise repository, select one bounded workflow. Map the users, groups, source systems, classifications, deny rules, revocation paths and permitted actions. Then build a small evaluation set that includes legitimate questions, ambiguous requests, adversarial prompts and pairs of users with different access.

This produces a more useful pilot than uploading every document and admiring plausible answers. It tests the commercial promise that matters: a responsive, emotionally intelligent avatar that can use enterprise knowledge without quietly becoming a new route around enterprise controls.