AI Configuration Drift: Keep Private Avatars in Sync

AI configuration drift occurs when a running avatar system no longer matches the version that was reviewed, tested and approved. In a private deployment, the difference may be a prompt edited through an administration console, a retrieval threshold changed on one node, an automatically selected inference profile, an expired certificate, a policy file that failed to reach a kiosk, or a model service that started with a different default.

For a bank, government department or regulated enterprise, the answer is not simply “put the configuration in Git”. Teams need a machine-readable baseline for the complete avatar service, a trustworthy view of effective runtime state, clear rules for permitted environmental differences, and a controlled method to reconcile or approve every deviation.

The practical output is a configuration-control register. It should connect each configuration item to an owner, approved value, runtime evidence, drift severity, remediation rule and production test.

Configuration drift is not model drift

The terms are easy to confuse:

• Configuration drift is a difference between intended and effective system settings.
• Model or data drift is a statistical or behavioural change caused by changing inputs, populations or relationships.
• Release change is an approved move from one known baseline to another.
• Compromise is an unauthorised change or manipulation. It may first appear as configuration drift, but not every drift event is malicious.

This article addresses the first problem. Behaviour monitoring remains necessary because two technically identical configurations can still produce different results through nondeterminism, external service changes or changing data.

NIST defines a baseline configuration as a reviewed and agreed set of specifications that is changed through control procedures and used as the basis for future builds and releases. That is a useful starting point, but an avatar baseline has more layers than a conventional web application.

Baseline the whole conversational system

A real-time avatar combines visual identity, live media and AI reasoning. Inventory at least nine configuration surfaces.

1. Avatar identity: approved likeness asset, animation model, disclosure, visual behaviour and permitted channels.
2. Speech: language detection, speech recognition, voice identity, pronunciation, synthesis settings and interruption behaviour.
3. Language and policy: model identifier, inference parameters, system instructions, safety rules, response limits and escalation policy.
4. Knowledge: source collections, index version, chunking, retrieval thresholds, freshness rules and permission filters.
5. Tools: tool schemas, allowed actions, approval requirements, scopes, endpoints and idempotency policy.
6. Runtime: container images, libraries, inference engine, precision, model profile, GPU allocation, drivers and resource limits.
7. Media and network: signalling, codecs, bitrate, TURN routes, certificates, origins, firewall rules and proxy behaviour.
8. Security and data: authentication, tenant mapping, egress policy, encryption references, retention, redaction and session timeouts.
9. Operations: logging level, metrics, alerts, feature flags, degradation modes, support access and rollback target.

Yepic’s AI bill of materials guide inventories what the system depends on. The configuration baseline answers a different question: which approved combination should be running here, now?

Create one release bundle with controlled overlays

Build a versioned release bundle that identifies the approved configuration for the complete service. It should contain or reference:

• a unique release and policy version;
• immutable model, image and dependency identifiers rather than moving labels;
• canonical prompt, guardrail, tool and retrieval configuration;
• hashes or signatures for controlled artefacts where appropriate;
• the permitted environment overlay for development, test, production, region or channel;
• required secret, certificate and key references without storing the secret values in the bundle;
• hardware and runtime compatibility constraints;
• approval, test evidence and deployment window; and
• the known-good rollback bundle.

Do not duplicate complete configurations for every environment. A copied production file soon becomes a second source of truth. Keep a shared base and small, explicit overlays for differences such as endpoints, GPU pools, replica counts, data regions and logging destinations.

Some variation is legitimate. Development may use synthetic knowledge, production may disable debug logging, and a kiosk may have a different microphone profile from a browser. Drift is the difference that remains after applying the approved overlay, not any difference at all.

Compare desired state with effective state

A configuration repository shows intent. It does not prove what a running process loaded.

Collect effective state from the runtime itself. That may include the resolved image digest, model and inference profile, loaded prompt hash, active feature flags, retrieval index, policy version, tool registry, certificate identity, network route and GPU allocation. Canonicalise the values before comparison so harmless ordering or server-added defaults do not create constant false alarms.

Use three complementary checks:

1. Structural comparison: compare the desired manifest with the effective configuration reported by each service and node.
2. Integrity comparison: verify controlled artefact hashes, signatures and immutable identifiers.
3. Behavioural verification: run synthetic conversations that check permissions, retrieval, disclosure, escalation, language, latency and prohibited actions.

Structural checks catch a changed prompt file. Behavioural tests may catch a provider-side model change or a new default that the manifest never represented. Neither is sufficient alone.

The OpenGitOps principles describe desired state as declarative, versioned and immutable, automatically pulled and continuously reconciled. That pattern is useful for supported infrastructure, but it is not a requirement to use Git or Kubernetes. A customer-hosted avatar may also include appliances, browser policies, external identity services and model runtimes that need their own collectors.

Classify drift before changing production

Not every difference should trigger automatic rollback. Classify it first.

• Approved variation: a documented overlay for the environment or channel.
• Pending release: a controlled rollout where old and new versions temporarily coexist.
• Representation difference: formatting, ordering or a harmless default that should be normalised.
• Manual change: an administrator altered live state outside the approved route.
• Partial deployment: some nodes, services or kiosks accepted the release and others did not.
• External change: a managed dependency, certificate, endpoint or policy changed independently.
• Suspected compromise: the origin, authority or impact of the change cannot be trusted.

Severity should follow impact, not file size. A one-character change to an approval endpoint can matter more than a large visual asset update. Prioritise drift that changes identity, data access, tool authority, disclosure, egress, retention or the ability to contain the service.

Reconcile safely, not blindly

Automatic reconciliation is useful for low-risk, well-understood infrastructure settings. It can also repeatedly overwrite an emergency fix or restart a fragile real-time service during a busy period.

Define one of four responses for every configuration item:

1. Alert only while an operator assesses the difference.
2. Block promotion until the live state returns to the approved baseline.
3. Automatically reconcile a safe and reversible setting.
4. Contain and investigate when the change affects security or has an unknown origin.

Before reconciliation, capture enough evidence to explain the difference without exporting sensitive customer content. Record the affected item, expected and observed version, source, time, node, actor where known, impact, decision and result. Link this to the privacy-minimised AI audit trail.

Emergency changes should have an expiry and retrospective approval. Otherwise a temporary diagnostic setting becomes a permanent exception. If the change exposed a secret or widened authority, restoring the old file is not enough; revoke or rotate the affected credential.

Promote configuration through evidence, not copying

Build the release once, test it, then promote the same identified artefacts through environments with only approved overlays. Rebuilding separately for production creates room for different dependencies, defaults or optimisation profiles.

The NCSC’s secure AI operation guidance notes that changes to data, models or prompts can change behaviour and recommends secure, modular update procedures, testing and versioned interfaces. Its machine-learning lifecycle principles also call for tracked, reviewed and reversible changes to models and datasets.

Use the deployment and rollback process in Yepic’s guide to updating on-premise avatars without breaking production. Configuration drift management operates between releases: it proves that the approved result stayed in place after the deployment completed.

Deployment model changes where drift hides

Public cloud may provide managed configuration stores, immutable releases and consistent fleet tooling. Provider-side defaults or automatic changes still need to be understood, and the customer may have limited visibility into effective state.

Private cloud can combine dedicated boundaries with familiar orchestration, although customer and provider responsibilities must be explicit.

Customer-hosted deployment gives the organisation direct control over configurations, GPUs and change windows. It also creates more places for local differences: drivers, firmware, certificates, proxies, identity integrations, browser policies and manual operations. Restricted networks may require local repositories, signing, time sources and drift collectors.

NVIDIA’s current NIM deployment guidance, for example, exposes image tags, model settings, credentials, storage, environment variables and GPU resources as separate deployment values. That is an implementation example, not a required Yepic stack. The broader lesson is that “the model version” describes only a fraction of the runtime configuration.

Build a configuration-control register

For each configuration item, record:

• owner, purpose and security impact;
• authoritative desired-state source;
• approved base value and permitted overlays;
• how effective state is collected and canonicalised;
• comparison frequency and acceptable propagation window;
• drift severity and alert destination;
• automatic, manual or containment response;
• approval and emergency-change route;
• rollback target and dependency order; and
• the production test that proves reconciliation worked.

Twelve tests before production

1. A console edit to a production prompt is detected and attributed.
2. A changed retrieval threshold on one node is visible against the approved overlay.
3. A partial rollout identifies every old and new component version.
4. Normalised defaults do not create a permanent false-positive alert.
5. An automatically selected inference profile is recorded as effective state.
6. A changed tool endpoint cannot silently widen data access or authority.
7. An expired certificate or altered network route is detected before service failure.
8. A kiosk that missed an update is identified without waiting for a user complaint.
9. Reconciliation cannot overwrite an authorised emergency change without warning.
10. Rollback restores the complete compatible bundle, not only the language model.
11. Synthetic conversations confirm disclosure, permissions, retrieval and escalation after reconciliation.
12. The audit record explains the drift and response without storing unnecessary conversation content.

Make the approved system observable

In the Abu Dhabi Aviation and Oracle enterprise avatar integration, Yepic worked across separate development and production environments, API and iframe integration, real-time streaming, captions and microphone behaviour, WebRTC and network testing, browser remediation, cybersecurity support and ongoing maintenance. Those operational layers illustrate why configuration cannot be reduced to a model name or prompt file.

The case study is not presented as a completed customer-hosted configuration-drift implementation. Yepic’s private, sovereign and on-premise avatar deployments are custom projects whose configuration controls depend on the selected components, customer environment and operating model. Yepic’s proprietary real-time avatar technology can be optimised for commercial GPUs, while cloud and private-cloud designs remain valid where they better fit the required controls and responsibilities.

Start with one question in the architecture review: can the team name the exact approved configuration of every running avatar, prove what each instance actually loaded, and explain every difference? If not, the production service has an unmeasured state.