AI Agent Secrets Management: Keep Secrets Outside the Model

The safest credential for an AI avatar is one the model never receives. A production avatar should prove its workload identity to a trusted broker or gateway, obtain only the short-lived authority needed for the next approved call, and keep reusable API keys, refresh tokens and service-account credentials outside prompts, model context, logs and general application memory.

This matters because a real-time avatar is not one model. It is a chain of speech recognition, language reasoning, retrieval, text-to-speech, rendering, streaming and enterprise APIs. Each connection may need a credential. If those credentials are copied into environment files and shared across replicas, a prompt-injection flaw, support bundle or misconfigured log can turn one compromised conversation into wider system access.

Customer-hosted deployment changes where the problem is operated; it does not remove it. Banks, governments and regulated enterprises still need an explicit design for identity bootstrapping, secret custody, token scope, rotation, revocation and reduced service.

First separate identity, secrets, tokens and encryption keys

Architecture reviews become confused when every cryptographic object is called a key. Use four distinct terms:

• Workload identity says which running component is making a request: for example, the production retrieval service rather than an arbitrary container.
• Reusable secret is long-lived material such as an API key, client secret or refresh token. It should remain in a dedicated custody boundary whenever the destination protocol allows that.
• Short-lived token or certificate is temporary authority issued to a verified workload, ideally limited by audience, action, environment and time.
• Encryption key protects data or other secrets. It does not, by itself, decide which workload may call an API. That is why customer-managed encryption keys and secrets management require different control registers.

A human user may also authenticate through enterprise SSO. That identity should be carried as delegated context where the use case requires it, but it should not be confused with the identity of the avatar workload executing the call. The distinction is central to a secure avatar authentication architecture: who requested an action, which service executed it and what authority was granted are three separate facts.

Inventory the credentials across the whole avatar pipeline

Start with a credential register, not a vault purchase. For every dependency, record the owner, issuer, consuming workload, environment, permitted operations, storage location, rotation method, expiry, revocation route and degraded-service behaviour. At minimum, inspect six classes.

1. Speech, language, voice and rendering services

A modular deployment may call customer-hosted components, private-cloud endpoints or external providers. One shared credential across development, test and production prevents clean attribution and makes rotation unnecessarily disruptive. Give each environment and workload its own policy boundary.

2. Knowledge and retrieval systems

Vector stores, document repositories and search services can expose sensitive source material. The retrieval service should use its own workload identity and enforce the caller's permissions before returning passages. Do not place database passwords, storage signatures or source-system tokens in the prompt. The same principle underpins permission-aware RAG for private avatars.

3. Tools and enterprise APIs

CRM, case-management, booking and payment APIs carry the greatest risk because credentials can enable action. Put them behind the policy and approval controls described in a safe AI action gateway. A secrets manager can protect a token at rest; it cannot decide whether a generated tool call is appropriate.

4. Real-time media and session services

Signalling, TURN and session admission commonly use temporary credentials. Their lifetime should match the session design, and a browser should receive only what it needs to establish that session. Customer-hosted inference does not automatically make the WebRTC media path private or eliminate external trust relationships.

5. Asset, storage and delivery systems

Avatar assets, voice artefacts, generated media and recordings may sit in object storage or content-delivery services. Prefer narrowly scoped, expiring access rather than a general storage credential. Read and write authority should be separated wherever the workflow permits it.

6. Deployment, registry and update systems

Container registries, package repositories, model stores and configuration pipelines need credentials even when runtime inference is fully on-premise. These administrative credentials deserve a stronger boundary than ordinary runtime access because compromise can alter what code or model is deployed. Keep them in the AI bill of materials and dependency register, with named owners and tested revocation paths.

A seven-boundary reference architecture

The objective is not simply to move secrets from a file into a vault. It is to reduce how often plaintext credentials enter the avatar process at all.

1. Attest the running workload

Give each service identity based on trustworthy runtime evidence: its orchestrator service account, signed workload, node, process or equivalent platform signal. Network location is insufficient. NIST's Zero Trust Architecture explicitly rejects implicit trust based only on physical or network location.

SPIFFE is one possible standard, not a requirement. Its workload identity model can issue short-lived X.509 or JWT identity documents without provisioning an authentication secret into the application. Cloud-native workload identity, a service mesh or a customer's existing machine-identity platform may provide the same architectural role.

2. Keep reusable material in a dedicated custody service

The secret store should enforce separation of duties, access policy, versioning, backup and recovery appropriate to the organisation. The UK NCSC's API guidance recommends secure credential storage, warns against secrets sprawl and says rotation should be automated where possible.

Do not assume that mounting a secret into every container solves the problem. Once a reusable value appears in an environment variable, file, crash dump or process memory, the blast radius includes that workload.

3. Exchange identity for limited authority

A verified workload asks a broker for access to a named destination and action. Policy checks the workload, environment, requested scope, user delegation, session risk and approval state. The result should be an audience-restricted token, temporary certificate or brokered call—not a copy of the master secret.

For OAuth-based services, RFC 9700, published in January 2025, recommends restricting an access token to the minimum privileges required and to a specific resource server where possible. It does not prescribe one lifetime for every system.

4. Prevent credential flow into model context

The language model should describe an intended tool and structured arguments. A deterministic executor should add authentication after policy approval. Prompt templates, retrieval documents, conversation transcripts and model-visible error messages must never contain reusable credentials.

Apply the same rule to observability. Redact authorization headers, signed URLs, connection strings and secret-bearing query parameters before logs or traces leave the component. Record the secret version or token identifier, not the value.

5. Bind access to purpose

Scope should cover more than read versus write. Define the destination, tenant, resource, operation, environment, maximum amount or data class where relevant. A token for reading a customer's appointment should not authorise creating one; a production voice service should not inherit access to a test knowledge base.

6. Rotate and revoke without stopping every conversation

Design rotation before launch. Issue the new version, validate it on a canary workload, allow a brief controlled overlap if the protocol requires it, move new sessions, drain old sessions, revoke the previous version and verify that old access fails. Long conversations need an explicit refresh or re-authentication path rather than a permanently valid token.

7. Produce evidence without recording secrets

For each credential decision, record workload identity, delegated user where applicable, policy version, requested destination and operation, token or secret version, decision, expiry and correlation ID. This supports an AI audit trail without full conversation logs while keeping secret material out of the evidence store.

Deployment model changes the control owner

Public cloud can offer mature managed identity and secrets services with automated rotation, but it introduces provider dependencies and may place some control planes outside the customer's environment. Private cloud can preserve similar automation within a dedicated tenancy, although contractual control is not the same as customer operation.

On-premise deployment can keep credential custody, policy and audit services inside the customer's boundary. It also makes the customer responsible for highly available secret storage, root-of-trust protection, certificate renewal, time synchronisation, backup, recovery and emergency access. In a restricted network, teams must also plan how trust bundles, revocation data and approved updates cross the boundary. Fully disconnected operation is a scoped engineering decision, not a property that follows automatically from installing inference on local GPUs.

A hybrid design is often rational: keep sensitive retrieval and action credentials customer-side while using separately governed credentials for an external speech or rendering service. Document every crossing rather than labelling the entire system simply “on-premise”.

Test rotation and compromise, not just the happy path

Before production approval, run these twelve tests:

1. Start an unregistered workload and confirm that it receives no identity or secret.
2. Copy a workload identity artefact to another host or environment and confirm that replay fails.
3. Request a token for the wrong audience, tenant or operation and confirm denial.
4. Inject instructions asking the model to reveal environment variables, headers or credentials and confirm none are model-visible.
5. Inspect prompts, traces, logs, crash dumps and support bundles for secret material.
6. Rotate a live dependency credential while existing avatar sessions continue or degrade according to plan.
7. Revoke a token or workload identity and measure how quickly new calls stop.
8. Disable the broker and verify that the avatar enters a defined read-only, scripted or human-handoff state.
9. Compromise one replica and confirm that its authority cannot be reused by another service.
10. Separate development and production credentials, then prove that cross-environment use fails.
11. Restore the secret service and its policy data from backup without resurrecting revoked access.
12. Trace one approved action from user request to workload identity, policy decision, downstream response and audit receipt without exposing the credential.

A failed rotation test is an availability finding as well as a security finding. If operators postpone rotation because it breaks sessions, a nominal 90-day policy will quietly become a multi-year credential.

What Yepic experience contributes

Yepic has built proprietary talking-photo and real-time avatar technology and supports cloud, private-cloud and customer-hosted architectures through properly scoped custom implementations. The relevant lesson is operational: credentials sit inside a larger integration, network and support model.

In the Abu Dhabi Aviation and Oracle enterprise avatar integration, the work covered separate development and production environments, API and iframe integration, real-time streaming, captions and microphone behaviour, WebRTC and network testing, browser remediation, cybersecurity support and ongoing maintenance. That project is evidence of production integration discipline; it is not presented here as a completed customer-hosted secrets-management deployment.

The architecture-review decision

Ask one question for every credential: must the avatar process possess this value, or can a verified workload request a narrowly authorised call without seeing it? Where brokering is feasible, choose the second design. Where a third-party protocol still requires a static key in the process, isolate the workload, minimise scope, automate rotation, prevent model and log exposure, and document the residual risk.

The practical next step is to build the credential register, then conduct a live rotation and revocation drill across one complete avatar journey. That test will reveal more than a screenshot of a vault ever can.