GPU Sizing for Real-Time AI Avatars: Capacity Planning

The right way to size GPUs for a real-time AI avatar is to start with a tested conversation workload, not a hardware shopping list. Measure how many simultaneous sessions one GPU can sustain while meeting agreed thresholds for response latency, frame rate, audio-video synchronisation and failure rate. Then size the production pool for peak concurrency, operating headroom and node failure.

There is no credible universal answer such as “one GPU supports 20 avatars”. The result changes with the avatar renderer, speech and language models, resolution, frame rate, voice, language, turn length, video encoder, client device and network. It also changes according to which stages share the same GPU.

This guide gives architecture and procurement teams a defensible GPU capacity-planning method for customer-hosted and private-cloud AI avatars. It is deliberately model- and hardware-neutral: the goal is to turn a vendor demonstration into a workload that can be tested, costed and governed.

Start with the whole conversation, not the avatar renderer

A real-time avatar is a pipeline. A spoken exchange may involve microphone capture, voice activity or turn detection, automatic speech recognition, retrieval, an LLM or other decision system, tool calls, text-to-speech, facial animation, video rendering, encoding, WebRTC transport and playback in a browser or application.

The visible video is only one part of the latency budget. A fast renderer cannot compensate for an overloaded speech service, a slow retrieval call or a corporate network that interferes with real-time media. Conversely, buying a larger GPU will not fix a microphone permission problem or an underpowered kiosk browser.

Yepic encountered this end-to-end reality while integrating a real-time avatar with Oracle-connected systems for Abu Dhabi Aviation. The production work included separate development and production environments, streaming and iframe integration, captions and microphone behaviour, plus latency, WebRTC, browser, mobile and corporate-network testing. Read how the enterprise avatar integration was tested.

Define the workload envelope before testing hardware

A sizing exercise needs a workload definition that both the buyer and supplier can reproduce. At minimum, record the following:

• Peak concurrent sessions: not just daily users, but the maximum number of conversations expected at the same moment.
• Arrival pattern: a steady queue, office-hour peaks, an event rush or many kiosks opening together.
• Conversation profile: average and long-turn duration, interruption frequency, follow-up questions and silence time.
• Visual quality: target resolution, frame rate, codec and whether the avatar is full screen, embedded or shown on a large display.
• Avatar variety: the number of identities, voices and models that must remain resident or be swapped during service.
• Language mix: languages and dialects used simultaneously, including whether different speech or voice models are required.
• Interaction features: captions, camera input, emotion signals, tool calls, retrieval and authentication.
• Availability objective: what must happen during maintenance, a GPU failure or loss of an upstream component.

Use a peak-hour trace where one exists. If the service is new, create several explicit scenarios: normal weekday, expected peak, launch-day burst and degraded operation. The difference matters. Ten continuously speaking sessions can produce a different load from 50 sessions in which most users are listening or thinking.

Set acceptance thresholds that reflect user experience

Average GPU utilisation is not a service-level measure. A deployment can show a comfortable average while a small proportion of users experience long pauses, frozen faces or dropped audio. Capacity tests should therefore combine infrastructure telemetry with user-facing measures.

Conversation measures

• time from the end of a user turn to first audible response;
• time to first rendered frame and time to stable playback;
• median and tail latency, normally including p95 or p99 rather than the mean alone;
• interruption and turn-detection success;
• audio-video synchronisation and completed-session rate.

Rendering and infrastructure measures

• sustained delivered frame rate and dropped frames;
• GPU compute, memory and encoder utilisation;
• queue time at each accelerated service;
• CPU, RAM, network throughput and packet loss;
• model-load time, restart time and error rate.

Thresholds should be agreed for the actual service. A wealth-management conversation may prioritise natural turn-taking and privacy. A transport kiosk may prioritise rapid recovery and legibility. A large-format event avatar may demand a higher visual target. Yepic’s five-metre agent at GAIN was designed for a target response time below one second and operated as a live public experience; that is evidence of low-latency event engineering, not a universal benchmark for every customer workload. See the GAIN real-time avatar deployment.

A capacity formula architecture teams can defend

Once the workload and thresholds are fixed, establish validated session density: the maximum simultaneous sessions a single GPU can sustain in a production-like test while every acceptance threshold still passes.

Working GPUs = ceiling of peak concurrent sessions ÷ (validated sessions per GPU × target load factor)

Total GPUs = working GPUs + failure and maintenance reserve

The target load factor is an engineering policy, not a magic percentage. It should leave enough capacity for traffic variation, long conversations, model changes and imperfect load distribution. The reserve should reflect the failure domain: losing one GPU, one server, one availability zone or an entire site are different design problems.

For example, do not take a vendor’s maximum session count, divide demand by it and call the result production capacity. Ask which model versions, resolution, frame rate, languages, turn pattern, codec and latency thresholds produced that number. Then rerun the test on the proposed hardware and software release.

Use a load-test staircase, not a single stress test

A useful capacity test progresses in controlled steps:

1. Warm baseline: run one representative session and confirm that every stage is observable.
2. Concurrency steps: add sessions gradually and hold each level long enough to expose queueing and thermal behaviour.
3. Steady-state soak: operate near the intended ceiling for hours, not minutes.
4. Burst test: start many sessions or turns together to reproduce shift changes, scheduled appointments or an event opening.
5. Variation test: change avatar, voice, language, response length and tool use.
6. Cold-start test: load models and establish sessions after restart rather than relying only on a warm cache.
7. Failure test: remove a GPU or server and confirm that traffic redistributes without breaching the agreed user experience.
8. Upgrade test: measure capacity while a node is drained, patched and returned to service.

If an implementation uses NVIDIA Triton, its official Performance Analyzer can generate concurrency- or request-rate-based inference load. Other stacks can use different tools; what matters is a repeatable workload and end-to-end measurement, not the brand of the test harness.

Decide which components share GPU capacity

An on-premise avatar may place rendering, speech recognition, language inference and speech generation on one GPU pool, or divide them across dedicated pools. The choice affects both utilisation and predictability.

A shared pool can use hardware efficiently when demand is low or complementary, but one component can consume memory or compute needed by another. A long LLM response, a multilingual speech model and a rendering burst may collide at precisely the wrong moment.

Separate pools make bottlenecks and scaling policies easier to reason about. They can also strand capacity when one stage is quiet and another is saturated. For regulated environments, separation may nevertheless be justified by ownership, patching cadence, data boundary or fault-isolation requirements.

Supported NVIDIA GPUs can also use Multi-Instance GPU partitioning to divide a physical GPU into isolated instances with dedicated resources. This can improve workload isolation, but it is not available on every GPU and the right partition layout must be benchmarked. It should not be written into a procurement specification without checking the proposed hardware, model footprint and failure behaviour.

Monitor queues and stages, not just the GPU

A production dashboard needs to explain why a response slowed down. Correlate user-facing latency with queue depth and execution time for speech recognition, retrieval, language inference, speech generation, rendering and transport.

NVIDIA’s DCGM Exporter can expose GPU telemetry to Prometheus-compatible monitoring systems and supports metrics for Multi-Instance GPU environments. It is one useful source, not the whole observability design. Application traces and WebRTC or client telemetry are still needed to locate delays outside the GPU.

For private deployments, monitoring should avoid exporting conversation content simply to diagnose performance. Prefer request identifiers, timings, queue lengths, component versions and aggregate error categories. The customer should decide where logs reside, how long they are retained and who may access them.

Plan graceful degradation before the peak arrives

Capacity planning is incomplete without an overload policy. A governed service should define what happens when demand exceeds the tested envelope or a node fails. Depending on the use case, options may include:

• reducing resolution or frame rate while preserving clear audio;
• limiting new sessions and showing an honest wait state;
• moving to voice-only or text interaction;
• routing eligible traffic to an approved private-cloud or cloud pool;
• switching to prepared information for a narrow service;
• handing the conversation to a human or closing safely.

The correct fallback depends on risk. A queue may be acceptable for an internal training coach and unacceptable for emergency information. Cloud bursting may be technically convenient but prohibited by data-residency or network policy. This is why the deployment choice should follow the workload and controls; our comparison of on-premise, private-cloud and public-cloud avatar deployment sets out those trade-offs.

Ten questions to put into an RFP or architecture review

1. What exact workload assumptions produced the proposed sessions-per-GPU figure?
2. Which components run on the GPU, and which share the same physical device?
3. What p95 and p99 conversation, rendering and queue thresholds must the design meet?
4. How does capacity change with resolution, frame rate, avatar, language and voice?
5. What happens when models are loaded cold or changed during service?
6. How much capacity remains after one GPU or server is removed?
7. Can a node be patched without breaching peak-load requirements?
8. Which telemetry leaves the customer environment, if any?
9. What user experience is presented during overload or dependency failure?
10. Who reruns the benchmark when a model, driver, runtime or avatar release changes?

Banking teams can place these questions alongside the security and integration controls in Yepic’s private-GPU reference architecture for banks. Teams deploying kiosks or large displays should also test the physical and browser conditions covered in our practical guide to real-time avatars in public spaces.

What Yepic can credibly bring to the sizing process

Yepic has proprietary talking-photo and real-time avatar technology, experience optimising avatar inference for commercial GPUs and operating low-latency multilingual experiences. The company has also delivered production integration work involving APIs, streaming, browsers, corporate networks and live operational support.

Those projects provide useful engineering evidence, but they do not create a universal GPU specification. Yepic’s customer-hosted, private and sovereign deployments are scoped custom implementations. Final capacity depends on the selected models, hardware, network, service objective and workload observed in a production-like environment.

The most useful first deliverable is therefore not a server quote. It is a capacity-planning pack: the workload envelope, latency budget, proposed component placement, test scripts, acceptance thresholds, failure reserve and costed deployment options. With that pack, a bank, government department or regulated enterprise can compare architectures on evidence rather than demonstrations.