Tensor Parallelism for LLMs: When One GPU Is Not Enough

Tensor parallelism lets one language model run across several GPUs by splitting work inside each transformer layer. It is useful when the approved model cannot fit on one GPU, or when a measured multi-GPU configuration meets the required response time better than a single device. It is not the default answer to every capacity problem. Each additional GPU introduces collective communication, a larger failure domain and tighter dependence on the physical interconnect.

For a private real-time AI avatar, the first question is therefore not “How many GPUs can we use?” It is: “Do we need to divide one model, or do we need more independent model replicas?” This guide gives bank, government and enterprise architecture teams a practical way to choose among single-GPU inference, tensor parallelism, pipeline parallelism and data-parallel replicas—and to test the choice as part of the complete conversation.

What tensor parallelism means

In tensor parallelism, the weights and computation of individual layers are divided across a group of GPUs. Each device calculates part of a matrix operation; the partial results are then combined through collective communication before the model can continue. No GPU in that group independently holds and executes the complete model.

This differs from four concepts that are often blurred together:

• Data parallelism: each worker has a complete model replica and serves different requests. It increases aggregate concurrency but normally does not make a model fit on a smaller device.
• Pipeline parallelism: consecutive groups of layers live on different GPUs or nodes. Activations move from one stage to the next.
• Expert parallelism: experts in a mixture-of-experts model are distributed, and a routing mechanism sends tokens to the selected experts.
• Context or sequence parallelism: work associated with the input sequence or attention context is divided across devices.

These patterns can be combined, but their purposes are different. Current vLLM deployment guidance recommends a single GPU when the model fits, tensor parallelism for a model that fits within one multi-GPU node, and a combination of tensor and pipeline parallelism when the model must span nodes. This is a useful starting heuristic, not a substitute for workload testing.

Start with the constraint, not the technique

Architecture reviews should identify which constraint is actually binding:

The model does not fit

Count model weights, runtime workspace, activations, KV cache and operational headroom at the chosen precision. If the complete allocation cannot safely fit on one device, sharding is necessary unless an approved smaller or quantised model can meet the same task and quality requirements.

One conversation is too slow

Tensor parallelism may reduce compute time by sharing layer operations, but every transformer block can require GPUs to exchange and combine partial results. The benefit depends on model shape, batch size, kernels and interconnect. More devices can make a lightly loaded decode path slower when communication dominates.

There are too many simultaneous conversations

If the model already fits and one replica meets per-session latency, add independent replicas before assuming the model itself must be split. Replica scaling creates more scheduling destinations and limits a single GPU-group failure to fewer sessions. It also duplicates weights and KV capacity, so it has a different memory cost.

Long contexts exhaust memory

Check whether the pressure comes from model weights or the LLM KV cache. Tensor parallelism can distribute some state, but context limits, admission policy and cache budgeting still matter. Sharding a model does not make unbounded prompts safe.

Choose among four deployment patterns

1. One GPU per model replica

This is the simplest failure and latency path. Use it when the model, cache and workspace fit with headroom and measured response time meets the service objective. Scale concurrency with more replicas and a permission-aware router. The trade-off is duplicated model memory across workers.

2. Tensor parallelism inside one node

Split each model replica across a tightly connected GPU group. This is a natural candidate when the model is too large for one GPU but fits within one server. High-bandwidth, low-latency GPU-to-GPU links are important because collective operations occur repeatedly during inference.

TensorRT-LLM’s sharding guide frames communication cost as the central decision factor. The current vLLM guide also notes that pipeline parallelism may outperform tensor parallelism on some nodes without NVLink because it requires less frequent communication. Treat the actual topology—not simply the GPU model name—as part of the design.

3. Pipeline parallelism across stages or nodes

Put contiguous layer ranges on different GPUs and pass activations between stages. This can support uneven splits and reduce the frequency of communication compared with tensor parallelism, but it can introduce pipeline bubbles and stage imbalance. For interactive traffic with small batches, an idle stage can be expensive.

4. Hybrid parallelism

Use tensor parallelism within each high-speed node, pipeline parallelism between nodes and replicas across independent GPU groups. Mixture-of-experts models may add expert parallelism. Hybrid designs can make very large models operable, but multiply configuration, observability and failure modes. Approve each dimension for a named reason.

Build a ten-field parallelism contract

Before procurement or production approval, record one contract for every model route:

1. Purpose: model fit, single-session latency, concurrency, context capacity or a combination.
2. Model identity: exact version, architecture, precision, licence and approved context limit.
3. Memory budget: weights, KV cache, workspace, activations and reserve per GPU.
4. Parallel dimensions: tensor, pipeline, data, context and expert sizes.
5. Placement: which ranks share a node, socket, switch and network path.
6. Communication: collective operations, library, link type and measured bandwidth under contention.
7. Traffic: prompt and output distributions, concurrent sessions, batch policy and service classes.
8. Objectives: time to first token, inter-token latency, time to first audio, tail latency and useful throughput.
9. Failure policy: health test, timeout, rank failure response, draining and fallback.
10. Evidence: benchmark version, golden-set results, load profile and approval owner.

Version this contract with drivers, firmware, collective-communication libraries and inference runtime. A change in any of them can alter memory, kernels or synchronisation behaviour even when the model is unchanged.

Map the physical topology

Logical GPU counts hide important differences. Four GPUs connected through a shared high-bandwidth switch are not operationally equivalent to four devices reached through separate PCIe roots, CPU sockets or network adapters.

For each proposed tensor-parallel group, document:

• direct GPU-to-GPU links and switch domains;
• PCIe generation, lane allocation and oversubscription;
• CPU and NUMA affinity;
• GPU-to-network-interface affinity for multi-node traffic;
• whether storage, speech, rendering or other tenants contend for the same fabric; and
• which links remain after a node or device is taken out of service.

Do not accept theoretical link bandwidth as production evidence. Measure collective latency and bandwidth with the intended placement, container security settings and competing workloads. A topology diagram should be tied to device and switch identifiers so that the scheduler cannot silently place a group across a slower path.

Protect the whole avatar GPU estate

The LLM is only one part of a real-time avatar. Speech recognition, text-to-speech, avatar rendering, safety models and video encoding may also need accelerator time. Reserving every local GPU for one tensor-parallel LLM replica can starve the components that turn tokens into a conversation.

Build a component-level placement plan before tuning the LLM:

• which services require GPUs and which can use CPUs;
• whether any components may share a device safely;
• which workloads are latency-critical versus deferrable;
• how memory and compute reservations are enforced; and
• what happens when the LLM group, voice service or renderer is degraded.

Yepic’s GPU scheduling framework for real-time avatars covers this broader allocation problem. Tensor parallelism should fit within that policy, not bypass it.

Test the communication tax

Run the same representative workload on the smallest viable configurations. Compare one GPU, two-way tensor parallelism, higher tensor-parallel sizes, pipeline alternatives and independent replicas where they fit. Keep the model version, precision, prompt set and output controls constant.

Measure at least:

• weight load and warm-up time;
• time to first token by prompt-length band;
• inter-token latency at median and tail;
• tokens and completed turns per second;
• per-rank GPU utilisation, memory and power;
• collective duration, bytes and waiting time;
• KV-cache capacity and pre-emption;
• time to first audio and gaps between playable speech chunks; and
• useful conversations completed per GPU-hour.

A good result is not the configuration with the highest synthetic token throughput. It is the smallest stable group that meets the end-to-end service objective at representative concurrency. Yepic’s guide to AI load testing for avatars explains how to test speech, retrieval, reasoning, voice, rendering and WebRTC as one path.

Run six failure and change tests

1. Rank failure: stop one worker process or GPU; confirm the group fails closed, affected sessions end clearly and no partial answer is presented as complete.
2. Link degradation: reduce or contend interconnect bandwidth; detect the change before conversational latency collapses.
3. Straggler: throttle one rank or create thermal variance; measure how collective synchronisation propagates delay.
4. Placement error: force a group onto an unapproved topology; verify scheduling policy or admission blocks it.
5. Rolling change: upgrade one runtime image, driver or model route through a drained group; prove mixed versions cannot form one replica accidentally.
6. Fallback: lose the multi-GPU route; test the approved smaller model, alternate pool or human hand-off without silently changing authority.

Tensor-parallel ranks normally form one failure unit: losing a required rank can stop the complete replica. Replica-level redundancy therefore remains necessary when availability matters. Do not count the GPUs inside one sharded replica as independent failover capacity.

Deployment location changes ownership, not physics

A customer-hosted deployment can keep model execution, active state and topology telemetry inside the customer’s environment and on customer-controlled GPUs, subject to a properly scoped implementation. It also makes the customer and implementation team responsible for fabric design, placement, driver compatibility, spare capacity, monitoring and incident response.

Private or sovereign cloud may offer residency controls with managed high-speed GPU infrastructure. Public cloud can make experiments and elastic replicas easier, though exact topology and capacity may depend on instance availability. None of these models removes the need to validate collective performance and the complete user path.

At GAIN Saudi Arabia, Yepic operated a five-metre real-time agent targeting sub-one-second responses alongside the holographic Farrah across three summit days. The GAIN real-time avatar deployment shows why infrastructure decisions ultimately have to be judged at the audience experience. It is not presented as a tensor-parallel or customer-hosted implementation.

Twelve questions for architecture approval

1. Which measured constraint requires more than one GPU?
2. Could an approved smaller or quantised model avoid sharding?
3. Would more complete replicas solve concurrency more cleanly?
4. What memory budget and headroom does each rank require?
5. Why was the proposed tensor-parallel size chosen?
6. Does the group stay within a proven high-speed interconnect domain?
7. How much time is spent computing versus waiting on collectives?
8. How are placement and NUMA affinity enforced?
9. Which avatar components compete for the same GPUs or links?
10. What happens to a live session when one rank fails?
11. Which configuration changes trigger revalidation?
12. Did the final test measure first audio and speech continuity?

Use the smallest parallel group that passes

Begin with one GPU when the model fits. Add tensor parallelism only when memory or measured latency requires it. Use pipeline parallelism where layer placement or slower interconnects make it a better fit, and use independent replicas when concurrency is the real problem.

Yepic can scope private, sovereign, private-cloud and customer-hosted real-time avatar deployments, including commercial-GPU optimisation and whole-path validation. The objective is not to involve the most GPUs. It is to choose the simplest topology that preserves model quality, operational evidence and a responsive conversation.