Streaming Speech Recognition: Set an ASR Commit Policy

For a real-time AI avatar, do not send every partial speech-recognition result straight into the language model. Treat interim text as provisional, commit only a versioned final utterance, and measure the delay, revision and task impact of that decision under real acoustic conditions.

The fastest transcript is not necessarily the safest input. Partial hypotheses improve responsiveness, but can change as more audio arrives. A committed transcript can drive retrieval, reasoning or action, so the system must know when the words stopped being provisional.

Separate four speech-recognition states

“The avatar heard me” can refer to four different facts:

1. Speech detected: the capture path or voice-activity detector believes speech is present.
2. Interim hypothesis: the recogniser has produced text that may still be revised.
3. Final segment: the recogniser will not revise a defined span of audio again.
4. Committed turn: the application has assembled the authorised segments, applied policy and released one immutable input to downstream systems.

These states are related but not interchangeable. NVIDIA's current streaming ASR interface, for example, distinguishes interim results with is_final=false from final results with is_final=true. Amazon Transcribe similarly returns evolving partial results until a speech segment is finished. Both patterns make the architectural point: displayed text and committed input need not be the same object.

This article complements Yepic's safe avatar turn-taking policy. Turn-taking decides whether the person has finished and whether the avatar should yield. The ASR commit policy decides which words may become durable conversational input.

Map the complete audio-to-input path

A useful design review follows the signal through seven stages:

1. Microphone capture, permissions and device routing;
2. echo cancellation, gain control and noise processing;
3. resampling, channel conversion, framing and transport;
4. speech detection and endpoint evidence;
5. streaming acoustic and language decoding;
6. interim, stable and final transcript assembly; and
7. application commit into the LLM, retrieval or workflow boundary.

An error can enter at every stage. A clipped microphone can remove syllables; loudspeaker echo can enter capture; a network gap can delay chunks; endpointing can split a reference number; and a late correction can arrive after an irreversible action starts.

The acoustic echo-cancellation guide explains why hosting recognition on a private GPU does not repair a poor endpoint signal. Speech recognition begins with the audio that actually arrives, not the audio the architecture diagram assumes.

Use interim text for preparation, not authority

Interim transcripts are valuable. They can animate a listening state, display responsive captions, begin speculative retrieval or pre-warm a language-model context. They should not, by default, authorise a payment, submit a form, update a case or become the permanent audit record.

Define three downstream lanes:

• Presentation lane: may show provisional text, clearly marked and replaceable.
• Preparation lane: may start cancellable work such as retrieval, classification or prefetching.
• Commit lane: accepts only the immutable utterance version approved by the turn policy.

Every speculative job needs a transcript-version identifier and cancellation handle. If “transfer fifteen” becomes “transfer fifty”, results derived from the older hypothesis must be discarded. The final utterance should carry the audio interval, language, recogniser and configuration versions, segment identifiers and commit reason.

Do not confuse finalisation with end of turn

A recogniser may finalise several segments inside one long utterance. Conversely, a silence detector may decide that a person paused even though the intended thought is unfinished.

Deepgram's current documentation illustrates the distinction between a finalised transcript segment and a speech_final endpoint event. NVIDIA Speech NIM also exposes configurable end-of-utterance behaviour and, for eligible models, a client signal that can force finalisation without closing the stream. These are implementation examples, not universal defaults.

For an avatar, the application should assemble final segments until its separate turn policy commits the input. A push-to-talk release, explicit submit button or controlled kiosk workflow can provide stronger evidence than silence alone. High-risk journeys may also require the avatar to read back a name, amount, date or identifier before any protected action.

Measure more than word error rate

Word error rate (WER) counts substitutions, insertions and deletions against a human-produced reference transcript. It remains useful, but a single average can hide the errors that matter most. Google's current accuracy guidance also notes that confidence and WER are independent: a confident result can still be wrong.

Build a scorecard that includes:

• First-interim latency: time from audible speech to the first useful hypothesis;
• final-segment latency: time from the end of the relevant audio to an immutable segment;
• turn-commit latency: time until the application releases the complete input;
• revision rate: how often displayed or speculative words change;
• false-final and split rate: premature finalisation or one intended turn divided incorrectly;
• critical-entity accuracy: names, amounts, dates, addresses, codes and domain terms;
• semantic task accuracy: whether the recognised input leads to the correct intent and outcome;
• real-time factor and concurrency: processing capacity under the expected session mix; and
• fairness slices: results by language, dialect, accent, speaking style, device and acoustic environment.

Report distributions, not only averages. A low mean finalisation time can coexist with damaging tail delays under concurrent load.

Control domain adaptation as production data

Phrase hints, custom vocabularies and model adaptation can improve product names, acronyms and specialist terminology. Google documents phrase sets and boosting; Amazon documents custom vocabularies for domain-specific words. These controls can also bias the model towards a plausible but incorrect term.

Treat every adaptation asset as a versioned configuration with an owner, approved data source, language, scope, expiry and evaluation set. Do not put confidential information into a provider vocabulary unless the data flow, contract and policy explicitly allow it. Amazon's documentation, for example, warns customers not to place confidential, personal or protected health information in a custom vocabulary.

Test both gains and regressions. Adding a bank's product name may improve one phrase while increasing substitutions for a similar surname. A pronunciation or display rule can also change what is shown to the user without changing the underlying acoustic decision.

Evaluate every language as its own service profile

“Multilingual” is not one capability. For each language and relevant dialect, confirm:

• streaming rather than batch support;
• interim and final-result behaviour;
• punctuation, numerals and inverse text normalisation;
• domain adaptation and pronunciation controls;
• code-switching and automatic language-detection behaviour;
• endpoint sensitivity to natural pause patterns; and
• the evaluation data, reviewers and acceptance thresholds.

Do not infer Arabic performance from English, or Modern Standard Arabic from a dialect. Numbers, names and mixed Arabic-English terminology require task-specific tests. A service can have strong general transcription while mishandling the exact entities a citizen or customer needs to provide.

Choose cloud, private cloud or customer-hosted ASR deliberately

Deployment model	Useful when	Control to examine
Managed public cloud	Fast evaluation, elastic capacity and managed model updates matter	Audio processing location, retention, network dependency, change cadence and service limits
Private or sovereign cloud	Defined regional, tenant or contractual boundaries are required	Which components and telemetry remain managed, and where every derived artefact travels
Customer-hosted	Live audio and transcript processing must remain inside the customer's environment	GPU capacity, model licensing, artefact supply, patching, observability, language coverage and support boundary
Hybrid	Immediate detection must remain local while selected recognition or fallback is remote	Failover semantics, data crossing the boundary and behaviour during disconnection

NVIDIA Speech NIM is one current example of containerised, GPU-backed ASR that can be deployed with Docker or Helm. Its documentation makes the operational responsibility visible: model-specific hardware requirements, licensing, model caching, GPU allocation, TLS or mTLS and profile selection still have to be managed. It is an implementation option, not a Yepic dependency or a universal recommendation.

Managed services can also offer explicit data-handling choices. Google states that streaming and synchronous speech requests are processed in memory rather than stored by the Speech-to-Text service, while request metadata may be temporarily logged; it also documents multi-region processing controls. Buyers should verify the currently contracted service, endpoint, logging choices and jurisdiction rather than generalise from a product family.

Minimise evidence without losing diagnosability

A regulated service needs enough evidence to explain why an input was committed, but that does not require retaining every raw recording.

A privacy-minimised ASR event record can contain:

• session, turn and transcript-version identifiers;
• recogniser, model, language and adaptation versions;
• audio-format and endpoint-class metadata;
• timestamps for speech start, interim, final segment and commit;
• revision, stability and selected confidence bands where available;
• endpoint and commit reasons;
• critical-entity validation result; and
• downstream cancellation, acceptance or correction outcome.

Raw audio, full transcripts and alternative hypotheses need separate purposes, access controls and retention periods. Derived text can remain personal or sensitive data.

Write a ten-field ASR contract

1. approved languages, dialects, devices and acoustic environments;
2. capture format and preprocessing owner;
3. streaming model, version and deployment location;
4. interim-result use and prohibited downstream uses;
5. segment-final and turn-commit semantics;
6. endpoint, timeout and explicit-submit policy;
7. domain adaptation assets and change control;
8. critical-entity confirmation and correction route;
9. latency, accuracy, capacity and fairness acceptance measures; and
10. telemetry, retention, rollback and retest triggers.

This contract should join the LLM-to-speech commit policy. Together they govern both irreversible directions of a spoken interaction: what the system accepts from the user and what it allows the avatar to say.

Run nine production acceptance tests

1. Revision test: feed utterances that change meaning near the end and prove speculative work is cancelled.
2. Mid-thought pause: test hesitation inside names, numbers, addresses and considered answers.
3. Critical entity: measure task accuracy for amounts, dates, codes and proper nouns, not only overall WER.
4. Acoustic matrix: repeat with approved microphones, speakers, rooms, noise, distance and double-talk.
5. Language matrix: test dialects, code-switching, numerals and language-specific normalisation.
6. Concurrent load: measure interim and final latency, queueing and resource use at expected and overload levels.
7. Stream fault: introduce late, duplicated, missing and reordered audio or transcript events.
8. Recognizer failure: prove timeout, fallback, correction, text-input and human-handoff behaviour.
9. Change regression: rerun the golden set after a model, vocabulary, driver, endpointing or audio-processing update.

Use the same test runs to inspect the avatar's visible listening state, response timing and user ability to correct errors. An ASR service can look accurate in a batch benchmark while producing an unusable conversation.

Apply Yepic delivery evidence precisely

The Abu Dhabi Aviation and Oracle enterprise-avatar integration covered development and production environments, API and iframe integration, real-time streaming, captions and microphone behaviour, WebRTC and network testing, browser remediation, cybersecurity support and ongoing maintenance. It is relevant evidence that live speech, endpoint and browser behaviour must be tested as one service. It is not presented as a completed customer-hosted ASR deployment.

Yepic can scope proprietary real-time avatar technology across public cloud, private cloud, sovereign and customer-controlled GPU environments. Speech recognition can be selected and integrated around the approved languages, data boundary, capacity and user journey. The exact ASR engine, hosting arrangement and support model remain implementation decisions.

Twelve questions for architecture approval

1. Which transcript states can the interface display?
2. Which state may reach retrieval, the LLM and business tools?
3. How are stale speculative results cancelled after a revision?
4. What is the difference between a final segment and a committed turn?
5. How are names, numbers, amounts and identifiers confirmed?
6. Which languages, dialects and code-switching patterns have representative tests?
7. Who owns microphones, audio processing, endpointing and recognition?
8. Where do raw audio, partial text, final text and telemetry travel?
9. Which adaptation data is loaded, and who approves it?
10. What happens when recognition is late, unavailable or uncertain?
11. Which metrics expose revision, finalisation, fairness and capacity?
12. Which model, vocabulary, driver or client change triggers regression testing?

Make the committed input auditable

The right streaming ASR design is not simply the model with the lowest headline WER. It is the path that turns uncertain audio into one controlled, observable input without hiding revisions, rushing hesitant speakers or leaking more data than the service needs.

For a regulated pilot, begin with one journey. Mark the provisional, final and committed boundaries; define critical entities; assemble a representative acoustic and language set; then test under concurrency and failure. That evidence lets Yepic and the customer scope the recognition, avatar and deployment layers around the environment the service must actually support.