A voice agent and a realtime transcription service can listen to the same audio while owning different outcomes. The agent must decide and respond; transcription must produce an ordered, attributable, revisable text record. Treating the transcript as a side effect of the conversation session hides timing, ordering, quality, and cost failures. A separate protocol contract makes each failure measurable and each consumer explicit.
Reading time: 8 minutes · About 1,650 words
TL;DR
- Live audio and streaming output are separate design decisions. OpenAI recommends
gpt-live-transcribefor live text and a transcription session when no spoken assistant response is required. - Transcript deltas are provisional; completion events are authoritative for a committed item.
- Completion events across turns may arrive out of order. Reconcile them with
item_id, not arrival order. - Keep transcription latency, accuracy, revisions, cost, and fallbacks separate from the voice agent's response metrics.
- Use one explicit contract for capture, turn boundaries, transcript events, finalization, provenance, and downstream consumers.
Shared audio does not imply shared responsibility
Voice systems are often drawn as one line:
microphone → voice agent → transcript → analytics
The line conceals several independent paths. Audio capture and transport feed turn detection. Transcription emits provisional and final text. The conversational model may start responding before a final transcript exists. Analytics, compliance, search, and human review usually need stable text after the interaction.
These consumers have different correctness requirements. A caption UI values early deltas. A legal archive values a stable final record. A routing model may need only keywords. Conversation logic may operate directly on audio and should not pretend its asynchronous transcript is the exact input that produced the response.
The right system boundary follows the dependency graph. Transcription is a service with its own contract, not a text field attached to a voice-agent object.
What OpenAI's current API makes explicit
OpenAI's transcription overview separates completed recordings from live audio. File transcription uses a bounded upload. Realtime transcription uses a persistent WebSocket or WebRTC connection when microphone, call, or other audio arrives continuously.
For live transcription, the recommended starting model is gpt-live-transcribe. Its model page lists audio and text input, text output, the realtime transcription endpoint, streaming, tunable latency, free-form context, keyword hints, and multiple language hints.
A transcription session uses type: "transcription". Audio arrives through input_audio_buffer.append. With automatic turn detection disabled, the client commits a turn through input_audio_buffer.commit; server-side VAD can commit boundaries instead.
Two events define the text lifecycle:
conversation.item.input_audio_transcription.deltacarries newly available provisional text.conversation.item.input_audio_transcription.completedcarries the final transcript for a committed item.
OpenAI explicitly warns that completion events from different turns are not guaranteed to arrive in order. The application must match events with item_id.
The API also states current capability limits. gpt-live-transcribe does not return word-level timestamps, speaker labels, or confidence scores. Applications that require them need a compatible file workflow or their own fallback. These absences belong in the product contract, not in a footnote.
Define the transcript as a state machine
A string field such as latest_transcript cannot represent the lifecycle safely. Use states and immutable identities:
session_open
→ audio_appending
→ turn_committed
→ partial(delta_1 ... delta_n)
→ completed(final_text)
→ downstream_acknowledged
Each transition should be observable. A minimal event envelope can look like this:
{
"schema_version": "transcript-event.v1",
"session_id": "ts_019a",
"item_id": "item_003",
"sequence": 17,
"event_type": "transcript.completed",
"source_model": "gpt-live-transcribe",
"audio": {
"format": "audio/pcm",
"sample_rate_hz": 24000
},
"text": "Please change order AC-42 to Friday.",
"is_final": true,
"received_at": "2026-08-02T02:31:08.441Z"
}
item_id binds provisional and final text to one audio item. sequence orders events inside your transport or event log; it must not replace the provider's item identity. schema_version protects downstream consumers from silent field changes. The source model and audio format make quality regressions diagnosable.
Keep provisional text and final text separate
Low-latency transcription trades immediacy for context. OpenAI exposes delay levels from minimal through xhigh: lower delay can emit text earlier, while higher delay gives the model more audio context and may improve word error rate. Exact milliseconds vary, so representative audio must be benchmarked.
The UI should render deltas as provisional. Downstream systems should declare whether they accept provisional text. A search preview may; a compliance archive should wait for completion. Never overwrite a final record silently with the last partial string.
For each item, store at least:
- current provisional text;
- final text and completion time;
- revision count or delta history policy;
- model and configuration version;
- audio reference or retention status;
- consumer acknowledgements and processing errors.
This also gives the product a correction policy. A UI can visually revise earlier words. Analytics can process only final items. An alerting system can decide whether latency justifies acting on a partial transcript.
Separate turn boundaries from transcript completion
Turn detection decides where audio units end. Transcription decides when enough evidence exists to emit text. Conversation generation decides when to respond. These clocks can differ.
With manual commits, the application controls boundaries. With voice activity detection, the server can detect and commit them. Either way, log the boundary source and configuration. Otherwise, a latency regression caused by silence thresholds will be misdiagnosed as model slowness.
Do not join turns by arrival order. The contract should map every delta and completion to item_id, then map that item to a local turn record. If the voice agent produces a response before the final transcript arrives, record that fact. The final transcript becomes a later observation, not a fictional explanation of an earlier response.
Give transcription its own service-level objectives
Voice-agent dashboards often report only end-to-end response latency. That number cannot locate a transcription failure. Measure the path in segments:
| Metric | What it reveals |
|---|---|
| Audio ingest gaps and dropped chunks | Capture or transport failure |
| Commit-to-first-delta latency | Live caption responsiveness |
| Commit-to-completed latency | Availability of stable text |
| Revision distance | How much partial text changes |
| Domain-term error rate | Product names, IDs, medication, acronyms |
| Empty, truncated, and delayed transcript rate | Failures hidden by average WER |
| Out-of-order completion rate | Reconciliation pressure |
| Cost per audio minute and final item | Capacity economics |
| Downstream acknowledgement lag | Analytics or storage bottleneck |
Test real microphones, telephony codecs, accents, background noise, code-switching, long sessions, interruptions, numbers, dates, currency, email addresses, and domain vocabulary. Synthetic clean audio provides a weak production baseline.
OpenAI supports prompt, keywords, and languages as transcription context. Use them to describe the recording, provide literal domain terms, and list expected languages. Keywords are hints rather than required output. Version these inputs with the eval result because they change system behavior.
Choose an architecture by consumer needs
Dedicated transcription session
Use it for live captions, meeting notes, call analytics, or any workflow that needs text without a spoken assistant response. This creates the cleanest ownership boundary.
Voice agent plus independent transcript path
Use one audio capture path to feed the voice interaction and a separate transcription contract to feed observability, analytics, and archives. Account for synchronization and duplicate audio transport. The benefit is that conversation behavior and transcript quality can evolve independently.
Voice agent with asynchronous input transcription
This can be sufficient when the transcript is a convenience for display or debugging. Document that it may arrive after response generation starts and should not be treated as a complete causal trace of the model's decision.
The smallest architecture that satisfies the consumers is usually best. The key requirement is explicit semantics, not the number of services.
Production acceptance checklist
Before release, verify:
- Every audio item has a stable identity and exactly one accepted final record.
- Deltas and completions reconcile correctly under out-of-order delivery and reconnects.
- Duplicate commits and replayed events are idempotent.
- Partial text remains visibly provisional and cannot silently enter final archives.
- Unsupported timestamps, speaker labels, and confidence fields have explicit fallbacks.
- Each target language and real audio condition has a quality threshold.
- Retention, redaction, and access controls apply to both audio and text.
- Transcription metrics can be separated from response generation and downstream analytics.
The transcript then becomes a product interface that other systems can trust, rather than an incidental string produced somewhere inside the voice stack.
FAQ
Is streaming transcription the same as realtime audio?
No. A completed file can stream text while being processed. Realtime is needed when audio itself arrives live or the application needs a persistent connection.
Should a voice agent wait for the final transcript before responding?
That depends on the architecture and use case. A speech-to-speech agent may respond before asynchronous transcription completes. Systems that use transcript text for business decisions should define a separate readiness rule.
How should out-of-order transcript events be handled?
Associate events with the provider's item_id, maintain a per-item state machine, and make finalization idempotent. Do not infer turn identity from arrival order.
Does GPT Live Transcribe provide speaker labels or timestamps?
The current official guide says it does not provide word-level timestamps, speaker labels, or confidence scores. Use a compatible file transcription workflow or an application-level fallback when these are required.
Which latency setting should be used?
Start from the product's target for first text and final text, then benchmark representative audio. The named delay levels do not promise fixed milliseconds.
References
- OpenAI: GPT Live Transcribe model
- OpenAI: Realtime transcription guide
- OpenAI: Transcription workflow overview
- OpenAI: Realtime and audio overview
- Related: OpenAI low-latency voice AI architecture
- Related: Closed-loop voice agent analytics
Start by writing the event contract before selecting the dashboard. Once item identity, provisional state, finalization, and unsupported capabilities are explicit, latency and quality work becomes a controlled engineering problem.