A voice agent often needs a decision, not a transcript. Is the user finished? Did a required term occur? Which language is being spoken? Is the audio safe to route to a downstream workflow? Returning text and asking another model to infer the answer adds a second failure boundary.
A better interface declares the decision type, confidence semantics, abstention rule, and action boundary. The transcript remains useful evidence, but it is no longer the only product of speech processing.
Choose the type before choosing the model
Define the output contract for each audio question:
| Decision | Output type | Safe fallback |
|---|---|---|
| term present | boolean plus calibrated probability | ask for repetition or route to review below threshold |
| language | finite choice plus confidence | keep the current route and request confirmation |
| turn complete | boolean or boundary score | wait briefly; never treat silence as confirmation of intent |
| speaker change | event plus score | keep the stream unassigned until evidence improves |
| content risk | label plus policy version | quarantine or human review, not automatic enforcement |
The type is part of the safety contract. A score is not a yes/no answer until a threshold, calibration set, and abstention policy define how it becomes an action.
What Marimba demonstrates, and what it does not
The November 2026 Marimba paper presents a typed speech-decision interface built on frozen Qwen3-ASR 0.6B features. Its design separates lexical questions, non-lexical questions, turn-end readouts, and language identification. The paper reports different metrics for different tasks: decoder AUROC, Speech Commands accuracy, binary lexical expected calibration error, speaker-change AUROC, and turn-end AUROC.
Those are useful examples of interface specialization. They are not a production guarantee for your language, microphone, accent, latency budget, or call-center population. The paper uses private conversational sets for some results and public or named benchmarks for others. Keep the dataset, task, and metric attached to every number.
The repository and model card are implementation evidence. Read their version, preprocessing, checkpoints, and integration assumptions before copying an API shape. A model can expose a probability while still being poorly calibrated on your traffic.
Calibration is a deployment property
Suppose a detector emits 0.8 for a set of clips. If it is calibrated, roughly 80 percent of that set should contain the event under the declared label definition. Calibration depends on the population, label policy, windowing, and post-processing. It is not inherited from the word probability or from the fact that the output is a float.
Use a held-out calibration set that matches the deployment task. Report reliability diagrams or expected calibration error alongside discrimination metrics. Check calibration by language, channel, noise condition, speaker group, and audio duration when those partitions affect the workflow. Recalibrate after changing the encoder, window size, label definition, or routing policy.
Selective prediction beats forced confidence
For an agent that can ask a clarification question, abstention is a valid output. Define three regions:
score >= accept_threshold -> perform the low-risk action
score <= reject_threshold -> reject, defer, or ask for repetition
otherwise -> review or collect another window
Tune the thresholds against the cost of false acceptance, false rejection, delay, and human review. A single threshold is appropriate only when those costs are symmetric, which is rare for voice workflows. A billing command, a meeting turn boundary, and a language route should have different error budgets.
Keep the decision and the transcript separate
Store a trace that lets an operator reconstruct why the action happened:
{
"audio_window_id": "window-17",
"question_type": "turn_complete",
"decision": false,
"score": 0.21,
"calibration_set": "turn-end-2026-11",
"model_version": "...",
"threshold_policy": "voice-turn-v3",
"abstained": false,
"transcript_ref": "redacted://..."
}
The transcript can support review, but downstream code should consume the typed decision and its policy metadata. Keep raw audio and personal data under the application’s retention rules. This article provides an interface pattern, not permission to retain recordings.
Test the failure routes
Use synthetic and consented audio to exercise one boundary at a time:
- clipped speech at the window edge;
- overlapping speakers;
- code-switching and language ambiguity;
- background speech that contains the trigger term;
- silence after a partial answer;
- a confidence score above threshold but outside the calibration population;
- a model or calibration-policy version change.
For each case, verify the decision, abstention behavior, user-visible response, and audit record. Do not report a transcript that looks plausible as a successful test of a typed decision.
The production contract
Before connecting a typed speech decision to an agent action, document five things: the question definition, the output type, the calibration population, the threshold and abstention policy, and the action that follows. Then state the boundary: what the system does when audio is missing, uncertain, out of distribution, or no longer covered by the calibration set.
This design reduces one common voice-agent mistake: treating transcription quality as a proxy for decision quality. A good transcript can still produce a bad intent decision. A specialized detector can produce a useful decision while its transcript is incomplete. Measure the object the workflow actually consumes.
Sources and scope
- Amadori, Marimba: A System One Model for Speech, November 2026.
- Marimba API repository and model card, accessed November 17, 2026.
- Bayesian VAD, used as a comparison for event detection and calibration concerns.
The reported benchmark values belong to Marimba’s paper and datasets. The thresholds and JSON trace above are a deployment template, not measurements from this site.