Your Voice Pipeline Is Too Flexible

Voice does not change what a tool needs. It changes what arrives first: audio, then a string, then a candidate payload that might still be wrong.

In Your Data Structure Is Too Flexible, a string crossed an HTTP boundary where the ledger expected an int. The stack trace blamed infrastructure. The bug was the handler that accepted the body.

Voice puts another door in front of that one. A reservation tool still wants a contract. Speech-to-text gives you a transcript. That transcript is not a tool argument any more than a raw JSON dict is a charge request.

The rule: Strict at the edges. A transcript is untrusted input. Only validated JSON crosses into the tool.

This piece follows one booking from mic to side effect: voice → transcribe → extract → validate → tool.

The runnable code lives in this repository.

Staging lied about the mic

Every voice demo uses some version of this utterance:

“Book me a table for four tomorrow at seven, outdoor if you have it.”

Whisper returns a readable transcript. You paste it into a prompt that says “return JSON”.

The model fills party_size, date, time, and seating. The booking lands. The pipeline looks finished.

Your reservation tool never wanted readable text. It wanted:

party_size: int
date: date
time: time
seating: Literal["indoor", "outdoor"] | None

If the extractor invents guests: "four", drops the date, or maps “patio” into seating, the booking is wrong even when every word in the transcript was transcribed correctly.

Staging did not lie about speech-to-text. It lied about the contract. The take was clean, the speaker was you, and nobody asked what happens when speech is partial, ambiguous, or wrong.

Production gets a different speaker. They trail off before the date. They say “Friday” with no week in context. They correct themselves mid-sentence: “actually make that eight, not four”. The model fills a required slot with a guess, coerces “a few people” into 3, or passes a free-text blob into a typed argument.

Those are not Word Error Rate (WER) failures. Whisper heard the words. The payload still does not match the tool. The bug is the same family as the flexible dict: untrusted input crossed the door, and nothing checked the shape.

Transcription quality is a UX problem. The boundary that protects the tool is the one that refuses the call until the payload validates.

Don’t ask one prompt to do two jobs

The staging lie often hides in a single instruction: “Transcribe this audio and return JSON for the booking tool.”

It works on a clean take. Under real speech it breaks in ways you cannot debug. The user corrects a time mid-utterance; the model has to choose between the transcript text and the fields it already filled. A required slot is missing; the model invents one to satisfy JSON rather than fail closed.

When party_size comes back wrong, you cannot tell whether Whisper misheard “four” or the extractor guessed. There is no stable artifact to re-run extraction against, and no clean place to validate before the tool call.

Keep a handoff.

Speech-to-text produces a Transcript: text, plus optional confidence and timestamps.

The tool never sees audio or raw STT output.

If extraction fails, retry or clarify against the same transcript. If STT fails, fix recognition. Do not ask the schema to guess from noise.

from pydantic import BaseModel, Field


class Transcript(BaseModel):
    """STT artifact. Text only — not a tool argument."""

    text: str
    confidence: float | None = Field(default=None, ge=0.0, le=1.0)
    timestamps: list[tuple[float, float, str]] | None = None

WER measures recognition accuracy. It does not measure whether the payload matches the tool.

The candidate payload still needs an explicit contract: which fields, which types, what is required.

That contract is the tool schema, the same Pydantic model extraction targets and validation checks against.

Name what each stage owns

Follow one utterance through five stages. If a design review cannot name what each stage owns, the boundary will leak, and the leak will look like “the model is bad at voice”.

Voice owns capture and audio quality. It does not own meaning or fields.

Transcribe owns speech-to-text. It produces a Transcript. WER belongs here.

Extract owns turning that string into a candidate payload shaped like the tool contract, usually with an LLM and PydanticAI. It may guess. It must not be the last gate before the tool.

Validate owns the schema: types, required fields, enums, constraints. On failure it rejects or returns structured errors. It does not “improve” the transcript or invent missing slots to be helpful.

Tool owns side effects: booking, query, write. It receives only validated data, never a raw transcript, an untyped dict, or best-effort JSON from the model.

In the repository, that map is wired in src/voice_structured/pipeline.py. Capture and transcribe produce a Transcript. Extraction and validation produce a ReservationRequest. The tool receives only what passed both gates.

The invariant from the earlier article, applied twice: once at transcript → payload, once at payload → tool. Fail at the door with a message about the contract. Do not debug a wrong booking three layers down.

The contract is the spine

Once transcription and extraction are separate, the tool schema does the same job ChargeRequest did for inbound JSON.

Write it as a Pydantic model: field names, types, required vs optional, and the constraints the tool will enforce anyway.

Extraction targets that type. Validation checks against it. The tool re-validates before any side effect.

from datetime import date, time
from typing import Literal

from pydantic import BaseModel, Field


class ReservationRequest(BaseModel):
    party_size: int = Field(ge=1, le=20)
    date: date
    time: time
    seating: Literal["indoor", "outdoor"] | None = None

PydanticAI puts that contract on the agent. The transcript is the input.

The agent’s job is to produce a ReservationRequest, not free-form JSON.

Relative dates such as “tomorrow” resolve against today in the system prompt, still extraction, not STT.

from datetime import date

from pydantic_ai import Agent

from voice_structured.schema import ReservationRequest


def build_reservation_agent() -> Agent[None, ReservationRequest]:
    today = date.today().isoformat()
    return Agent(
        "openai:gpt-4o-mini",
        output_type=ReservationRequest,
        retries={"output": 2},
        system_prompt=(
            "Extract reservation fields from the transcript. "
            "Do not invent values for missing slots. "
            f"Today's date is {today}. "
            "Resolve relative dates such as 'tomorrow' against that. "
            "party_size must be an integer. "
            "seating is indoor, outdoor, or omit if unknown."
        ),
    )

The agent proposes a payload. The schema accepts or rejects it inside run().

Output retries feed structured validation errors back to the model, not a brand-new free-form prompt.

The side-effect tool receives validated JSON only:

from voice_structured.schema import ReservationRequest


async def book_table(payload: dict) -> dict:
    """Side-effect tool. Receives validated JSON only — never a raw transcript."""

    ReservationRequest.model_validate(payload)
    return {"status": "booked", "reservation": payload}

handle_transcript in pipeline.py is the orchestration point.

Empty text refuses. STT confidence below 0.5 clarifies and skips extraction.

A valid ReservationRequest is the only path that calls book_table.

When the output-retry budget is exhausted, PydanticAI raises UnexpectedModelBehavior, catch it and clarify. Do not call the tool.

async def handle_transcript(
    transcript: str | Transcript,
    *,
    agent: Agent[None, ReservationRequest] | None = None,
) -> PipelineResult:
    artifact = (
        transcript if isinstance(transcript, Transcript) else Transcript(text=transcript)
    )
    extractor = agent or build_reservation_agent()

    if not artifact.text.strip():
        return await clarify_or_refuse(artifact)
    if artifact.confidence is not None and artifact.confidence < LOW_CONFIDENCE:
        return await clarify_or_refuse(artifact)

    try:
        result = await extractor.run(artifact.text)
    except UnexpectedModelBehavior:
        return await clarify_or_refuse(artifact)

    payload: ReservationRequest = result.output
    await book_table(payload.model_dump())
    return PipelineResult(
        outcome="booked",
        message="Reservation booked.",
        payload=payload,
    )

The CLI --transcript flag skips the mic so you can test extract → validate → tool as text. LOW_CONFIDENCE lives in transcribe.py.

Tune it per domain and STT vendor; 0.5 is a starting point for whole-utterance scores.

The repository wires the model from OPENAI_API_KEY / VOICE_STRUCTURED_MODEL; the snippet above is the spine.

When speech is wrong

The clean demo utterance is not the story. The story is what happens when the user says “book a table for four” and stops.

Validation failure is normal with voice. The useful question is which kind you have, and whether you retry, clarify, or refuse.

Partial: a required slot was never spoken. “Book a table for four” with no date.

Ambiguous: the transcript has a value the schema cannot resolve. “Friday” with no week. “Around seven” with no time.

Wrong: extraction or coercion produces an invalid payload. party_size: "four". seating: "patio" outside the enum.

Each maps to a different response at the boundary.

Retry extraction inside Agent(retries=…) when the speech might be mappable but the first pass guessed or dropped a field. PydanticAI feeds structured validation errors back to the LLM.

Clarify when a required slot is genuinely missing or ambiguous. Ask for the date, the time, or seating. Keep the partial payload out of the tool until the user fills the gap. Voice UX usually tolerates one visible clarify round better than three silent retries. Cap the attempts and fail where the user can see it.

Refuse when the request cannot be satisfied safely: empty audio, repeated validation failure, nonsense, or a slot the tool cannot represent. Do not call the tool with a best-effort fill.

build_reservation_agent() sets retries={"output": 2}. When that budget dies, handle_transcript catches UnexpectedModelBehavior and routes to clarify_or_refuse.

Low STT confidence never reaches the extractor. Empty speech refuses immediately.

LOW_CONFIDENCE = 0.5


async def clarify_or_refuse(transcript: Transcript) -> PipelineResult:
    text = transcript.text.strip()
    if not text:
        return PipelineResult(
            outcome="refuse",
            message="No speech to extract. Refusing — the tool will not be called.",
        )
    if transcript.confidence is not None and transcript.confidence < LOW_CONFIDENCE:
        return PipelineResult(
            outcome="clarify",
            message=(
                "STT confidence is too low to coerce into typed fields. "
                "Please repeat the date, time, party size, and seating."
            ),
        )
    return PipelineResult(
        outcome="clarify",
        message=(
            "Could not fill the reservation schema from this transcript. "
            "Please give a date, a time, and party size "
            "(and indoor or outdoor if you care)."
        ),
    )

handle_transcript never calls book_table on those paths.

Low STT confidence on an utterance is a clarify signal, not a reason to coerce into a typed field.

Log validation failures at the schema boundary: field paths (date, party_size), attempt count, and outcome. src/voice_structured/observability.py has a minimal log_boundary hook.

That is how you debug a bad booking in production instead of chasing a stack trace inside the reservation service.

Try it

Install from the repository: pip install -e ".[dev]", copy .env.example to .env, set OPENAI_API_KEY. Full setup is in the README. The --transcript flag skips capture.

Happy path: full utterance, validated booking:

python -m voice_structured --transcript "Book me a table for four tomorrow at seven, outdoor if you have it."

{
  "outcome": "booked",
  "message": "Reservation booked."
}
{
  "party_size": 4,
  "date": "2026-09-02",
  "time": "19:00:00",
  "seating": "outdoor"
}

Dates resolve against the day you run the command. The important part is outcome: booked with a validated ReservationRequest, the same object book_table receives.

Partial speech: output retries exhaust, the pipeline clarifies, and book_table is not called:

python -m voice_structured --transcript "Book a table for four."

{
  "outcome": "clarify",
  "message": "Could not fill the reservation schema from this transcript. Please give a date, a time, and party size (and indoor or outdoor if you care)."
}

No second JSON block means the tool was not called. That is the designed path for "Book a table for four." and similar partial utterances. Live models may still invent missing slots despite the system prompt. Treat that as prompt tuning, not a reason to drop the schema gate.

Low STT confidence: extraction never runs:

python -m voice_structured --transcript "Book me a table for four tomorrow at seven." --confidence 0.2

{
  "outcome": "clarify",
  "message": "STT confidence is too low to coerce into typed fields. Please repeat the date, time, party size, and seating."
}

Again, no reservation payload.

What this isn't

This is not an argument against Whisper, or against asking a model to extract fields. Extraction is how you get from a string to a candidate. The point is that extraction is not the door.

It is also not a voice-UX handbook. One clarify round, a confidence threshold, and a refuse path are enough to prove the design rule. They are not a dialogue manager.

Skip the conclusion that every voice app needs this exact five-stage graph. Dicts, transcripts, and JSON blobs are fine as owned data after a boundary has already checked the shape.

The problem is treating untrusted speech as if it were already a tool argument.

The rule, twice

Voice adds a door. It does not replace the one you already needed.

Map the pipeline so each stage owns one transformation. Keep transcription and extraction separate so you can re-run and debug. Define the tool contract in Pydantic and extract into it with PydanticAI.

When speech is partial, ambiguous, or invalid, retry inside the agent, clarify with the user, or refuse.

Never call the tool with a transcript, an untyped dict, or a payload that failed validation.

Validated JSON at the tool boundary, every time.

For HTTP and config boundaries, see Your Data Structure Is Too Flexible. For voice, add one more door, and keep the same rule: validate, then move on.

Follow me on Twitter: https://twitter.com/DevAsService

Follow me on Instagram: https://www.instagram.com/devasservice/

Follow me on TikTok: https://www.tiktok.com/@devasservice

Follow me on YouTube: https://www.youtube.com/@DevAsService