Multi-Turn Eval

A framework for evaluating multi-turn LLM conversations with support for text, realtime audio, and speech-to-speech models.

The two benchmarks here in this public repo are:

• aiwf_long_context - older long-context benchmark described here
• aiwf_medium_context - newer medium-context benchmark

Thank you to Modal for providing compute resources for this benchmark. And to Charles Frye for advice about models and inference tuning.

Text mode models:

The standalone copy of this table is leaderboard-medium-context.md.

Rows are ordered by pass rate descending, then TTFAT P50 ascending. The solid dot is median time to the first user-visible answer token or tool-call output; the open dot and connecting hairline show P95. The dashed reference marks the approximate 700ms LLM-latency guideline for voice use.

This table emphasizes current-production routes while retaining selected historically important benchmark results. The four Nemotron rows are measurements from Baseten deployments that no longer exist in their original form; Baseten identifies the provider used for those published runs. An August 2026 freshness audit removed other retired routes, superseded previews, and unavailable self-hosted configurations. Their transcripts and dedicated study documents remain in the repository and its history.

Each conversation in this benchmark has 30 scripted turns. Refreshed filler-study rows use a fixed-denominator, attempt-based analysis in which missing, malformed, and post-abort future turns count as errors. Some legacy rows with early exits use their available observed turns and are therefore not directly comparable on completion reliability; detailed sample sizes and protocols live in the linked study artifacts rather than this summary table. Any Error is the percentage of turns where at least one of tool use, instruction following, or KB grounding fails; it is the complement of Pass Rate. The three dimension error rates overlap and therefore do not sum to Any Error.

TTFAT is the latency reported by the Pipecat service from request to the first user-visible answer token or tool-call output; reasoning deltas are excluded where the service exposes them separately. An optimized speech-to-speech pipeline with typical network latencies should be able to achieve a total voice-to-voice latency of approximately LLM TTFAT + 500ms. In general, a model with TTFAT above ~700ms is too slow for most voice agent use cases.

Models labeled "(thinking)" were run with reasoning/chain-of-thought enabled. Models labeled with a reasoning effort like "(low)", "(medium)", or "(none)" were run at that effort level on the OpenAI Responses API.

Gemini 3 rows labeled (minimal) use Google's lowest supported thinking_level, the closest current equivalent to no-think for latency testing. Google notes that minimal matches no thinking for most requests but does not guarantee that reasoning is completely off; these rows therefore are not labeled thinking off.

gemini-2.5-flash (thinking off) explicitly sets thinking_budget=0. This is also Pipecat's low-latency default for Gemini 2.5 Flash, but the benchmark pins it so every included run has an auditable thinking-off signature rather than relying on an implicit service default.

poolside/laguna-s-2.1 (thinking off) uses the exact paid poolside/laguna-s-2.1 OpenRouter route, served upstream by Poolside in BF16. The benchmark explicitly sets reasoning.enabled=false, so thinking is disabled; its TTFAT is specific to this OpenRouter/Poolside serving route.

muse-glimmer-30b (thinking low, GGUF) is the public quantized model served locally through llama.cpp on one RTX 5090 with a 32,768-token context, Q8_0 K/V cache, and DFlash draft length 15. A balanced, interleaved N=30-per-arm sweep used Meta's recommended temperature 1.0, top-p 0.95, and top-k 64 sampler, no request-level output-token cap, and the embedded chat template with exact chat_template_kwargs.reasoning_strength values. low scored 775/900 strict turns (86.1%), versus 769/900 (85.4%) for medium, 766/900 (85.1%) for high, and 764/900 (84.9%) for xhigh; every pairwise conversation-cluster bootstrap 95% interval includes zero. low is nevertheless the clear operational choice: it used 34.7% fewer mean completion tokens than high and measured 232ms P50 / 2799ms P95 TTFAT across scripted and recovery responses, versus 238ms / 7458ms for high. Meta documents no supported off mode: none and minimal only render unsupported literal labels, while top-level reasoning_effort=none and enable_thinking=false are template no-ops. See the reasoning-strength sweep and earlier high-strength campaign.

gemma-4-31b-it (thinking off) replaces the retired Lilac route with the official google/gemma-4-31B-it BF16 checkpoint on a dedicated Baseten deployment. The selected serving stack is SGLang v0.5.16 on two H100s with RadixAttention prefix caching and the official Gemma 4 NEXTN/MTP assistant. A matched three-conversation bakeoff measured 422ms P50 / 537ms P95 TTFAT for NEXTN/MTP, versus 434ms / 648ms for vLLM with prefix caching and no MTP, and 496ms / 728ms for SGLang without MTP. The published row pools an immutable 30-conversation cohort with a matched 120-conversation extension: all 150 conversations completed all 30 scripted turns, scored 4346/4500 strict turns (96.6%, conversation-cluster bootstrap 95% CI 96.1–97.0%), and produced zero thinking tokens. Pooled TTFAT was 490ms P50 / 718ms P95, but the extension exposed intermittent serving stalls: 27/3600 turns exceeded 10 seconds and the maximum was 38.25 seconds. SGLang's Gemma 4 parser incorrectly emits request-schema tool positions as OpenAI streaming call indices; the campaign uses an explicit, default-off compatibility option that remaps them to response-local ordinals. See the pooled campaign report, original campaign, and serving bakeoff. The historical Lilac 97.7% aggregate remains in docs/ten-run-aggregates/lilac-gemma-4-31b-it-off-2026-06-15.txt and is not pooled with the Baseten result.

A separate RTX 5090 serving study used the public Red Hat NVFP4 checkpoint at batch one with the same no-filler benchmark configuration. Pooled N=150 cohorts scored 95.5% with FP8 KV and 96.2% with a compact BF16-KV layout. BF16 minus FP8 KV was +0.67 percentage points with an independent whole-conversation bootstrap 95% interval of -0.07 to +1.40 points: the point estimate favors BF16, but the global difference remains inconclusive. BF16 measured 128ms P50 / 336ms P95 TTFAT versus 105ms / 309ms for FP8; both completed 150/150 conversations and retained 100% KB grounding. The Baseten BF16 row remains the production-table result because the local experiment uses quantized weights and omits MTP. See the pooled local three-way comparison.

The two historical kimi-k2.6 Cerebras rows have been removed from the current-production table, but their content-aware TTFAT results remain useful serving context: thinking measured 452ms P50, 1350ms P95, and 3483ms max; instant measured 256ms P50, 480ms P95, and 1639ms max. The exact Cerebras route used by those runs is no longer generally available and now returns 404. Access to Kimi K2.6 on Cerebras requires a large commercial commitment, so these results are not representative of a route most developers can use, and the old accuracy or latency numbers are not transferred to another provider. The two current Kimi rows are separate 30-conversation, fixed-900-turn campaigns against Baseten's OpenAI-compatible Model API, model ID moonshotai/Kimi-K2.6, with no filler. Thinking on explicitly sets chat_template_args.enable_thinking=true, temperature 1.0, and top-p 0.95; it scored 885/900 strict turns (98.3%, conversation-cluster bootstrap 95% CI 97.7–99.0%) and measured 1596ms P50 TTFAT. Thinking off omits chat_template_args, leaving Baseten's default-off behavior, and uses temperature 0.6 with provider-default top-p; it scored 845/900 strict turns (93.9%, 95% CI 92.3–95.3%) and measured 480ms P50 TTFAT. Although the off request transmitted reasoning_effort=none, Baseten does not support that control for Kimi K2.6 and ignores it; zero thinking tokens on all 900 off-cohort turns confirm the effective state. Thinking on called end_session correctly on scripted turn 29 in 30/30 conversations, while thinking off did so in 9/30, used recovery in 16/30, and omitted it in 5/30. This reproduces the two vendor-mode sampling signatures rather than changing only one setting. See the paired comparison, the individual thinking-on and thinking-off Baseten reports, and the historical Cerebras notes.

claude-fable-5 cannot run without thinking: adaptive thinking is always on for this model and thinking: {"type": "disabled"} is rejected by the API. The "(default)" row is the out-of-the-box configuration (adaptive thinking at the default effort, high); the "(low)" row sets output_config: {"effort": "low"}. Both rows request thinking.display: "summarized" and measure TTFAT to the first non-thinking token, the same convention used for other reasoning models in this table. We planned a low/medium/high/xhigh effort sweep but stopped after low: with a TTFAT P50 above 3.5s, no higher effort level can pass the ~1500ms bar for voice use. A follow-up "voice-optimized" probe (effort low, thinking.display: "omitted", plus a system-prompt instruction suppressing deliberation) cut median thinking delay to 14ms and still measured 2980ms median TTFAT at 100% pass rate — the latency is serving-side prefill, not reasoning. See docs/claude-fable-5-sweep-2026-06-09.md.

claude-sonnet-5 is run with thinking disabled (thinking: {"type": "disabled"}). Unlike claude-sonnet-4-6, Sonnet 5 runs adaptive thinking by default when the thinking parameter is omitted, so an explicit disable is required to measure the no-thinking (voice) configuration. A paired output_config: {"effort": "low"} adaptive comparison scored the same pass rate within noise (92.0%) at ~600ms higher median TTFAT (1802ms) and ~1.7s higher P95 (4202ms), so low-effort thinking buys no accuracy on this benchmark and is omitted from the voice row. The remaining ~7% is a stochastic over-confirmation failure (the model re-asks for a user name it already collected instead of calling the tool); KB grounding and turn-taking are perfect. See docs/ten-run-aggregates/claude-sonnet-5-disabled-2026-07-01.txt and docs/ten-run-aggregates/claude-sonnet-5-low-2026-07-01.txt.

gpt-5.6-terra and gpt-5.6-luna are two of the three GPT-5.6 versions (the third, sol, is the flagship). Like gpt-5.4, GPT-5.6 requires the OpenAI Responses API when tools and a reasoning effort are combined — reasoning_effort with function tools returns a 400 on /v1/chat/completions. The parenthesized label is the reasoning_effort level. The terra (medium) aggregate excludes a transient OpenAI overload. luna (none) runs with reasoning off; its 12s TTFAT Max is an OpenAI-overload artifact from the capture window, not representative (P50 671ms). The refreshed terra (none) and sol (none) rows use the fixed-denominator study pools described below.

inkling is Thinking Machines' earlier 975B-parameter (41B active) open-weights model. Its historical (none) row uses Baseten's serverless Model API and should not be confused with the newer Inkling Small results below. See docs/inkling-notes.md and docs/inkling-baseten-integration.md.

inkling-small is Thinking Machines' newer smaller model, tested through the Baseten Model API in a frozen paired effort campaign. Its historical aggregates were 75.1% at 279ms P50 TTFAT with reasoning_effort=none and 51.7% at 277ms with low; the separate exploratory +96-dot arm was +1.8 points versus the frozen none control. The model has been removed from the current-production table after the August 2026 freshness audit: none completed its current 30-turn probe, but low produced no complete conversation in 10 attempts (eight ended at turn 16 and two stalled without a valid runtime record). The frozen results remain in the campaign artifacts and Section 3 of the filler report.

gemma-4-26b-a4b-it (thinking off) uses the official google/gemma-4-26B-A4B-it checkpoint on a dedicated Baseten vLLM endpoint with automatic prefix caching and one-token MTP speculative decoding. The row contains 30 no-filler conversations, all of which completed the full protocol. Its 167 errors are highly concentrated: 163 occur on six workflow turns involving a second suggestion, remembered user details, technical support, venue/reference follow-ups, and voting. KB grounding is perfect, but the model repeatedly asks for known information instead of invoking the required tool and on two workflow turns repeatedly falls back to an inappropriate generic scope deflection. See ops/baseten-gemma4-26b-a4b-vllm/analysis/REPORT.md.

The Qwen3.6 rows use official Qwen/Qwen3.6-27B BF16 and Qwen/Qwen3.6-35B-A3B-FP8 checkpoints on dedicated Baseten vLLM 0.26 endpoints with automatic prefix caching and two-token MTP speculative decoding. Native thinking is explicitly disabled. Each no-filler row contains 30 conversations and uses the same fixed 900-turn denominator as the focused filler-study rows; missing future turns after an early exit count as errors. Qwen3.6-27B completed all 30 conversations, while Qwen3.6-35B-A3B completed 27/30. See docs/filler-study-data/qwen36-dots-2026-07-28/protocol.md and ops/baseten-qwen36-27b-vllm/aiewf-medium-qwen36-baseten-vllm026-apc-mtp-n30-20260728T110824Z/analysis/REPORT.md.

gpt-5.4 (none, +96 dots) is a filler-token experiment (arxiv 2607.03502), not the standard fixed-prompt config: gpt-5.4 is run thinking-off (reasoning_effort: none) with 96 space-separated dots appended to the final user turn of each request (the conversation history is left filler-free). The refreshed fixed-denominator comparison is summarized in the table and analyzed in docs/filler-token-latent-scratchpad-study.md. Set via MTE_FILLER_DOTS.

gpt-5.4-mini has been removed from the current-production table because the August 2026 freshness audit found severe premature-termination behavior. At medium, only 1/6 attempts completed: four called end_session incorrectly on scripted turn 13, and one reached only turn 28. At none, 0/4 completed; all four called end_session on turn 13. The historical aggregates remain in the study artifacts. gpt-4.1-mini remains in the table and completed its freshness conversation on the first attempt.

Speech-to-speech models:

For speech-to-speech models, we measure voice-to-voice latency by analyzing the conversation recording. We measure the overall time from the end of the user's speech to the beginning of the model's speech response.

For voice agent use cases, voice-to-voice latency needs to be under 1,500ms.

The voice-to-voice latency measured here is different from the TTFB reported by the Pipecat service for these models, because all of these models were tested in server-side VAD mode (the server-side turn delay is opaque to the Pipecat pipeline), and all of the models send initial silence bytes before actual speech audio. (Text-to-speech models do this, too. The initial silence segments are typically between 150ms and 250ms for standalone TTS models.)

We also include a "Turn Ok" column for these models, which counts how often the model does not respond at all when we expect it or responds with control characters, API refusals, or generic errors.

The Nova 2 Sonic model performs very well on instruction following and tool calling but has a high rate of safety refusals for normal content. It also has a connection limit of 8 minutes. Fixes for both of these issues are in flight from AWS.

gemini-3.1-flash-live-preview is a reasoning model. We show the results with minimal thinking enabled, above. Here are the results all thinking levels. Even minimal is too slow for voice agent use cases, today. With additional reasoning, the model's benchmark results improve, but latency increases. Note the tool use and instruction following regression from medium to high. This is something we see regularly in multi-turn testing of thinking models. The model is not always "smartest" at the highest thinking setting.

Listen to full 30-turn benchmark conversations from each speech-to-speech model:

In the recordings, the left channel is the user's speech and the right channel is the model's speech. You'll hear a small audio tag (a short beep) at the start of each model speech segment. The benchmarking code adds this audio tag to make it easier to double-check track alignment and to provide a visual indicator of segment start for manually inspecting the recordings.

Visually inspecting silence padding by looking at the audio waveform. Here we're using the Descript editor.

# Install dependencies
uv sync

# List available benchmarks
uv run multi-turn-eval list-benchmarks

# Run a benchmark. Results will be saved to runs/aiwf_medium_context/<timestamp>_<model_name>
uv run multi-turn-eval run aiwf_medium_context --model claude-sonnet-4-5 --service anthropic

# Judge the results
uv run multi-turn-eval judge runs/aiwf_medium_context/<timestamp>_claude-sonnet-4-5

Requires Python 3.12+ and uv.

git clone <repo-url>
cd multi-turn-eval
uv sync

Set the appropriate API keys for the services you want to use:

# Required for judging (Claude evaluates all benchmark results)
export ANTHROPIC_API_KEY=sk-ant-...

# Text-mode model services
export OPENAI_API_KEY=sk-...          # OpenAI (GPT-4o, GPT-4o-mini, etc.)
export GOOGLE_API_KEY=...             # Google (Gemini models)
export OPENROUTER_API_KEY=...         # OpenRouter (access to multiple providers)
export GROQ_API_KEY=...               # Groq
export CEREBRAS_API_KEY=...           # Cerebras

# Speech-to-speech model services
export OPENAI_API_KEY=sk-...          # OpenAI Realtime (gpt-realtime)
export GOOGLE_API_KEY=...             # Gemini Live (gemini-native-audio)
export ULTRAVOX_API_KEY=...           # Ultravox (ultravox-v0.7)
export XAI_API_KEY=...                # xAI Grok Realtime (grok-realtime)

# AWS Nova models (text and speech-to-speech)
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SESSION_TOKEN=...          # Optional: for temporary credentials
export AWS_REGION=us-east-1           # Optional: defaults to us-east-1

You can also create a .env file in the project root with these variables.

# Basic usage with text model
uv run multi-turn-eval run <benchmark> --model <model> --service <service>

# Examples:
uv run multi-turn-eval run aiwf_medium_context --model claude-sonnet-4-5 --service anthropic
uv run multi-turn-eval run aiwf_medium_context --model gpt-4o --service openai
uv run multi-turn-eval run aiwf_medium_context --model gemini-2.5-flash --service google

# Realtime audio models 
uv run multi-turn-eval run aiwf_medium_context --model gpt-realtime --service openai-realtime
uv run multi-turn-eval run aiwf_medium_context --model gemini-3.1-flash-live-preview --service gemini-live --thinking minimal
uv run multi-turn-eval run aiwf_medium_context --model ultravox-v0.7 --service ultravox-realtime

# Nova Sonic (no --service needed, pipeline creates its own LLM)
uv run multi-turn-eval run aiwf_medium_context --model amazon.nova-2-sonic-v1:0 --pipeline nova-sonic

# Grok (xAI) Realtime
uv run multi-turn-eval run aiwf_medium_context --model grok-voice-think-fast-1.0 --pipeline grok-realtime

# Debug with limited turns
uv run multi-turn-eval run aiwf_medium_context --model gpt-4o --service openai --only-turns 0,1,2

# Verbose logging
uv run multi-turn-eval run aiwf_medium_context --model gpt-4o --service openai --verbose

After a benchmark run completes, judge the results using Claude:

# Judge a specific run
uv run multi-turn-eval judge runs/aiwf_medium_context/20251213T123456_claude-sonnet-4-5

# Judge with specific turns
uv run multi-turn-eval judge runs/aiwf_medium_context/20251213T123456_claude-sonnet-4-5 --only-turns 0,1,2

# Use a different judge model
uv run multi-turn-eval judge runs/aiwf_medium_context/20251213T123456_claude-sonnet-4-5 --judge-model claude-sonnet-4-5

Judge outputs (saved to the run directory):

• claude_summary.json - Score metrics
• claude_analysis.md - Human-readable report with failures
• claude_judged.jsonl - Per-turn judgments with reasoning

For speech-to-speech model runs (OpenAI Realtime, Gemini Live, Ultravox, etc.), the judge automatically performs turn-taking analysis when a conversation.wav file is present. This adds a 4th evaluation dimension:

Turn-taking failures are detected when:

• Missing timing data: No user speech detected before bot response
• Negative TTFB: Bot started speaking before user finished
• Empty response: Model returned only control tokens (e.g., <ctrl46>)
• No response: Model never responded (15s timeout)
• Alignment drift: Audio tag positions drift >150ms from expected
• Audio overlap: User and bot speaking simultaneously

When turn-taking failures occur, the judge is more lenient on instruction_following since garbled audio may cause transcription issues.

# Judge a speech-to-speech run (turn-taking analysis runs automatically)
uv run multi-turn-eval judge runs/aiwf_medium_context/20260111T123456_gpt-realtime_abc123

# Skip turn-taking analysis
uv run multi-turn-eval judge runs/aiwf_medium_context/20260111T123456_gpt-realtime_abc123 --skip-turn-taking

Additional outputs for speech-to-speech runs:

• claude_summary.json includes turn_taking_failures list and turn_taking_affected_instruction count
• claude_analysis.md includes Turn-Taking Analysis section with per-turn issues
• claude_judged.jsonl includes turn_taking_issues array for failed turns

# List available benchmarks
uv run multi-turn-eval list-benchmarks

# List available pipelines
uv run multi-turn-eval list-pipelines

# List service aliases
uv run multi-turn-eval list-aliases

For convenience, common service classes have short aliases:

You can also use fully-qualified class names:

uv run multi-turn-eval run aiwf_medium_context \
    --model gpt-4o \
    --service pipecat.services.openai.llm.OpenAILLMService

Benchmarks are located in benchmarks/. Each benchmark is a Python package with:

• config.py - Benchmark configuration (turns, tools, system instruction)
• prompts/system.py - System prompt with knowledge base
• data/knowledge_base.txt - Knowledge base content

Both benchmarks share the same 30 turns, tools, and audio files. Only the knowledge base size differs.

Runs are saved to runs/<benchmark>/<timestamp>_<model>/:

runs/
└── aiwf_medium_context/
    └── 20251213T123456_claude-sonnet-4-5/
        ├── transcript.jsonl        # Turn-by-turn results
        ├── runtime.json            # Run metadata and metrics
        ├── run.log                 # Debug logs
        ├── claude_summary.json     # Judge summary (after judging)
        ├── claude_judged.jsonl     # Per-turn judgments (after judging)
        └── claude_analysis.md      # Human-readable analysis (after judging)

multi-turn-eval/
├── src/multi_turn_eval/           # Main package
│   ├── cli.py                     # CLI entry point
│   ├── pipelines/                 # Pipeline implementations
│   │   ├── base.py                # Abstract base pipeline
│   │   ├── text.py                # Text pipeline
│   │   ├── realtime.py            # Realtime pipeline (OpenAI/Gemini)
│   │   └── nova_sonic.py          # Nova Sonic pipeline
│   ├── processors/                # Frame processors
│   │   ├── tool_call_recorder.py  # Records tool calls
│   │   └── tts_transcript.py      # TTS transcript handling
│   ├── transports/                # Input/output transports
│   │   ├── paced_input.py         # Paced audio input
│   │   └── null_audio_output.py   # Null audio sink
│   ├── recording/                 # Transcript recording
│   │   └── transcript_recorder.py # Records transcripts
│   └── judging/                   # Judge implementations
│       └── claude_judge.py        # Claude-based judging
│
├── benchmarks/                    # Benchmark definitions
│   ├── _shared/                   # Shared benchmark data
│   │   ├── turns.py               # 30 turns with golden data
│   │   ├── tools.py               # Tool/function definitions
│   │   └── audio/                 # Audio files for turns
│   ├── aiwf_long_context/         # Long context benchmark
│   └── aiwf_medium_context/       # Medium context benchmark
│
├── runs/                          # Output directory (gitignored)
├── scripts/                       # Utility scripts
└── pyproject.toml                 # Project configuration

To use a git branch of pipecat instead of the PyPI release, edit pyproject.toml:

[tool.uv.sources]
pipecat-ai = { git = "https://github.com/pipecat-ai/pipecat.git", rev = "main" }

Then run uv sync to update.

The Claude judge evaluates each turn on three dimensions:

1. tool_use_correct - Did the assistant call the expected function with correct arguments?
2. instruction_following - Did the assistant answer the question or advance the task?
3. kb_grounding - Is the response factually consistent with the knowledge base?

For detailed per-turn timing analysis of speech-to-speech models, use the comprehensive metrics script:

# Analyze a run with summary statistics
uv run python scripts/analyze_turn_metrics.py runs/aiwf_medium_context/<timestamp>_<model>

# Show per-turn breakdown table
uv run python scripts/analyze_turn_metrics.py runs/aiwf_medium_context/<timestamp>_<model> -v

# Output as JSON (for programmatic use)
uv run python scripts/analyze_turn_metrics.py runs/aiwf_medium_context/<timestamp>_<model> --json

The script consolidates timing data from multiple sources and calculates the following metrics:

Key metric relationships:

• WAV V2V = Pipeline TTFB + Silent Pad (VAD) - The total voice-to-voice latency includes both the time waiting for audio to arrive and any initial silence in the audio stream
• Pipeline TTFB measures when audio starts arriving at the pipeline
• Silent Pad measures how much silence is at the beginning of the audio (most models send 40-120ms of silence before speech)

The script verifies that log-based timestamps match actual audio positions by detecting audio tags (2kHz tones) embedded in the WAV file:

• Bot tags: Inserted when bot audio arrives at the pipeline
• Alignment OK: Log and WAV positions match within ±20ms tolerance
• Issues detected: Missing tags, extra tags, or drift outside tolerance

When run with --json, the script outputs structured data that can be saved:

# Save metrics to JSON file
uv run python scripts/analyze_turn_metrics.py runs/aiwf_medium_context/<timestamp>_<model> --json > turn_metrics.json

Use this prompt with Claude Code to run comprehensive benchmarks across multiple speech-to-speech models:

Run a full 30-turn test with all four speech-to-speech models: ultravox-v0.7,
gpt-realtime, grok-realtime, gemini-2.5-flash-native-audio-preview-12-2025.

For each model:
1. Run the 30-turn benchmark
2. Analyze using scripts/analyze_turn_metrics.py and save turn_metrics.json
3. Judge the model performance using the Claude judge

After completing all models, create a summary comparison table with these columns:
- Model
- Tool Use (X/30)
- Instruction (X/30)
- KB Ground (X/30)
- Turn Ok (X/30)
- Pass Rate
- Non-Tool V2V Median
- Non-Tool V2V Max
- Tool V2V Mean
- Silence Pad Mean

Separate metrics for tool-call turns vs non-tool-call turns in the analysis.

This will run all four models (which takes approximately 15-20 minutes each), analyze their timing metrics, judge their responses, and produce a comparison table.

MIT