Series: AgentZero Lite — Part 8. The previous post (Part 7 — Step 2 Landed: Akka.NET Streams Now Carry the Whole Voice Path) put a streaming substrate under the voice subsystem. This one puts a measurement instrument on top — so we know whether it's getting better or worse between commits.

TL;DR

• 5-case xUnit suite that runs TTS (Windows SAPI default voice) → WAV → resample → STT (Whisper.net medium, CPU) and compares the transcript to the input.
• Pure-model, file-based, no Akka, no streams. Deterministic baseline; the streaming variant becomes measurable as a delta against it.
• Comparison folds case + punctuation + whitespace before equality so tokeniser cosmetics ("hello" → "Hello.") aren't reported as content drift.
• 4 / 5 pass. The one fail is the interesting case — Whisper auto-corrected the user's input typo 모래 (sand) to the contextually correct 모레 (day after tomorrow). Edit distance 1, similarity 96.8 %. The test reports the drift honestly; the post argues it should.
• Six lessons worth writing down so the next person doesn't re-derive them. The middle four are technique; the bookends are philosophy.

Implementation: Project/AgentTest/Voice/TtsSttRoundTripTests.cs, commit b1a0ec8.

Why we needed this now

The Part 7 → Part 8 gap is a classic refactor-then-instrument move.

By the end of Part 7 the voice pipeline had a streaming substrate, a barge-in detector, a sentence chunker, and an opt-in feature flag. What it didn't have was a way to answer the question "is the new version better?" — every quality check was a human listening to a speaker and squinting at a transcript.

For a single change that's fine. For a series of small tuning passes (slow the SAPI rate by 15 %, pad the WAV with 1 s of room noise, switch the resampler, swap the model) it's the wrong loop. We were debating whether 안녕하세요 came out as 안녕하세요 or as Thank you for watching, please subscribe and like based on memory. Time to measure.

Constraints I set going in:

• No mic, no speaker. The acoustic path has its own problems (Windows mic enhancements, room noise, hardware variance). Validating the raw model pair means a file-based round-trip — synthesise to WAV, hand the WAV to STT.
• No actors, no streams. Those are operational concerns. The question here is "what do TTS and STT do to a string when neither has any other moving parts in the way?"
• Real providers. Mocking either end defeats the point — we want to know the actual hallucination rate of the actual deployed pipeline, not the rate of a mock.

That gave the topology that landed:

flowchart LR
    A["input string"] --> B["WindowsTts.SynthesizeAsync"]
    B --> C["WAV bytes"]
    C --> D["File.WriteAllBytes<br/>*-tts.wav"]
    C --> E["WavToPcm.To16kMono<br/>(NAudio resample)"]
    E --> F["PCM 16k mono"]
    F --> G["File.WriteAllBytes<br/>*-stt-input-16k.wav"]
    F --> H["WhisperLocalStt.TranscribeAsync<br/>(medium, CPU)"]
    H --> I["transcript string"]
    A --> J["Normalize<br/>(case + punct + WS)"]
    I --> J
    J --> K{"== ?"}
    K --> L["VERDICT + diff + timings"]

    classDef io fill:#1f2937,stroke:#0ea5e9,color:#e5e7eb
    classDef pure fill:#0b3d2e,stroke:#10b981,color:#d1fae5
    class B,H,E pure
    class D,G io