TL;DR — On real-time streaming voice, ElevenLabs is the bar. Sub-second latency, near-natural prosody, actual real-time. AgentZero Lite today doesn't chase that — we VAD-batch utterances before sending to STT and accept the latency hit, which lets us treat voice behind a clean byte[] PCM → string interface. But to feel like the assistant we want — and to do it without a Realtime API in the loop — we need to climb. Adopting ElevenLabs Realtime or OpenAI Realtime is genuinely cheap on the implementation side: they already ship the streaming protocol, sentence-boundary handling, and interrupt semantics. The mountain only exists when you stitch a local open STT model and a local open TTS model into something Realtime-API-like yourself, because none of the open pieces ship that orchestration layer for you. Step 1: prove that an open voice-LLM (vLLM) embeds end-to-end on consumer hardware. Step 2: an Akka.NET Streams Graph that handles mid-stream long-sentence speech, request-unit switching, and mid-stream LLM function calls for sentence-boundary detection. The goal isn't to beat ElevenLabs Realtime or OpenAI Realtime — it's to reach a minimum usable approximation of that experience on a local-only stack. Step 2 may end up harder than the multi-CLI core itself. So Realtime APIs stay plugged in throughout — I'm a paying user; this isn't a cost argument, it's an ownership-and-architecture one. The road for fully open local realtime voice is long.

This is a follow-up to Voice-Driven Multi-Terminal Control — How a Local Gemma 4 Function-Caller Pilots Claude, Codex and Anything Else with a Prompt, and indirectly a response to a sharp X critique that landed yesterday: the OpenAI-compatible interface holds for text but leaks for voice. That critique is correct, and it's the doorway to this whole conversation.

Where realtime streaming voice actually is in 2026

Before talking architecture, calibrate the bar.

ElevenLabs is the gold standard. The number of seconds of polish their team has put into the streaming protocol alone is the kind of investment a side-project can't replicate, and shouldn't pretend to.

This article is not "we're going to outdo ElevenLabs." It's "what does the open path look like, honestly, and where does it meet the premium path."

Why even attempt the open path

Three reasons, none of them about price:

1. Local control of the audio modality. The same reason we ship an on-device Gemma 4 for the function-call agent applies here — voice data is sensitive, and a captain's bridge (per the previous chapter) shouldn't have to leave the machine for every utterance.
2. Architectural ownership of the streaming protocol. Bek's critique on X named it precisely: behind a unified OpenAI-compatible HTTP endpoint, the streaming chunk semantics still differ per backend — chars vs ms boundaries, finalization shape, interim vs final signaling. If we own at least one path end-to-end, we get to shape that protocol seam intentionally instead of papering over it.
3. The captain-can-interrupt problem. Real conversational voice means the user can cut in mid-output. That requires the TTS layer to expose cancellation primitives that match how our Akka actor mesh talks. ElevenLabs' API has good interrupt support; doing it ourselves teaches us exactly how that protocol should look.

And to be explicit about my position: I'm a paying ElevenLabs user, not on a free tier. This article isn't framed by cost. It's framed by curiosity about whether the open path is good enough for the experience we want, and what the actual gap is.

Today's design — VAD utterance batching, not streaming

To set context: AgentZero Lite does not currently do streaming STT or streaming TTS. We deliberately punted that whole layer. Quick recap of what exists in main:

flowchart LR
  Mic[🎙 NAudio<br/>16 kHz mono PCM]
  Mic --> VAD[Frame VAD + Utterance VAD<br/>40 silent frames = 2 s threshold]
  VAD -->|UtteranceEnded| Drain["ConsumePcmBuffer to byte array"]
  Drain --> STT{ISpeechToText<br/>factory}
  STT --> Whisper[Whisper.net]
  STT --> OAI[OpenAI Whisper-1]
  STT --> Webnori[Webnori-Gemma audio]
  STT --> LocalG[Local Gemma audio]
  Whisper --> Txt[full transcript]
  OAI --> Txt
  Webnori --> Txt
  LocalG --> Txt
  Txt --> Bot[AgentBotActor.StartReactor]

  classDef now fill:#1e3a3a,stroke:#4EC9B0,color:#fff
  class Drain,Txt,Bot now

The abstraction is byte[] PCM → string transcript per utterance. Backend chunk semantics never escape into our consumer code, because we never operate on chunks — we operate on completed utterances. This is why our ISpeechToText factory stays clean even as we add or swap engines: the leak Bek named exists at the streaming layer, and we side-step that layer entirely.