Flavio Espinoza

Browser Audio Capture | v2 — The front end is a React and Vite app in TypeScript. Its only job on the audio path is to be a clean, dumb pipe -- capture raw microphone PCM and stream it, do no processing in the browser.

1. A.An AudioWorklet (`pcm-recorder`) captures 16kHz mono int16 PCM in 100ms chunks and sends each chunk as a binary WebSocket frame -- raw audio out, no encoding or VAD in the browser.
2. B.The React UI is a full chat surface, not a toy: streamed reply bubbles, a live "karaoke" highlight that tracks the sentence currently being spoken, a one-click voice toggle, and a context-usage indicator in the footer.
3. C.Playback is event-driven -- the browser plays each mp3 the server sends, and on the audio `ended` event it posts `playback_done` back to the server, which is the signal that re-opens the mic. The loop closes itself.

Silero VAD | v2 — Voice-activity detection runs server-side so the browser stays a dumb pipe. It decides when I started talking and, more importantly, when I stopped -- the hard part of a hands-free interface.

1. A.Silero VAD (ONNX) is the primary detector, with a hand-written RMS energy detector as an automatic fallback if the ONNX runtime is unavailable -- the loop never hard-fails on a missing model.
2. B.A rolling pre-roll buffer keeps the last ~500ms of audio before speech is detected and prepends it to the utterance, so the first word is never clipped.
3. C.The endpoint silence threshold is set at 2500ms on purpose -- it gives me room to pause and think mid-sentence without being cut off, a deliberate tradeoff of latency for not getting interrupted.
4. D.Tuned by use, not by guess -- the energy fallback threshold was moved from 800 (too sensitive, ghost triggers) to 1200 (missed soft words) and settled at 900; a post-playback cooldown stops the speaker's own audio tail from re-triggering the mic.

Deepgram STT | v2 — The chosen ears of the system, after the mid-build correction away from local Parakeet. Speech-to-text runs as a streaming WebSocket to Deepgram so transcription happens live, not after I stop talking.

1. A.Streams PCM to Deepgram nova-3 with interim results on, so the UI shows a live "typing" partial while I am still speaking and commits finalized segments as they settle.
2. B.Keyterm prompting -- a `config/stt-keywords.txt` file feeds proper nouns (project names, tooling) to nova-3 so domain words transcribe correctly instead of becoming gibberish.
3. C.Hardened against race conditions -- audio that arrives before the Deepgram socket is ready is buffered and flushed on open, and a short finalize grace window lets short utterances settle before the stream closes.

The AI at the Seam | v2 — The heart of the v1-to-v2 win, and the architectural decision that makes the platform model-agnostic. The seam is deliberately narrow: a transcript goes in, text deltas come out. That contract is the same whether Claude, Gemini, or ChatGPT is sitting behind it.

1. A.v1 runs on the Claude Agent SDK -- the same engine as the Claude Code CLI, not the basic chat API. Full tool access: Bash, Read, Write, Edit, Grep, Glob. Claude can explore the repo, run commands, read real file contents instead of hallucinating them. The v1 Python middleware anti-hallucination guards became dead code the moment the SDK had a real shell.
2. B.Picks up project context automatically -- the SDK loads `CLAUDE.md` from user and project setting sources the way the CLI does, so the voice instance obeys the same rules as every other Claude in my workflow with zero custom plumbing.
3. C.Session continuity -- each turn captures the SDK session id and resumes it on the next turn, so the conversation has real memory across turns and survives a page reload.
4. D.Designed for Opus 4.8 with 1M context -- the deep model is the point, since I pay for the Max plan precisely to get it. There is a live bug where the SDK falls back to Haiku despite the request, which is why the server already instruments a `MODEL_FALLBACK` log and a `cost_alert` to the UI the moment the returned model is not the one I asked for. The fix is pending; the detector that catches it is already shipped.
5. E.The system prompt is a small piece of TTS engineering in its own right -- it teaches Claude to write text that sounds right when read aloud by Aura-2: real newlines between list items (never "One.Two.Three." mashed together), no markdown syntax, acronyms without internal periods, and the "C2 not C.2" rule that sidesteps the voice engine's decimal parser.
6. F.Phase 2 swaps the model at the seam -- Gemini for research sessions, ChatGPT for brainstorming. Same VAD, same STT, same TTS pipeline, same persistence layer. Swapping the AI is one integration change, not a rebuild. The audio platform is the durable investment; the model behind it is a choice made per task.

Streaming TTS Pipeline | v2 — The output half, and the part with the most hard-won detail. Claude's reply streams back token by token; this pipeline turns that stream into natural speech without waiting for the whole reply to finish.

1. A.Sentence-boundary streaming -- the pipeline buffers deltas, splits on sentence punctuation, and fires each finished sentence to Deepgram Aura-2 as its own synthesis call, so audio starts playing while Claude is still writing.
2. B.Code is seen, not spoken -- a stateful filter strips fenced and inline code from the spoken path (carrying fence state across chunk boundaries) while the raw text still streams to the chat bubble, so you read the code and hear the prose.
3. C.Deferred mode -- if a reply looks like a code request, or a backtick appears before speech has started, the pipeline holds all audio until the bubble has fully rendered, then reads from the top, so the voice and the visible code never drift apart.
4. D.Paragraph pacing -- a double newline becomes a 1.2-second audible pause, giving real breathing room between topics instead of one run-on wall of speech.
5. E.Interruptible and resumable -- pressing stop aborts generation, captures the partial text, and feeds it back as context on the next turn, so saying "keep going" makes Claude continue from exactly where it was cut off rather than starting over.

Persistence + Instrumentation | v2 — The plumbing that makes it a tool I trust daily rather than a demo. Every conversation is durable, searchable, and measured.

1. A.Dual-format persistence -- every turn auto-saves the chat as both machine-readable JSON and human-readable Markdown, with an auto-generated title from the first substantial message; restart the app and the conversation is still there.
2. B.A small REST surface (`/api/chats`, fetch-by-id, full-text search) over the saved chats, plus a per-connection session log that records every state transition and event to disk for debugging the real-time loop.
3. C.A live context-usage indicator -- the server estimates tokens actually in play from the running chat record and pushes it to the footer each turn, so I can see context filling up. Paired with the model-fallback cost alert, the system watches its own spend and its own model choice and tells me when either drifts.

The Loop, Restated — Control, variant, measure, ship the winner -- one change at a time. v1's Python middleware was the control; deleting it for the Agent SDK was the variant that won and became v2, the new control. Inside v2 the same loop runs at small scale: a VAD threshold moved 800 to 1200 to 900 until soft speech registered without ghost triggers; the wrong STT engine swapped out mid-build in thirty-five minutes; a TTS pipeline tuned sentence by sentence until reading code aloud stopped sounding broken. Phase 2 is the same loop at model scale: run Gemini at the seam for research, run ChatGPT for brainstorming, measure which AI is actually better for each cognitive mode when the friction of switching is zero. Build one thing that works, make it the control, test every new idea against it, and promote only what beats it. Then begin again.