The streaming voice loop - Whisper → Claude → Piper - tool use, and wake mode.
Date: 2026-06-06 Commits: cba1530, 1bd5db5, 0c97c28, 84f8df8, e914b19 Theme: Stand up a complete, tested voice loop end-to-end - then teach it to use tools and to only respond when addressed.
Starting from an empty repo (just a structure sketch in excar_project_structure.txt), this session produced a working, tested system that runs on a laptop today.
cba1530)What. The core loop: microphone → STT (Whisper) → LLM (Claude, streamed) → sentence chunker → TTS (Piper) → speaker, plus the interfaces, typed config, factory, CLI, logging and tests around it.
Why. A voice assistant's single most important feature is that it feels instant. The naive design - transcribe, send the whole prompt, wait for the whole reply, synthesise it, then play it - stacks every latency in series and feels dead. We needed an architecture where time-to-first-word is short and independent of reply length.
How. The pipeline overlaps stages. As Claude streams its reply, a SentenceChunker (core/text_chunker.py) accumulates deltas and emits complete sentences as soon as they form. Each finished sentence is pushed onto a queue.Queue that a background _speak_worker thread drains - synthesising and playing sentence *N* while the LLM is still generating sentence *N+1* (core/pipeline.py). The result: EXCAR starts talking after the first sentence, not the whole answer.
Audio capture uses energy-based VAD endpointing (audio/recorder.py): speech is detected above vad_start_rms, and a turn ends after silence_hangover_ms of trailing silence. The whole thing is wired together by factory.build_pipeline, which reads the validated ExcarConfig and returns a ready Pipeline.
Purpose / goal. Establish the "walking skeleton" - every layer present, thin but real, swappable behind interfaces - so future work is *filling in* rather than *rearchitecting*. This is the spine the entire product hangs off.
Diagnostics included. app.py exposes doctor / listen / say / chat so each layer can be validated in isolation:
excar doctor - checklist: audio, STT, API key, Piper voice.excar listen - record once, print the transcript (STT only).excar say "..." - synthesise and play a line (TTS only).excar chat - text-mode conversation (LLM only - the cheapest way to validatethe persona and streaming feel; no mic or voice file needed).
1bd5db5)What. An agent loop inside ClaudeEngine so the model can call tools, with three tools to start: open_maps, get_datetime, get_current_location. Also a hands-free activation mode.
Why. A co-pilot that can only talk is a toy. It needs to *do* things - open navigation, tell you the time, know where you are. And in a car you can't press a button to talk; it has to be hands-free.
How. Tools are Tool subclasses (services/tools.py) with a name, description, and JSON input_schema, collected in a ToolRegistry. The Claude engine runs the standard agent loop: stream → if the model emits a tool_use block, run the tool, feed the tool_result back, continue - until a final text reply. factory.build_tools is the single registration point. Tools live in services/ so adding a capability never touches the pipeline.
Purpose / goal. Turn EXCAR from a chatbot into an *agent* that acts on the car/driver's behalf, with a clean extension seam for every future capability (music, calendar, vehicle systems).
0c97c28)What. Reply language is now driven by personality.reply_language in config (English for laptop dev, Turkish for the car, null for auto).
Why / the bug. EXCAR was intermittently replying in Turkish on the *very first* turn even in English dev mode. Root cause: with no language instruction, the model inferred language from the persona/context and defaulted inconsistently on the first turn before any conversational signal existed. Fix: load_system_prompt (factory.py) appends an explicit instruction - *"always reply in {language}, including your very first reply, unless the driver clearly speaks a different language - then match them."* Deterministic from turn one, still adaptive if the driver code-switches.
Purpose / goal. The car ships Turkish; development happens in English. Language had to be a config knob, not a code change, and the first impression had to be correct.
84f8df8)What. A wake activation mode (now the default): EXCAR transcribes everything it hears but only runs the LLM and speaks when *addressed by name* ("EXCAR"). It opens a follow-up window so you don't repeat the name on every turn, and a "stand-down" phrase silences it instantly.
Why. In continuous mode EXCAR answered *every* utterance - including when you turned to talk to a passenger. That's the fastest way to make an in-car assistant unbearable. It needed to know when it's being talked *to* versus talked *near*.
How. A WakeController (core/addressing.py) is the gate. In _run_wake (pipeline.py) every utterance is cheaply transcribed, then on_utterance(text) decides: if a wake word is present, engage and open a follow_up_seconds window (default 15s); while the window is open, keep responding without the name; a keyword stand-down ("don't interrupt") drops to standby immediately and silently. Transcription is cheap; LLM calls and speech are gated.
Purpose / goal. Social awareness - the difference between a co-pilot and an eavesdropper. This is the behaviour that makes EXCAR liveable in a shared cabin.
Known limitation noted at the time. The follow-up window still responds to *any* speech while open, and stand-down is keyword-based (robust for common phrasings, not exhaustive). Both were later strengthened by diarization and the intent gate (Report 06).
e914b19)What. The first docs/STATUS.md - a living snapshot of capabilities, run instructions, config knobs, and known limitations.
Why. Even at one day old, the project had collaborators who needed a single source of truth that wasn't "read all the code". STATUS.md is the *current state*; this sessions/ journal (added later) is the *history and reasoning*.
Fixes shipped this session: Python 3.10 support (originally assumed 3.11+); config-driven STT language; the Turkish-first-reply fix above; location *spoken* instead of wrongly opening a map.