Architecture

VoxGraph is a single FastAPI process exposing one WebSocket endpoint. All real-time work happens over that connection — binary PCM in, binary PCM + JSON status events out.

Pipeline overview

Client

WebSocket

STT

Utterance

Memory

LLM

TTS

JSON events

PCM 24 kHz

Key modules

Module	Role
`voxgraph/api/audio.py`	WebSocket handler, Deepgram STT stream, debounce, response pipeline
`voxgraph/api/config.py`	Hosted setup: `/config/health`, `/config/test`, `/config/apply`
`voxgraph/providers/llm.py`	Gemini + Ollama streaming, system prompt, chat history
`voxgraph/providers/tts.py`	Deepgram Aura TTS, phrase chunking, PCM streaming
`voxgraph/pipeline/response.py`	Orchestrates STT → memory → LLM → TTS for each utterance
`voxgraph/voice/utterance.py`	Debouncing, barge-in, pending utterance staging
`voxgraph/memory/store.py`	SQLite semantic facts + episodic turns
`voxgraph/voice/intents.py`	Fast-path replies without LLM (names, greetings)
`voxgraph/transport/outbound.py`	Serialized WebSocket sends (safe concurrent send/receive)
`scripts/live_mic_client.py`	Reference client: continuous mic + speaker playback

Concurrency model

Main loop: receive_bytes() from client → forward to Deepgram
Deepgram callbacks: append transcripts, schedule debounced LLM task
LLM/TTS runs in asyncio task; outbound audio via WsOutbound queue
Keepalive pings Deepgram during long LLM+TTS so the STT stream stays open
Barge-in cancels in-flight TTS when new speech is detected

Design goals

Hackable — small modules under voxgraph/, clear extension points
Local-first — Ollama + Deepgram without mandatory cloud LLM
Conversation-aware — episodic turns fed back as Ollama chat messages
Hostable — Docker + Render; runtime config API for online demos