Build log — bugs & fixes

A chronological knowledge base of 51 real issues hit while building the VoxGraph Python backend. Not the marketing site — the voice server, WebSocket pipeline, STT/LLM/TTS, and memory layer.

Use this when something feels familiar: search this page (Ctrl+F) or jump to a category below. Each entry has symptom → cause → fix → files.

TTS — ElevenLabs & Deepgram

ElevenLabs free-tier policy violation (WebSocket 1008)

Symptom: STT and Gemini succeed; TTS fails with `1008 (policy violation) Unusual activity detected. Free Tier usage disabled`.
Root cause: ElevenLabs blocked the account server-side (anti-abuse). Not an app bug — often VPN, shared IP, or usage patterns. Rotating API keys does not help.
Fix: Switch to Deepgram Aura TTS (`TTS_PROVIDER=deepgram`). Optionally appeal via ElevenLabs support. Added multi-provider TTS with `auto` fallback.
Files: voxgraph/providers/tts.py
voxgraph/core/settings.py
.env

New ElevenLabs key seemed ignored (“cached .env”)

Symptom: Changed API key in `.env` but same ElevenLabs error after save.
Root cause: `.env` is read once at process start. Old uvicorn process still running. `load_dotenv()` without explicit path could miss repo-root `.env`. Shell env vars can override file values.
Fix: Pin env file to repo root (`voxgraph/core/paths.py` + `settings.py`), use `load_dotenv(..., override=True)`, print masked key at startup, restart uvicorn, clear `Env:ELEVENLABS_API_KEY` in PowerShell if set.
Files: voxgraph/core/settings.py
voxgraph/core/paths.py

No TTS fallback when ElevenLabs unavailable

Symptom: Pipeline dead-ended whenever ElevenLabs failed.
Root cause: Only ElevenLabs WebSocket TTS was wired.
Fix: Added `resolve_tts_provider()` and Deepgram Aura HTTP streaming. Env: `TTS_PROVIDER=auto|elevenlabs|deepgram|none`.
Files: voxgraph/providers/tts.py

Deepgram TTS disconnect — silent failure in mic client

Symptom: LLM returns text; user hears nothing. Server may show `Server disconnected without sending a response`.
Root cause: Intermittent Deepgram TTS API errors; no retries or client notification.
Fix: Added `TtsPlaybackError`, 3-attempt retry with fresh HTTP connection, JSON events (`tts_failed`, `speaking`, `ready`) via `transport/events.py`.
Files: voxgraph/providers/tts.py
voxgraph/transport/events.py
scripts/live_mic_client.py

TTS bytes logged but client hears nothing

Symptom: Server logs `[tts] sent N bytes`; mic client shows no audio received.
Root cause: Concurrent WebSocket sends from LLM/TTS background task while main loop blocked in `receive_bytes()` — outbound frames dropped or stalled.
Fix: Introduced `WsOutbound` — single sender task serializing all `send_bytes()` / `send_text()` through an asyncio queue.
Files: voxgraph/transport/outbound.py
voxgraph/api/audio.py
voxgraph/providers/tts.py

Word-by-word TTS (robotic speech)

Symptom: Logs show many tiny TTS calls: `'It's going'`, `'well,'`, `'thanks!'` — choppy voice.
Root cause: Over-aggressive latency tuning: `TTS_MIN_PHRASE_CHARS` too low (~6–8), debounce 50ms, endpointing 120ms.
Fix: Reverted to sentence/clause batching (~12+ chars), debounce `0.12s`, endpointing `180ms` in demo mode.
Files: voxgraph/providers/tts.py
voxgraph/core/settings.py
.env

Long delay before first spoken word

Symptom: Noticeable lag between LLM start and first audio.
Root cause: TTS buffered until full sentences; new HTTP client per phrase.
Fix: Phrase/clause streaming, shared `httpx.AsyncClient`, pipelined synthesis, `TTS_MIN_PHRASE_CHARS`, connection keepalive.
Files: voxgraph/providers/tts.py

ElevenLabs return type — no “audio sent” signal

Symptom: Hard to detect TTS failure vs success in pipeline.
Root cause: Stream function returned only text string.
Fix: Return tuple `(text, saw_audio: bool)` for error handling paths.
Files: voxgraph/pipeline/elevenlabs.py
voxgraph/providers/tts.py

Environment & configuration

Shell env overrides `.env` (STT_ONLY mismatch)

Symptom: Server behaves like `STT_ONLY=1` while `.env` says `0`.
Root cause: PowerShell/session env vars override unless `load_dotenv(override=True)` with explicit file path.
Fix: Pinned `.env` path + `override=True` + startup log showing effective config.
Files: voxgraph/core/settings.py

#10

Editing `.env` while server running has no effect

Symptom: Changed keys in `.env`; behavior unchanged.
Root cause: Environment loaded once at import/startup.
Fix: Stop uvicorn (`Ctrl+C`) and restart. For hosted: use `/config/apply` or redeploy.

#11

Gemini `gemini-flash-latest` → 429 quota errors

Symptom: Long pauses; HTTP 429 from Gemini; ~7s retry backoff.
Root cause: Unstable model alias hitting free-tier limits.
Fix: Default to `gemini-2.5-flash`; document Ollama as unlimited local alternative.
Files: voxgraph/providers/llm.py
.env.example

#12

Gemini free-tier exhausted during testing

Symptom: Quota errors after heavy dev/testing.
Root cause: Google AI free-tier rate limits.
Fix: Added `LLM_PROVIDER=ollama` for local unlimited dev.
Files: voxgraph/providers/llm.py
.env.example

#13

Setup wizard ignored server `.env` keys

Symptom: Online/local setup forced re-entering keys already on server.
Root cause: `/config/test` only validated submitted form keys.
Fix: `/config/test` falls back to server env when form fields empty; local mode skips wizard.
Files: voxgraph/api/config.py
voxgraph/core/runtime_config.py

#14

`/config/apply` did not refresh running pipeline

Symptom: Applied keys via HTTP but STT/LLM still used old values.
Root cause: Module-level globals not reloaded after `os.environ` update.
Fix: `settings.reload()` + `reload_voxgraph_globals()` after apply.
Files: voxgraph/core/settings.py
voxgraph/core/runtime_config.py

WebSocket, STT & utterance handling

#15

Turn-based mic — Enter required every turn

Symptom: Not conversational; had to press Enter after each reply.
Root cause: `live_mic_client.py` was half-duplex / turn-based.
Fix: Continuous streaming mic + background TTS playback + barge-in + mic ducking during AI speech.
Files: scripts/live_mic_client.py

#16

Continuous mode: STT fires but LLM never runs

Symptom: Transcripts appear; no AI reply until WebSocket disconnects.
Root cause: Debounce waited for silence since last **audio byte**. Continuous mic never stops → timer never completes.
Fix: Debounce on **`last_utterance_at`** (time since last `speech_final` from Deepgram), not last PCM chunk.
Files: voxgraph/voice/utterance.py
voxgraph/api/audio.py

#17

Giant merged prompt only on disconnect

Symptom: All utterances merged into one prompt when client closes.
Root cause: Same debounce bug as #16 — pipeline only flushed on disconnect.
Fix: Fixed utterance debounce timing (#16).
Files: voxgraph/voice/utterance.py

#18

Echo stacking in utterance merge

Symptom: `Debounced utterance (merged): Hello. Hello? Hello, VoxGraph.` + cancelling previous response.
Root cause: Speaker echo → multiple STT segments concatenated in one debounce window.
Fix: Stage **best single utterance** per window instead of concatenating all fragments. Use headphones.
Files: voxgraph/voice/utterance.py

#19

STT active during AI playback (spurious turns)

Symptom: Empty or `Hello?` transcripts while assistant is speaking.
Root cause: Deepgram processed mic audio including speaker echo during TTS.
Fix: Gate STT when `_ai_is_responding()` unless `_is_likely_barge_in()` (≥3 words or ≥12 chars).
Files: voxgraph/voice/utterance.py
voxgraph/api/audio.py

#20

Endpointing too aggressive — split questions

Symptom: One question triggers multiple LLM calls (“can you tell what” + “sport I wanted”).
Root cause: `DEEPGRAM_ENDPOINTING_MS=100` in aggressive demo tuning.
Fix: Raised to `180ms`; interim transcript stitching in merge logic.
Files: voxgraph/voice/utterance.py
voxgraph/core/settings.py
.env

#21

Mic client ignored server JSON events

Symptom: No `Thinking…` / `Speaking…` / error lines in terminal.
Root cause: Client loop did not parse text JSON frames from server.
Fix: Added `parse_client_event()` / `print_client_event()` in transport layer; wired in mic scripts.
Files: voxgraph/transport/events.py
scripts/live_mic_client.py
scripts/send_test_pcm.py

#22

TTS on closed WebSocket (log noise / errors)

Symptom: Tracebacks sending audio after client disconnected.
Root cause: Pipeline continued after WebSocket close.
Fix: `notify_client()` catches disconnect; guarded sends via `WsOutbound`.
Files: voxgraph/transport/events.py
voxgraph/transport/outbound.py

#23

`live_mic_client.py` NameError: REPO_ROOT

Symptom: `NameError: REPO_ROOT` before import from `send_test_pcm`.
Root cause: Used `REPO_ROOT` before module import defined it.
Fix: Define `_REPO_ROOT = Path(__file__).resolve().parent.parent` first, then `sys.path.insert`.
Files: scripts/live_mic_client.py

#24

Scripts can't import voxgraph modules

Symptom: ImportError running mic scripts from wrong directory.
Root cause: Repo root not on `sys.path`.
Fix: Bootstrap `sys.path` with repo root in client scripts.
Files: scripts/live_mic_client.py
scripts/send_test_pcm.py

LLM — Gemini & Ollama

#25

“What is YOUR name?” → answered with user’s name

Symptom: Repeated “What is your name?” → keeps saying “Raj” (user’s name). Loop with small Ollama models.
Root cause: No pronoun disambiguation; 1B models confuse “your” vs “my”; every phrase hit full LLM.
Fix: Added `try_direct_voice_reply()` in `voice/intents.py` for name, presence, greeting intents.
Files: voxgraph/voice/intents.py
voxgraph/providers/llm.py

#26

Ollama got flat prompt blob — no chat history

Symptom: Could not remember 2 turns ago; incoherent multi-turn sessions.
Root cause: Ollama received one concatenated string instead of `messages[]` with prior turns.
Fix: Proper multi-turn `messages[]`: system → history → current user in `providers/llm.py`.
Files: voxgraph/providers/llm.py
voxgraph/memory/store.py

#27

DEMO_MODE capped replies at 8 words

Symptom: Ultra-brief non-sequitur answers even when memory loaded.
Root cause: Prompt: “ONE very short sentence (max 8 words)”.
Fix: Replaced with `VOICE_MAX_WORDS` (default ~45); demo mode no longer hard-caps at 8.
Files: voxgraph/providers/llm.py
.env

#28

Ollama silent context truncation

Symptom: Long sessions lost early context.
Root cause: Default `num_ctx` ~2048 not overridden.
Fix: Set `OLLAMA_NUM_CTX` (4096 demo / 8192 normal), increased `OLLAMA_NUM_PREDICT`.
Files: voxgraph/providers/llm.py
.env

#29

Slow Ollama first reply / cold start

Symptom: High latency on first turn; new HTTP connection every request.
Root cause: Model load on first call; no connection reuse.
Fix: Shared HTTP client, `keep_alive: 30m`, `warm_ollama()` on startup and WebSocket connect.
Files: voxgraph/providers/llm.py
voxgraph/api/app.py

#30

Bad fact extraction: “I am done” → name is done

Symptom: Nonsense semantic facts in memory.
Root cause: Over-broad regex matching “I am …” as name patterns.
Fix: Tighter rules in `extract_facts_from_transcript()`.
Files: voxgraph/memory/store.py

#31

Simple greetings always hit full LLM

Symptom: Unnecessary latency on “hey there”.
Root cause: No intent shortcuts beyond name queries.
Fix: Extended `voice/intents.py` for common greetings and presence checks.
Files: voxgraph/voice/intents.py

Memory (SQLite)

#32

Semantic memory not updating for default user

Symptom: Memory “not updating”; test data under wrong user id.
Root cause: DB had rows for `u1`; app uses `DEFAULT_USER_ID = "default"`.
Fix: Rewrote store with proper keyed semantic facts for `default` user.
Files: voxgraph/memory/store.py

#33

Episodic memory was one truncated 500-char blob

Symptom: Could not recall earlier turns; useless LLM context.
Root cause: `new_summary[-500:]` string append instead of structured turns.
Fix: `episodic_turns` table, bounded turn/char limits via env vars.
Files: voxgraph/memory/store.py
voxgraph/memory/service.py

#34

“My name is Raj” not stored as semantic fact

Symptom: “What is my name?” failed after user stated name.
Root cause: Only episodic summary updated; no structured extraction.
Fix: Regex extraction + `upsert_fact()` for names, preferences, etc.
Files: voxgraph/memory/store.py
voxgraph/memory/service.py

#35

Fire-and-forget memory tasks lost on shutdown

Symptom: Facts/turns sometimes not persisted.
Root cause: `asyncio.create_task(update_memory(...))` without lifecycle management.
Fix: Background save after reply completes; cache during debounce window.
Files: voxgraph/memory/service.py
voxgraph/pipeline/response.py

#36

LLM ignored semantic facts in prompt

Symptom: Model contradicted stored facts.
Root cause: Facts appended weakly in prompt.
Fix: Stronger system prompt treating facts as ground truth; inject episodic chat history.
Files: voxgraph/providers/llm.py

#37

Synchronous DB writes delayed “ready” state

Symptom: Lag after each reply before next turn.
Root cause: SQLite writes in hot path blocked pipeline.
Fix: Background memory writes; snapshot cached at utterance time.
Files: voxgraph/memory/service.py
voxgraph/pipeline/response.py

Latency tuning (fixes & regressions)

#38

200ms utterance debounce felt sluggish

Symptom: Fixed wait after user stops speaking.
Root cause: Conservative debounce for non-demo mode.
Fix: `DEMO_MODE=1` → `0.12s` debounce; tunable via `UTTERANCE_DEBOUNCE_SEC`.
Files: voxgraph/core/settings.py
.env

#39

Stacked latency optimizations backfired

Symptom: After aggressive tuning, experience felt worse (see #6 word-by-word TTS).
Root cause: 50ms debounce + micro-TTS + 8192 ctx + 30 turns + blocking DB combined.
Fix: Partial revert: natural phrase TTS, balanced memory window, demo-specific limits.
Files: voxgraph/providers/tts.py
voxgraph/providers/llm.py
voxgraph/memory/store.py

HTTP, CORS & config API

#40

No streaming tokens to browser client

Symptom: Full reply appeared only after TTS finished in `/try` demo.
Root cause: Backend only sent final text + PCM; no incremental events.
Fix: Stream `reply_start`, `reply_delta`, `reply_done` JSON events via `WsOutbound`.
Files: voxgraph/pipeline/llm_stream.py
voxgraph/transport/outbound.py

#41

Duplicate assistant messages in web UI

Symptom: Assistant bubble printed twice.
Root cause: Both streaming deltas and final `reply_text` event treated as new messages.
Fix: Frontend dedupes; backend keeps `reply_text` for non-streaming clients.
Files: voxgraph/pipeline/llm_stream.py
voxgraph/transport/events.py

#42

CORS blocked browser `/try` from Vercel origin

Symptom: Browser demo or marketing site cannot call `/config/*` or connect from different origin.
Root cause: No CORS middleware for HTTP routes; cross-origin WebSocket from Next.js.
Fix: `CORSMiddleware` + `ALLOWED_ORIGINS` env (comma-separated Vercel domains).
Files: voxgraph/api/app.py
voxgraph/core/settings.py

#43

No HTTP API for hosted setup wizard

Symptom: Demo UI could not test or apply API keys on Render.
Root cause: Missing REST endpoints.
Fix: Added `/config/health`, `/config/test`, `/config/apply`.
Files: voxgraph/api/config.py

Deployment, Docker & repo

#44

Git merge conflicts on repo split push

Symptom: `git pull --allow-unrelated-histories` → add/add conflicts on README, requirements, scripts.
Root cause: Local restructured backend-only layout vs old GitHub monolith (Python + web/).
Fix: Resolved keeping local restructured versions; merge commit + push.
Files: .gitignore
README.md
requirements.txt
scripts/

#45

Render deploy missing `/try` and CORS

Symptom: Hosted backend had no demo UI; cross-origin blocked.
Root cause: No static file serving or CORS for split frontend architecture.
Fix: Dockerfile multi-stage build (demo → static/), CORS origins, redirect `/` → `/try/`.
Files: Dockerfile
render.yaml
voxgraph/api/app.py

#46

`static/` gitignored — clone has no demo UI

Symptom: Fresh clone → `/try` 404 until demo built.
Root cause: `static/` intentionally gitignored; built artifacts not in repo.
Fix: Document `cd demo && npm run build`; Docker builds automatically on Render.
Files: demo/
Dockerfile
DEPLOY.md

#47

Dockerfile copied demo static from wrong path

Symptom: Docker build succeeds but `/try` empty or 404 in container.
Root cause: `COPY --from=demo /app/static` — Vite `outDir` is `/static` (parent of `/demo` workdir).
Fix: Changed to `COPY --from=demo /static ./static`.
Files: Dockerfile

#48

Ollama on Render — LLM never responds

Symptom: Deployed backend accepts WebSocket but LLM times out or errors.
Root cause: Ollama runs locally; Render container has no Ollama daemon.
Fix: On Render set `LLM_PROVIDER=gemini` + `GOOGLE_API_KEY`. Keep Ollama for local dev only.
Files: DEPLOY.md
render.yaml
.env.example

Architecture refactor

#49

~1000-line monolithic `voxgraph.py`

Symptom: Hard to navigate; WebSocket + STT + LLM + TTS + memory tangled in one file.
Root cause: Organic growth during rapid prototyping.
Fix: Split into package: `api/`, `core/`, `providers/`, `pipeline/`, `voice/`, `memory/`, `transport/`. Entry point unchanged: `uvicorn voxgraph:app`.
Files: voxgraph/__init__.py
voxgraph/api/
voxgraph/pipeline/
...

#50

Circular import after package split

Symptom: `ImportError: cannot import name 'PROJECT_ROOT' from partially initialized module 'voxgraph.core.settings'`.
Root cause: `settings.py` imported `providers/tts.py` which imported `PROJECT_ROOT` from `settings`.
Fix: Moved `PROJECT_ROOT` to `voxgraph/core/paths.py` (no internal voxgraph imports).
Files: voxgraph/core/paths.py
voxgraph/core/settings.py

#51

Docs referenced old flat filenames

Symptom: Architecture docs listed `voxgraph.py`, `llm_providers.py` after refactor.
Root cause: Docs site not updated when modules moved.
Fix: Updated docs to `voxgraph/api/audio.py`, `providers/llm.py`, etc.
Files: voxgraph-web/app/docs/architecture/page.tsx

Operational notes

Not code bugs — recurring gotchas worth remembering:

Restart uvicorn after any `.env` or Python code change.
Use headphones with the mic client to reduce echo (#18, #19).
Run `ollama serve` when `LLM_PROVIDER=ollama`.
ElevenLabs account blocks persist across API key rotation — switch provider instead.
Free Render tier cold-starts can take 30–60s before WebSocket connects.
Set `ALLOWED_ORIGINS` on Render to your exact Vercel URL (https, no trailing slash).

Related docs

Architecture — how the pipeline fits together today
Configuration — env vars referenced in many fixes
WebSocket API — protocol and JSON events
Customize — where to extend after bugs are fixed