Original Summary

Hey everyone, Building real-time Voice AI applications with LLMs usually runs into a massive latency bottleneck—buffering raw audio chunks over HTTP leads to delays of 3–5 seconds, which ruins natural conversation flow. To solve this, I built a lightweight Python backend using FastAPI and WebSockets designed specifically for real-time audio streaming. ### 🏗️ Core Architecture Breakdown Bi-directional WebSockets (FastAPI): Instead of standard REST POST requests, raw audio frames (PCM/mulaw) are streamed continuously over a single persistent WebSocket connection from Twilio or web clients. In-Memory Audio Buffer & Caching (Redis): Incoming audio bytes are temporarily queued in Redis. As soon as silence/end-of-turn is detected, the audio stream is instantly dispatched to OpenAI / Deepgram. Persistent Conversation Memory (PostgreSQL): User conversation transcripts, caller metadata, and turn histories are saved asynchronously to PostgreSQL using SQLAlchemy async sessions without blocking the WebSocket loop. Docker Containerization: Everything (FastAPI worker, Redis, Postgres) is unified in a docker-compose.yml for single-command deployment. --- ### 💡 Key Technical Takeaways Keep the main event loop clean: Heavy database writes or audio processing should never run synchronously inside the WebSocket consumer loop. Chunk size matters: Streaming smaller 20ms audio chunks over WebSockets gives instantaneous response times compared to batch processing full audio files. --- I’ve open-sourced the basic architecture and built a complete production-ready Voice AI Backend Starter Kit complete with Docker setup, environment configs, and pre-built memory pipelines. Check out the repository & starter kit here: 🔗 GitHub / Gumroad: https://abishnavi.gumroad.com/l/voice-ai-backend Happy to answer any questions about WebSocket state management or handling audio streams in Python!   submitted by   /u/Abishnavi [link]   [comments]

  • 情报分类：技术学习与提效
  • 分类依据：内容涉及技术、AI、软件工具或工程实践
  • 信息来源：Reddit · SideProject
  • 发布时间：2026/10/15 12:24:36