LLM, RAG & Conversational AI
Real-Time Conversational Voice Agent
HawkEyes · Voice-AI-Agent · most active repo · 2026
Browser-based, low-latency spoken AI — WebRTC audio into a streaming STT → agent → TTS loop with tool-calling.
- 122
- commits · 6 branches
- <latency
- parallel TTS + GPU-batched STT
- 4+
- LLM providers
Problem
Enable natural, low-latency spoken conversation with an LLM assistant directly from the browser — the foundation for production AI agent-driven inbound/outbound telephony (hospital appointment automation, sales, and support agents).
Approach
- 01WebRTC-first (WebSocket fallback) audio streaming into a VAD → STT → LangGraph agent → TTS loop.
- 02Per-session checkpointed memory and MCP tool-calling (Twilio, email, dates) so the agent can take real actions.
- 03Multi-provider LLM backends (Ollama / OpenAI / Gemini / Hugging Face) behind one interface.
- 04Explicit latency engineering: parallel TTS, GPU-batched STT, and concurrent LLM+TTS to cut perceived response time.
Results & Impact
- The single most active repository in the portfolio: 122 commits across 6 branches (rakib +88 ElevenLabs STT; dev +82 TTS providers; WebRTC fixes).
- Latency work (parallel TTS, GPU-batched STT, concurrent LLM+TTS) targets real-time telephony use.
- Bangla speech to be supplied by the in-house custom Bangladeshi-accent TTS (see the TTS case study).
Related repositories
Production real-time conversational voice agent (WebRTC → STT → LangGraph → TTS).
Hospital appointment-booking voice & chat agent — LangGraph slot-memory core with a Gemini Live audio bridge, Twilio/email tool-calling, and WebRTC transport.
Local, GPU-accelerated voice assistant loop (whisper → local LLM → TTS).
Bangla text-to-speech fine-tuning (Coqui VITS + grapheme frontend, expanding toward XTTSv2).