Voz
voicev0.3.0Inline voice mode in chat: dictation and hands-free conversation with live voice spectrum and 3 selectable engines
Description
Talk with the chat without opening any modal: the composer transforms into a voice surface with a reactive spectrum, while the conversation thread remains visible. Two experiences on the same microphone — momentary dictation (voice to text for review before sending) and hands-free interactive voice mode (you speak, it sends automatically, the AI responds aloud). Three interchangeable engines from settings: 'cascaded' (default; uses the native chat flow, so it RETAINS all tools, artifacts, and KB and respects the user's Auto Tools state), 'audio-native' (speech-to-speech in a call with gpt-audio, lower latency, pure conversation without tools) and 'elevenlabs-realtime' (low-latency duplex with interruption/barge-in via Conversational AI). Supports BYOK: each user can use their own ElevenLabs key (encrypted at rest) or the system's.
Features (6)
Tap on the mic: speak and the text appears in the input for review before sending
Hold down: hands-free conversation with VAD for silence; the AI responds aloud
Reactive voice visualizer in the composer itself (Web Audio), no modals. Different color per turn
The default engine inherits the Auto Tools state: artifacts, KB, and tools remain available
cascaded · audio-native (gpt-audio, low latency) · elevenlabs-realtime (duplex with barge-in)
Each user can use their own ElevenLabs API key, encrypted with AES-256-GCM
Changelog
- Phase 3: ElevenLabs realtime engine (signed URL + duplex WebSocket, barge-in)
- BYOK per user encrypted (secret-vault AES-256-GCM): endpoints /key and /realtime/session
- Plan gate + mandatory configurable BYOK for realtime
- Settings UI in superadmin (engine/voice/agent + BYOK key management)
- Phase 2: Quality TTS via ttsService (ElevenLabs/AIML) with reactive spectrum to real audio
- Audio-native engine (openai/gpt-audio) speech-to-speech in a call
- Endpoint /voices and /audio-chat (webm→mp3 conversion with ffmpeg-static)
- Phase 1: dictation (A) end-to-end (getUserMedia → /stt Whisper → input)
- Voice mode (B) cascaded with VAD for silence + speechSynthesis reading
- Web Audio spectrum, inline mic in the composer (1-line anchor in index.ejs)