Why real-time voice is genuinely hard
A text chatbot can think for two seconds and nobody notices. A voice agent cannot. The moment a person stops speaking, silence becomes awkward in under a second, so every stage — speech-to-text, the model's reasoning, any tool it calls, and text-to-speech — has to complete inside a budget most systems blow through on the first turn. Three things have to hold at once, live:
- Low latency. Transcription, model response, tool calls, and speech synthesis all share one sub-second budget. There is nowhere to hide a slow retrieval or a chatty prompt.
- Tool use. A useful voice agent doesn't just talk — it looks things up, books, checks status, and takes actions. Doing that while keeping the conversation flowing, and handling the case where a tool is slow or fails, is where most builds fall apart.
- Grounding. The agent has to answer from real, current information rather than confidently inventing it — and it has to do the retrieval fast enough that grounding doesn't cost you the latency budget.
On top of that: interruptions ("actually, wait —"), overlapping speech, background noise, and the fact that a wrong answer spoken aloud with total confidence is worse than a wrong answer on a screen. This is why so many "voice AI" projects look great in a scripted video and fall apart on a real phone call.
Proof it can be done: a live agent you can talk to
The single best filter for "who can build this" is simple: can they point you at a real-time voice agent you can actually talk to right now? Hibiscus can. There is a live real-time voice agent built by Hibiscus — not a recording, not a slide, a working agent that handles a live conversation with the latency, tool use, and grounding described above.
That distinction matters because a scripted demo hides exactly the parts that are hard: it never gets interrupted, never hits a slow tool call, never has to admit it doesn't know. A live agent can't hide any of that. If you can pick it up and have a real back-and-forth, the hard problems have already been solved.
Voice sits on the same production stack
A voice agent that works isn't a novelty bolted on the side — it's the same production AI stack Hibiscus builds for text and API surfaces, with a real-time speech layer on the front. The pieces underneath are the ones that make any serious agent reliable:
Model gateways routing tool-using agents
A gateway routes each turn to the right model, enforces policy, and falls back gracefully when a provider degrades — so the voice agent keeps talking instead of going silent. The same gateway that powers a tool-using agent over an API powers the voice one.
RAG for grounded answers
Retrieval-augmented generation gives the agent real, current facts to answer from, with retrieval quality actually measured rather than assumed. That's what keeps a voice answer grounded instead of invented — see production AI consulting: RAG and agents.
Evaluation harnesses
Output is backtested against historical ground truth, so a change that "sounds better" is proven, not vibed. For a live voice surface, that regression safety is the difference between shipping confidently and hoping.
Because voice reuses this stack, you're not paying to invent an agent from scratch — you're adding a real-time speech surface to an architecture that already works.
Real-time systems pedigree
Low-latency, real-time media is its own discipline, separate from AI. Getting audio and video to move between people in real time — with all the buffering, connection, and timing problems that entails — is exactly the muscle a real-time voice agent needs.
Blake built a WebRTC platform for the Emmys in React in five weeks — a hard real-time media system, live, on a fixed deadline, for a high-stakes event. That's the same class of problem a voice agent poses: real-time transport, tight timing budgets, and no room to fail in front of the audience. Add roughly a decade of React and TypeScript plus Rust and Node backend work, and one engineer can own both the real-time surface and the agent behind it.
Every voice interaction can be signed and auditable
A voice agent that takes actions and speaks with authority raises an obvious question: what exactly did it say, and why? Hibiscus's core positioning answers it — production AI whose every decision is signed, anchored, and independently auditable. Every voice interaction can be captured in a tamper-evident audit trail signed with post-quantum cryptography (ML-DSA; provisional patent filed), so you can review what the agent said, what it retrieved, and which tools it called — after the fact, independently, without taking anyone's word for it.
For anything customer-facing, regulated, or high-stakes, that reviewability isn't a nice-to-have. It's the difference between deploying a voice agent you can stand behind and one you're quietly nervous about.
Scope a real-time voice agent
Tell me what conversation you want to automate and what it needs to look up or do. I'll tell you honestly what's straightforward, what's hard, and what it takes to make it work live. Remote (US) or Triangle-local; founding-engineer, senior/staff/director, or contract. SBIR-eligible small business.
Email blake@hibiscus.buzz