Sub-100ms Speech Processing: Integrating Deepgram Nova-3 with ElevenLabs and Retell

Deepgram’s own benchmark page is specific about what Nova-3 actually delivers: transcripts in under 300 milliseconds, a claimed 54.2% word error rate reduction against the nearest competitor on Deepgram’s own internal test suite, at $4.30 per 1,000 minutes. That’s the number worth anchoring on. Sub-100ms speech processing gets thrown around in voice AI marketing constantly, and no credible STT vendor, Deepgram included, publishes a sub-100ms latency spec for the transcription step alone. What Deepgram actually promises, and delivers according to its own published figures, is sub-300ms.

That distinction matters for a genuine deepgram nova 3 integration, because the sub-100ms feeling a well-built voice agent can achieve doesn’t come from any single component hitting 100ms. It comes from architecting STT, the LLM, and TTS to overlap instead of running in sequence, which is a WebSockets voice stack problem, not a model selection problem. This article covers what Nova-3 actually delivers, where independent testing complicates the vendor’s own numbers, and how STT latency optimization in a real pipeline gets you closer to a sub-second voice agent than picking the fastest-sounding model alone ever will.

What Deepgram’s Own Numbers Actually Say

Deepgram is explicit that Nova-3 targets sub-300ms latency as the threshold for real-time voice applications, not sub-100ms. The company’s own internal benchmark suite reports a 6.84% median streaming word error rate, a claimed 54.3% reduction against competitors on that specific test set, alongside a 5.26% batch WER for non-real-time transcription. Pricing sits at roughly $4.30 per 1,000 minutes for standard streaming use.

These are vendor-reported figures from Deepgram’s own benchmark methodology, worth using as a baseline rather than an independently verified ceiling. Deepgram’s documentation is reasonably transparent about this, presenting the numbers as results from its own test suite rather than a neutral, multi-vendor comparison run by a third party.

Where Independent Testing Tells a More Complicated Story

Independent benchmarks measuring Nova-3 under different methodologies report meaningfully different numbers, and the gap is worth understanding rather than ignoring. A Coval benchmark testing five STT models across 2,400 runs each found Nova-3 posting a median time-to-first-token around 992 milliseconds in that specific test setup, alongside a word error rate in the 25% range on the same difficult dataset, notably higher than Deepgram’s own reported figures.

That gap isn’t necessarily a contradiction. Deepgram’s own sub-300ms figure likely reflects processing latency under favorable, controlled conditions, while an independent benchmark measuring full request-to-response time across a real network path, with a harder audio dataset, will produce a higher number almost by construction. The lesson for anyone planning a deepgram nova 3 integration is the same one that applies to any vendor latency claim: the published number describes a specific measurement methodology, and your production numbers depend on your specific network path, audio conditions, and integration architecture, not just which model you picked.

STT Latency Optimization: What Actually Moves the Number

Choosing Nova-3 is the easy part of the decision. Getting close to its published latency figures in a real deployment requires the same integration discipline any low-latency system needs.

• Streaming partial results as audio arrives, rather than waiting for a complete utterance, so the downstream LLM can begin reasoning before the caller has finished speaking
• Keeping the WebSocket connection to Deepgram warm and persistent across a call rather than establishing a new connection per utterance, since connection setup overhead is a fixed cost that compounds across a long conversation
• Choosing a geographically appropriate API region relative to where the telephony traffic actually originates, since network round-trip time is a real, controllable component of the number that shows up in production and not in a vendor spec sheet
• Tuning endpointing sensitivity, how confidently the system decides a caller has finished speaking, since overly conservative endpointing adds real latency that has nothing to do with the STT model’s raw processing speed
• Testing against your actual audio conditions, phone line quality, background noise, accents in your specific caller base, rather than trusting a benchmark run on a different dataset entirely

None of these show up in Deepgram’s own headline latency figure, and all of them meaningfully affect what a real caller actually experiences.

WebSockets Voice Stack: Why Architecture Beats Model Choice

A sub-second voice agent isn’t the product of any single fast component. It’s the product of an architecture where STT, LLM reasoning, and TTS generation overlap through streaming rather than running as three sequential steps that each wait for the previous one to fully complete. Deepgram’s own architecture guidance makes this point directly: a well-designed voice agent stack starts generating a response before transcription is fully final, and starts synthesizing speech before the full response text is complete, chaining partial results through a WebSocket pipeline instead of waiting at each stage.

That’s the actual mechanism behind a genuinely responsive voice agent, and it’s an integration decision, not a model selection decision. A team that picks Nova-3 for its latency spec and then builds a request-response pipeline that waits for each stage to fully complete before starting the next one will never see anything close to the published number, regardless of which STT model sits at the front of the pipeline.

What a Genuine Deepgram Nova 3 Integration Actually Requires

• A realistic benchmark run against your own audio conditions and network path before committing to a latency budget based on the vendor spec sheet alone
• A persistent, warm WebSocket connection architecture rather than per-utterance connection overhead
• Streaming integration across the full pipeline, STT into LLM into TTS, with each stage starting on partial output rather than waiting for the prior stage to fully finish
• Endpointing tuned specifically for your use case, since the default sensitivity is rarely the right tradeoff for every conversation type
• Honest measurement of end-to-end latency as callers actually experience it, not just the STT component in isolation

Why This Matters More for Some Use Cases Than Others

Not every voice application needs to chase a sub-second voice agent target. A deepgram nova 3 integration for an outbound survey or a low-stakes appointment reminder can tolerate more latency than a live customer support call where every extra pause reads as the system not understanding the caller. The same discipline that applies to production-grade voice call workflows generally applies here: the engineering investment in shaving milliseconds should scale with how much a slow response actually costs the specific use case, not be applied uniformly because a benchmark number looked impressive.

The Model Is Necessary. It’s Not Sufficient.

Deepgram’s own published figures make Nova-3 a genuinely strong foundation for a low-latency voice agent, and the sub-300ms target it’s built around is real, achievable, and well-documented. What that figure doesn’t include is everything else that determines whether a caller actually experiences a fast conversation: the WebSockets voice stack around it, the endpointing tuning, the network path, and whether the pipeline was built to overlap stages or run them in sequence. A deepgram nova 3 integration that stops at model selection is doing roughly half the work a genuinely sub-second voice agent requires. The other half is exactly what this article covers, and it’s the part most vendor documentation skips entirely.

Frequently asked questions

Does Deepgram Nova-3 actually achieve sub-100ms speech processing?

No, based on Deepgram’s own published specification, which targets sub-300ms latency, not sub-100ms. Independent benchmarks under different test conditions have measured latency figures higher than Deepgram’s own reported number in some cases. A sub-100ms feeling in a full voice agent interaction comes from pipeline architecture, overlapping STT, LLM, and TTS through streaming, not from any single component’s raw processing speed.

Why do independent Nova-3 benchmarks report different numbers than Deepgram’s own figures?

Different benchmarks measure different things: Deepgram’s own figures likely reflect processing latency under controlled conditions, while independent tests measuring full request-to-response time over a real network path, on harder audio datasets, will naturally produce different numbers. Neither figure is necessarily wrong; they’re measuring different parts of the pipeline under different conditions.

What actually determines whether a voice agent feels like a sub-second voice agent to a real caller?

The architecture connecting STT, LLM, and TTS matters more than any single component’s spec sheet number. A pipeline that streams partial results and lets each stage begin processing before the prior stage fully completes feels dramatically faster than one that waits for each step to finish sequentially, even when both use the identical underlying models.

What is the most impactful STT latency optimization most teams skip?

Keeping the WebSocket connection to the STT provider warm and persistent across a call, rather than establishing a new connection per utterance. Connection setup overhead is a fixed, avoidable cost that compounds noticeably across a longer conversation and rarely shows up in a vendor’s published latency figure.

Is Deepgram Nova-3 the right choice for every voice agent use case?

It’s a strong choice specifically when latency is the primary constraint and the audio conditions are reasonably close to what Deepgram’s own benchmarks reflect. Independent testing suggests accuracy can vary meaningfully by audio condition and dataset, so testing against your own actual call audio before committing to a deepgram nova 3 integration in production is worth the time regardless of which vendor’s spec sheet looks best on paper.

Bhavesh Modi

Project Manager – AI