In building Vivik, an execution-grade telephony AI engine, we faced a brutal constraint: the human conversational loop.

In psychoacoustics, a delay under 250 ms feels instantaneous. At 500 ms, users notice lag. Beyond 800 ms, conversations start feeling strained, and by 1.5 seconds, the illusion of real-time interaction collapses.

That creates an extremely tight latency budget for voice AI:

• Network RTT: 50–200 ms • LLM inference: 200–800 ms • TTS synthesis: 100–400 ms • ASR processing: 100–300 ms

To consistently stay under a sub-500 ms SLA, the orchestration and media layers themselves must add almost no overhead.

We initially built the entire system in Go. It worked well for concurrency and distributed orchestration, but under production-scale load, we hit an architectural wall: non-deterministic GC tail latency.

The Media Plane processes raw PCM audio in strict 20 ms frames. Even tiny scheduling delays create audible jitter, packet drift, and conversational instability.

Under a 25,000 RPS stress test:

• Go implementation → P99 latency: 1,550 ms • Rust (Tokio) implementation → P99 latency: 310 ms

The issue wasn’t average latency. It was the tail.

Even highly optimized GC pauses become catastrophic in real-time telephony. A tiny scheduler interruption under heavy throughput creates queue backpressure that cascades across live audio streams. In practice, a 1.5-second spike means the system goes silent mid-sentence.

We solved this by separating the architecture into two isolated worlds:

1. Control Plane (Go + NATS) Handles orchestration, routing, distributed state, and API coordination. Managed GC is acceptable here because it never touches live media streams.

2. Media Plane (Rust) Handles resampling, low-pass filtering, VAD, and packet-level audio processing with deterministic memory behavior.

Rust’s ownership model eliminates the need for a background garbage collector entirely. Allocation and deallocation are resolved at compile time, allowing the Media Plane to maintain a flat latency profile even under sustained throughput.

We also eliminated traditional synchronization primitives.

Mutexes in real-time audio systems introduce priority inversion risks that immediately surface as glitches or packet jitter. Instead, the engine relies on fully lock-free communication patterns:

• SPSC ring buffers for PCM transfer between socket and DSP threads • Michael-Scott queues using atomic CAS operations for multi-producer coordination

Rust’s SIMD support additionally allowed us to leverage AVX-512 and ARM NEON instructions to process multiple audio samples per instruction cycle, significantly increasing call density per CPU core.

The takeaway: managed runtimes are exceptional for distributed systems and asynchronous I/O. But once your workload crosses into hard real-time media constraints and human perceptual boundaries, averages stop mattering. Tail latency becomes the entire system.

By separating orchestration from deterministic signal processing, we reduced P99 latency from 1,550 ms to a stable 310 ms under load.

Our full engineering breakdowns, including the mathematical foundations behind our O(n) dual-gate VAD signal logic, are detailed in the Vivik whitepaper:

https://vivik.bajpailabs.com/whitepaper

Would love to hear how others are approaching real-time media constraints alongside LLM execution boundaries.

MLX Vulkan Back End

Superhot Rock Holds the Energy of the Future

Connections (British TV Series)

Trump wants $1B to protect White House ballroom from drones and other threats

AI is just unauthorised plagiarism at a bigger scale

Gemini accused of 30k-line code purge and fake recovery report

Setting Up OpenClaw with Slack in a Sandbox

Hating AI Is Good

Xiaomi YU7 GT Breaking the Nürburgring SUV Lap Record 7:22:755[video]

Micropatching Brings the Abandoned Equation Editor Back to Life (2018)

AI slop is flooding maths YouTube [video]

OWASP PTK is now OWASP Lab project

Ask HN: Are there any social media sites that are AI positive?

Avoid unnecessary parser lookahead for operators

AI chief of staff framework with invisible shadow predictions

Show HN: We dropped Go for Rust in our real-time telephony AI media plane

If Your Mother Says She Loves You: A Reporter's Cautionary Tale

Designing Firefox for the Future

US employers spend more than $1.5B a year to fight labor unions, report finds

Another Book Giveaway on LaTeX.org: Win the LaTeX Beginner's Guide 2026 Edition

1-Wire

Title: Show HN: myVanilla.js – In-memory JIT compiler, 100k reactive nodes in 4s

Creating another MCP server, but this one is for research

The Federal Data Field Guide

Matterhorn pandas

Magic the Gathering format: Fun 40

Second Quantisation – A Quantisation Too Far

Ma Configuration SSH · Accueil

My 1993 Atari Mega STE Retro Battlestation

Anthropic to open Milan office, expanding push into Europe

Show HN: Aisbf, a self-hostable OpenAI-compatible AI proxy/router

Comments