Distributed systems. Production AI. Nanosecond C++.

A live caller on one end and a language model on the other, inside the latency budget a person will tolerate mid-sentence. Split into two services that share no memory and no RPC, only a broker.

1. 01 Ingress call accepted, agent and prompt resolved
2. 02 Dispatch job to a stream, capacity from a heartbeat registry
3. 03 Bridge audio over pub/sub, no socket back to ingress
4. 04 Clean ONNX noise suppression, then Silero VAD
5. 05 Transcribe streaming, partial hypotheses
6. 06 Turn VAD primary, transcription fallback, min-word confirm
7. 07 Reason LLM turn with a tool schema, retrieval injected
8. 08 Speak synthesis back to the caller, barge-in aware

Decisions, and why

• Ingress and workers share nothing but a broker The two halves scale on different signals, one on concurrent sockets and one on concurrent pipelines. A worker restarting is invisible to a caller who is mid-sentence.
• Two broker instances, not one Application cache traffic must never contend with the call plane. Audio pub/sub latency sits on the critical path; a cache read does not.
• Each of STT, LLM and TTS sits behind a proxy A live call can swap provider, or hand off to a human, without rebuilding the pipeline or dropping the call.
• Workers recycle on a call counter A reference-cycle leak in a dependency was cheaper to reclaim by restarting the worker every few calls than to chase upstream. Nothing holds caller state, so the restart is free.

• Heartbeats expire, so a dead worker stops being handed jobs instead of black-holing them
• Caller barge-in cancels the model turn already in flight
• Idle turn counting and a hard call timeout end a stuck call deterministically
• Recordings and telemetry fan out asynchronously, never on the audio path

Contributed to this product from the proof-of-concept stage.

A question in natural language goes in, the passages that actually answer it come back ranked. Hybrid, because semantic search alone misses exact terms and keyword search alone misses meaning.

1. 01 Embed dense plus SPLADE sparse, computed concurrently
2. 02 Scope tenant and scope filters applied to both searches
3. 03 Search cosine on dense, IDF-weighted inner product on sparse
4. 04 Over-fetch max(top_k x 3, 30) candidates per side
5. 05 Fuse reciprocal rank fusion, k = 60
6. 06 Rerank BGE cross-encoder scores query and passage jointly
7. 07 Log latency breakdown and hit summary, off the response path

Decisions, and why

• Reciprocal rank fusion, not a weighted score blend RRF is rank-based, which makes it robust to the scale mismatch between cosine similarity (bounded from -1 to 1) and a sparse inner product (unbounded). Weighted blending would need constant retuning.
• Cross-encoder rerank rather than trusting the retrieval order The bi-encoder used for embedding never sees the query and the passage together. A cross-encoder does, which is why it reorders the top of the list so much better.
• Reranker runs locally, not behind an API It removes an external dependency from the latency path, and the model is small enough to run in-process. It goes in a thread pool so it cannot block the event loop.
• Query logs buffer in memory, not through a broker A broker adds a round trip and buys no durability worth having for analytics data. A bounded queue with periodic batch flush is cheaper, and sheds load by dropping entries rather than by blocking a response.
• Sparse encoder chosen for its runtime, not its benchmark The faster alternative depends on a Rust stemmer extension that segfaults on the target Python version. SPLADE ships ONNX-only with no native extensions, so it was the one that could actually run.

• Backpressure sheds log entries instead of blocking the query response
• Zero-hit queries are indexed separately so retrieval gaps stay measurable
• The two searches are independent, so one degrading does not empty the result set
• Every query records a per-stage latency breakdown for embedding, search and rerank

Designed and built this retrieval pipeline end to end, including the fusion and reranking strategy above.

Call recordings in, structured and actionable insight out, with every analysis dimension configurable by the team that owns the calls.

1. 01 Ingest event-driven, partitioned, ordered
2. 02 Transcribe speech to text plus diarization
3. 03 Compose prompt built per tenant config
4. 04 Extract LLM with retrieval grounding
5. 05 Contract structured output, schema checked
6. 06 Evaluate quality, tool errors, regressions
7. 07 Act downstream triggers and reporting

Decisions, and why

• The prompt is composed, not written Every team wants different signals from the same call. A collections team and a sales team share no analysis dimensions, so the prompt became an artifact the system builds per configuration.
• A rigid output contract the model fills Parsing must never have to guess. A fixed structure holds no matter how many dimensions a tenant configured, and anything that fails validation goes to the dead-letter queue rather than downstream.
• Bound the data, never the instructions Dimensions are capped per call and long transcripts are chunked, so as a configuration grows the model sees more data but never a longer instruction scaffold. That is what keeps quality stable across configurations nobody tested.
• Evaluation built before scale, not after Output quality, tool-call failures and regressions are tracked per release. Without that an LLM pipeline is a demo you cannot safely put on call.

• Retry with backoff on transient failures
• Dead-letter queue for anything that will not validate
• Deterministic replay from the event log
• Schema validation before any downstream write