Taming an H200 — Part 2: Speech, Streaming, and Stress Tests

In Part 1, I covered how we migrated from 7 Ollama containers to SGLang on an NVIDIA H200 — FP8 quantization, EAGLE speculative decoding, FlashAttention3, 37 kernel parameters, and a 4–6x throughput improvement. The LLM side was humming. But the H200 was still sitting at 68% VRAM utilization, and there was a second workload waiting: speech-to-text for Indian languages.

Government services in Karnataka interact with citizens who speak Kannada, Hindi, Tamil, Telugu, Urdu, and dozens more. Any STT system that only does English is a nice demo and nothing more. I needed something that handled 22 Indian languages with usable accuracy, at latencies low enough for real-time streaming transcription, while sharing the GPU with the LLM stack.

Why two models, not one

The obvious choice for multilingual STT is Whisper large-v3. It supports 99+ languages, it's well-understood, and there's mature tooling around it. But here's the thing about Whisper and Indian languages: it's… okay. Not great. The word error rates for Hindi, Kannada, and other Indic scripts are significantly worse than for English. This isn't a Whisper bug — it's a training data distribution issue. Whisper saw overwhelmingly more English audio during pretraining.

AI4Bharat's IndicConformer (600M parameters) was built specifically for this gap. It's a Conformer-based ASR model trained on the IndicSuperb dataset covering 22 Indian languages. On the Vistaar benchmark, it holds the best open-source WER for Hindi at ~13.6%. It's not perfect, but it's the best available for this specific domain.

So the architecture became a language router: detect the language of the incoming audio (or accept a language hint from the client), and route Indic languages to IndicConformer and everything else to Whisper. This might seem over-engineered — "why not just use Whisper for everything?" — but the accuracy difference on Kannada and Hindi was large enough that deploying a single model would have meant shipping something that works well for English-speaking officers and poorly for Kannada-speaking citizens. In a government context, that's not a tradeoff you get to make.

The other reason is architectural: Whisper runs on the GPU (CTranslate2 with INT8 quantization), and IndicConformer runs entirely on the CPU (ONNX Runtime with AVX-512). This means they never compete for GPU resources. The LLM stack keeps its VRAM, Whisper gets a small GPU allocation (~2.6 GB — model weights, KV cache, activations), and IndicConformer saturates CPU cores that would otherwise be idle. Natural resource isolation without any orchestration complexity.

Whisper: CTranslate2, INT8 Tensor Cores, and beam size economics

I didn't serve Whisper via the standard PyTorch path. CTranslate2 is a C++ inference engine that converts Whisper's PyTorch weights to an optimized format with INT8 quantization and custom fused CUDA kernels — fused layer normalization, fused GELU activation, INT8 GEMM via cuBLAS dispatched to Tensor Cores, and fused beam search scoring. On our H200, PyTorch Whisper takes ~1200 ms for 3-second audio. CTranslate2 does it in ~260 ms. 4.6x faster, same model, same accuracy.

The INT8 quantization here is particularly well-suited to the H200. The Tensor Cores on Hopper execute INT8 matrix multiplications at 2x the throughput of FP16. CTranslate2 actually dispatches the encoder's attention and FFN layers as INT8 GEMMs via Tensor Core instructions (SM 9.0 mma.sync with INT8 operands, FP32 accumulation). The weights go from 6.2 GB (FP32) to 1.6 GB (INT8), and VRAM for the whole Whisper pipeline — weights, KV cache, activations — is about 2.6 GB. That's 1.8% of the H200's 140 GB. CTranslate2 also allocates all CUDA memory once at model load via cuBLAS directly (no PyTorch autograd overhead), so there's zero dynamic allocation during inference — CUDA OOM is structurally impossible.

The single biggest Whisper-specific optimization was embarrassingly simple: reducing beam size from 5 to 2. Beam search explores K candidate transcriptions in parallel. Each hypothesis requires its own decoder forward pass. The decoder cost scales as O(beam_size × sequence_length × d_model²). Reducing beam from 5 to 2 cuts decoder compute by 60%, and since the decoder is about 30% of total inference time, the end-to-end speedup was nearly 2x (480 ms → 241 ms on 3.16s audio).

Why not beam=1 (greedy)? I tested it. Greedy was actually slightly slower than beam=2 — 267 ms vs 241 ms. The reason is subtle: beam=2 gives CTranslate2's internal pruning heuristics something to work with. With beam=1, there's no alternative hypothesis to discard, so the engine can't prune early. Beam=2 gives one fallback hypothesis and allows aggressive pruning, which on average terminates decoding steps faster.

Silero VAD: filtering silence before it wastes GPU

Before any audio hits Whisper, it passes through Silero VAD v5 — a small ONNX-based voice activity detector that processes audio in 512-sample frames (32 ms at 16 kHz). Each frame gets a speech probability between 0.0 and 1.0. Frames below the 0.5 threshold are discarded as silence.

Why Silero instead of WebRTC VAD? WebRTC VAD is faster (~0.01 ms/frame vs Silero's ~0.1 ms/frame), but for 3 seconds of audio that's 93 frames — so 0.93 ms vs 9.3 ms total. The 8.4 ms difference is irrelevant against Whisper's 260 ms inference. Silero wins because it's LSTM-based (stateful, tracks speech context across frames), outputs a probability score (not just binary), and is significantly more robust to background noise. Fewer false-positive speech segments = fewer unnecessary Whisper calls = better throughput under load.

On typical government audio (meetings, call center recordings), 40–60% of the signal is silence or background noise. VAD effectively halves the audio Whisper needs to process, which directly translates to proportional encoder speedup.

IndicConformer: the 650 ms → 107 ms journey

This was the most satisfying optimization of the entire project. The original IndicConformer setup — load the ONNX model, feed it audio, get text — took 650 ms per utterance for a 3-second clip. That's an RTF of 0.22x — technically real-time, but barely, and under load it would degrade fast.

I spent two days profiling and the latency came down to 107 ms. That's a 6x improvement. Here's every change, in order of impact:

1. CTC-only decoding (the 84x decode win)

IndicConformer ships with two decoder heads: CTC (Connectionist Temporal Classification) and RNNT (Recurrent Neural Network Transducer). The default pipeline runs both and merges their outputs for slightly better accuracy.

But here's what the profiler showed: the Conformer encoder took about 90 ms. The CTC decode took 1 ms. The RNNT decode took 900 ms.

Why such a gap? CTC decoding is embarrassingly simple: take the encoder's output (a matrix of shape [T, vocab_size]), apply a language-specific mask to zero out irrelevant tokens, argmax each timestep, collapse consecutive duplicates, and remove blanks. It's O(T × V) — a single linear scan. The whole thing is a matrix operation followed by a filter. 1 ms.

RNNT, on the other hand, is autoregressive. It runs a 2-layer LSTM prediction network that generates one token at a time, each dependent on the previous output. For every encoder frame, the RNNT joint network combines the encoder output with the prediction network output, projects through per-language post-nets, and loops until it emits a blank token. That's O(T × U × H) where T is encoder frames, U is output length, and H is hidden dimension (640). On a CPU, this is death by sequential computation. 900 ms.

Short version: the fancy decoder was 900x slower than the simple one and barely more accurate. The quality difference was ~1–2% WER on Hindi — perfectly acceptable for real-time government transcription. Kill it.

I kept RNNT available behind a decode_type="rnnt" parameter for batch jobs where latency doesn't matter and that extra 1–2% accuracy is worth the wait. But for streaming and real-time? CTC only.

2. ONNX Runtime thread tuning for Sapphire Rapids

The original code created ONNX sessions with entirely default settings. That meant intra_op_num_threads=0 (OS default — effectively 1–2 active threads), inter_op_num_threads=0, and execution_mode=SEQUENTIAL. On a 32-core Xeon, this was using maybe 5% of available CPU.

After experimentation, the sweet spot for the encoder was 16 intra-op threads, 1 inter-op thread, SEQUENTIAL mode. The Conformer encoder is a linear chain of 17 blocks — block N depends on block N-1 — so there's zero graph-level parallelism. Setting inter_op_threads=1 avoids wasting threads on inter-op scheduling. But within each block, the self-attention and convolution modules have branches that ONNX Runtime can parallelize with the 16 intra-op threads.

I tested the encoder at 8, 12, 16, and 24 threads: 8t→261 ms, 12t→249 ms, 16t→161 ms, 24t→varies due to contention. The jump from 12 to 16 threads is non-linear because at 16 threads the GEMM kernels can fully utilize the AVX-512 vector units across half the available cores. Beyond 16, threads start contending for L3 cache and memory bandwidth.

For the CTC decoder (which is tiny): 12 intra-op, 2 inter-op, ORT_PARALLEL mode. The decoder's computation graph has parallel branches (logprob computation + language masking can overlap), so ORT_PARALLEL with 2 inter-op threads actually helps here, unlike the encoder.

I also enabled enable_mem_pattern=True (pre-computes buffer reuse patterns across the graph — fewer allocations, better locality) and enable_cpu_mem_arena=True (pre-allocates a ~256 MB contiguous arena for all intermediate tensors, reused across inference calls). And session.intra_op.allow_spinning="0" — disables spin-waiting after inference, so threads are immediately released between requests instead of busy-waiting for work that isn't coming.

On the OpenMP side: KMP_AFFINITY=granularity=fine,compact,1,0 pins threads to adjacent physical cores on the same NUMA node. Sapphire Rapids uses a tile-based architecture where cores within a tile share L2 cache. Compact affinity keeps the 16 ONNX threads on cores 0–15 (same tile group), maximizing L3 cache sharing for the Conformer's shared K/V matrices. Scattered affinity (the default) would spread threads across tiles, causing cache misses on every shared tensor. Combined with KMP_BLOCKTIME=0 (no idle spinning between parallel regions — critical for multi-model coexistence with Whisper and FastAPI on the same machine), this brought the encoder from 897 ms down to 161 ms — a 5.6x speedup just from thread configuration.

3. Killing the failed CUDA EP probes

This one was subtle but satisfying to find. On startup, the IndicConformer pipeline was attempting to initialize the ONNX Runtime CUDA Execution Provider — 28 times. And failing 28 times. The container image (nvidia/cuda:12.9.0-runtime) doesn't include cuDNN (libcudnn.so.9), which the CUDA EP requires. Each failed probe took a few hundred milliseconds of CUDA context initialization before falling back to CPU, adding ~1.4 seconds to startup and logging 28 warning messages.

Even if cuDNN were present, GPU wasn't the right choice for this model. The Conformer's hidden dim (512) is too small to benefit from GPU parallelism — the overhead of transferring mel spectrograms from CPU→GPU→CPU would eat any compute savings. And putting both STT models on the GPU would cause CUDA context contention with the LLM stack. The Sapphire Rapids AVX-512 VNNI instructions handle this model's GEMMs at ~2 TOPS per core, which at 16 cores gives ~32 TOPS — more than sufficient for a 600M parameter model doing one inference at a time.

The fix: explicitly configure providers=['CPUExecutionProvider']. Clean initialization, zero CUDA overhead, zero GPU contention.

The final numbers

IndicConformer (ONNX, CTC-only, CPU, optimized):
  Before: 650 ms / 3.16s audio, RTF 0.22x
  After:  107 ms / 3.16s audio, RTF 0.034x (30x real-time)
  Throughput: 8.7 requests/sec

Component-level breakdown of the 107 ms:
  Audio I/O + resampling:       ~5 ms
  Preprocessor (mel-spec, GPU): ~8 ms  (TorchScript JIT)
  GPU→CPU tensor transfer:      ~2 ms
  Conformer encoder (16 threads): ~90 ms (AVX-512 VNNI)
  CTC decoder + lang mask:      ~1.5 ms
  Vocab lookup + post-process:  ~0.5 ms

30x real-time on CPU only. For a 600M parameter model. Running on cores that the GPU stack wasn't using anyway. The preprocessor (mel-spectrogram computation) still runs on GPU via TorchScript because the pre-emphasis filter + Hanning window + 80-bin mel filterbank is embarrassingly parallel. But it's just 8 ms and uses negligible VRAM.

The streaming ASR pipeline: WebSocket, VAD state machine, and partial transcriptions

Batch transcription (upload file, get text back) was working. But the real requirement from government departments was live streaming transcription — an officer speaks into their browser microphone and sees text appearing in real time, word by word. That's a fundamentally different problem from batch. You're receiving audio in small chunks as the person speaks, you need to figure out when they've finished a sentence (or paused), and you need to show progressive results without waiting for the entire utterance.

The streaming pipeline runs over WebSocket at /v1/audio/stream. Here's the full data path:

Browser microphone (ScriptProcessor)
  → PCM16 LE, 16kHz, Mono
  → Chunk: 4096 samples (256 ms)
  │
  ▼
WebSocket (wss://)
  → nginx terminates TLS, passes ws:// to backend
  → proxy_read_timeout: 3600s (1 hr sessions)
  → proxy_buffering: off (real-time, no buffering)
  │
  ▼
FastAPI WebSocket Handler
  → Receives config JSON: {language, min_silence_ms, partial_interval_s}
  → Initializes StreamingASR state machine
  │
  ▼
StreamingASR State Machine
  │
  ├─ Audio Buffer
  │   PCM16 LE → float32 normalized [-1.0, 1.0]
  │
  ├─ Silero VAD (per 512-sample frame = 32 ms)
  │   │  speech_probability: 0.0–1.0
  │   │  cost: ~0.1 ms/frame
  │   │
  │   ├─ prob > 0.5 → SPEECH state
  │   │   ├─ First speech frame → emit {"type":"vad","is_speech":true}
  │   │   ├─ Append to speech buffer
  │   │   └─ Every partial_interval (2s of speech):
  │   │       → Transcribe accumulated buffer (non-destructive copy)
  │   │       → emit {"type":"partial","text":"..."}
  │   │
  │   └─ prob ≤ 0.5 → SILENCE state
  │       └─ If was speaking:
  │           ├─ Append trailing silence frames
  │           └─ If silence ≥ min_silence (600 ms = 9600 samples):
  │               → Transcribe complete segment
  │               → emit {"type":"final","text":"..."}
  │               → Reset state machine
  │
  └─ Max segment guard: 30s (480,000 samples)
      → Force-transcribe, reset

The VAD state machine: why 600 ms silence, 250 ms minimum speech

The silence threshold is the single most important parameter in the whole streaming pipeline. Too short and you split mid-sentence — "I need to apply for" [pause] "a new ration card" becomes two separate transcriptions. Too long and the user waits awkwardly after finishing a sentence.

Human conversational pauses between words are typically 200–500 ms. Sentence boundary pauses are 500–1000 ms. I settled on 600 ms — just above the typical within-sentence pause, catching most sentence boundaries without splitting compound sentences. I tested 300 ms (too aggressive, splits "main aapko… batana chahta hoon" mid-phrase) and 1000 ms (user finishes speaking and stares at the screen for a full second before text appears — feels broken).

The 250 ms minimum speech duration filters micro-sounds — clicks, coughs, breaths, the "um" at the start of a sentence. The shortest meaningful utterance ("yes", "haan") is ~200–300 ms. Anything shorter is noise. Without this filter, Silero's sensitivity would trigger transcription on keyboard clicks and door sounds, each burning a 260 ms Whisper inference for garbage.

The 30-second max segment exists because Whisper's positional encoding is hard-capped at 448 tokens, and its attention window is trained on 30-second segments. Feeding it longer audio would produce degraded output or truncation. 30 seconds at 16 kHz = 480,000 samples = 1.9 MB of float32 buffer — comfortable for memory.

Partial transcriptions: progressive text while the user is still talking

Without partials, the user experience is: talk for 15 seconds, stop, wait 260 ms, see all the text at once. That feels like a batch job with extra steps. With partials, the experience is: start talking, see text appearing every 2 seconds, final polished text replaces partials when you stop.

Every partial_interval_s (default 2.0 seconds) of continuous speech, the state machine copies the current speech buffer (non-destructive — the original keeps accumulating), routes it to the appropriate model (Whisper or IndicConformer), transcribes, deduplicates against the last partial, and emits {"type":"partial","text":"..."}. The client displays partials with a "…" suffix to signal "still listening."

When silence is finally detected, the complete segment (including everything from the partials plus trailing audio) gets transcribed as a {"type":"final"} that replaces all partials. The final may differ from the last partial because Whisper now has more context — the full sentence instead of the first 6 seconds of it. This is expected behavior and matches how commercial streaming ASR works (Deepgram, AssemblyAI — all show "interim" results that get refined).

The latency budget for partials matters: if a partial transcription takes longer than the partial interval, it would block audio ingestion and create a backlog. Whisper at ~260 ms and IndicConformer at ~110 ms are both well under the 2-second interval, so partials never block the audio pipeline. The WebSocket receive loop and VAD processing continue in the uvicorn async event loop while transcription runs.

nginx WebSocket proxying

The WebSocket connection from the browser goes through nginx before reaching the FastAPI handler. Nginx natively supports WebSocket proxying with the Upgrade and Connection header pass-through — zero overhead, no buffering. I set proxy_read_timeout and proxy_send_timeout to 3600 seconds because streaming sessions can be long (an officer transcribing a 45-minute meeting).

One tradeoff: the WebSocket endpoint currently bypasses the API gateway's authentication layer. The gateway (FastAPI) handles REST endpoints with SHA-256 hashed API keys, but the WebSocket stream goes directly from nginx to the STT container. For production, this needs auth token validation in the STT service's WebSocket handler — it's on the to-do list.

Long audio scaling: how both models behave beyond 3 seconds

Most STT benchmarks only test with short clips. Real government audio is meetings, dictations, call recordings — 10 seconds, 30 seconds, a minute. The scaling profiles of the two models are very different:

Audio Duration │ Whisper avg │ Whisper RTF │ Indic avg │ Indic RTF
──────────────┼─────────────┼─────────────┼───────────┼──────────
   3s         │    86 ms    │   0.029x    │  163 ms   │  0.054x
   5s         │    89 ms    │   0.018x    │  257 ms   │  0.051x
  10s         │   101 ms    │   0.010x    │  265 ms   │  0.027x
  30s         │   147 ms    │   0.005x    │  825 ms   │  0.028x
  60s         │   212 ms    │   0.004x    │ 1,942 ms  │  0.032x

Whisper scales sublinearly with audio length. 60-second audio takes only 2.5x longer than 3-second audio, not 20x. This is because the GPU parallelizes the encoder's convolution subsampling and multi-head attention across the full spectrogram — longer audio means bigger matrices, and bigger matrices utilize Tensor Cores more efficiently. The decoder length grows sublinearly too because longer audio tends to produce proportionally fewer tokens per second (more silence, longer words).

IndicConformer scales linearly — RTF stays constant around 0.03x regardless of audio length. This is expected: the Conformer encoder processes frames sequentially through 17 blocks, and the computation per frame is constant. 60 seconds is 20x more frames than 3 seconds, so it takes ~20x longer. Still well within real-time (0.032x RTF = 31x real-time), but the linear scaling means that for very long audio, the 30-second segment guard in the streaming pipeline isn't just a Whisper limitation — it also keeps IndicConformer's latency bounded.

The stress test: 128 concurrent users

Numbers at single-request latency are nice, but they mean nothing if the system falls apart under load. Government services don't have the luxury of "works fine for one user." I needed to know: what happens when 128 users hit this thing simultaneously?

Individual service tests (128 concurrent each)

Whisper @ 128 concurrent:
  256 requests, 100% success, 0 errors
  Throughput: 3.7 RPS
  Server p50: 256 ms, p99: 318 ms
  → Latency stays flat (GPU serializes gracefully)

IndicConformer @ 128 concurrent:
  256 requests, 100% success, 0 errors
  Throughput: 8.2 RPS
  Server p50: 112 ms, p99: 125 ms
  → CPU parallelism absorbs load well

Mixed STT (50% Whisper, 50% IndicConformer) @ 128 concurrent:
  Combined: 4.9 RPS, 0 errors
  → No cross-model interference (GPU vs CPU isolation confirmed)

The Whisper concurrency profile is interesting. Server-side latency stays almost perfectly flat from 1 to 128 concurrent users — p50 hovers at 256–267 ms regardless. This is because CTranslate2 serializes GPU requests internally: each audio clip gets exclusive GPU access for its forward pass, and the others queue in uvicorn's async loop. The queue fills fast but each item drains in ~260 ms, so individual request latency stays constant. What increases is wall-clock time: 128 concurrent requests at 3.7 RPS means the last request finishes about 35 seconds after it was submitted. But it was never waiting on a slow model — it was waiting on the 127 requests ahead of it.

IndicConformer scales slightly better because CPU parallelism allows ~2 concurrent inferences to overlap (16 threads each, 32 cores total). Peak throughput hits 8.7 RPS at 96 concurrent, then drops slightly to 8.2 RPS at 128 due to thread contention and memory bandwidth saturation.

Full GPU stress: all 4 services simultaneous

This was the test I was most nervous about. 32 Whisper requests + 32 IndicConformer requests + 32 LLM requests (Qwen3.5-35B) + 32 embedding requests, all hitting at the same time:

128 total concurrent requests across 4 services:
  128/128 successful, 0 errors
  Peak GPU utilization: 91%
  Average GPU utilization: 50%
  GPU temperature: 31–33°C (48°C headroom to thermal throttle)
  GPU power: peak 162W of 700W TDP (23%)
  GPU memory: 109 GiB, stable (no growth)

The H200 barely noticed. At full 128-concurrent load across all four services, it was running at 50% average compute, 23% power, and 42% thermal capacity. The machine has so much headroom that the stress test felt more like a warm-up than a limit test.

The burn test: 60 seconds of sustained 128-concurrent Whisper

60-second burn (128 concurrent Whisper, continuous):
  350 requests processed, 100% success, 0 errors
  Sustained throughput: 3.6 RPS
  GPU temp: 32°C → 35°C (only +3°C over 60 seconds)
  Peak power: 162W of 700W (76% power headroom)
  Memory delta: 416 MB (within normal alloc/dealloc variance, no growth trend)
  → Could sustain indefinitely

No memory leaks, no thermal creep, no error accumulation. The 416 MB memory delta is allocation/deallocation variance — no monotonic growth. CTranslate2's static memory allocation pattern means the VRAM footprint is identical at request 1 and request 350.

Making it survive a reboot

None of this matters if the whole thing falls over when someone restarts the server (which in a government datacenter, happens more often than you'd think). The production hardening was less exciting than the inference optimization but just as important:

• Boot chain: systemd → nvidia-persistenced → gpu-init.service (locks clocks, sets RPS/RFS) → docker → ai-stack.service (docker compose up with 600s timeout for model loading)
• Health watchdog: A cron job every minute that checks SGLang's /health endpoint. Three consecutive failures trigger an auto-restart of the affected container. Self-rotating log so it doesn't fill the disk.
• Docker live-restore: Containers survive Docker daemon restarts. A systemctl restart docker doesn't kill running inference.
• Database backup: The gateway's SQLite database gets copied to /data/backups/ daily at 2 AM, with 7-day retention.

Recovery times:

Container crash:       30–60 sec (Docker restart + watchdog)
Docker daemon restart: 0 sec    (live-restore: true)
Full server reboot:    5–10 min (model loading is the bottleneck)
GPU hang:              ~3 min   (watchdog detects, restarts SGLang)

The 5–10 minute reboot time is almost entirely model loading — SGLang loading Qwen3.5-35B FP8 weights from disk, initializing the KV cache pool, and warming up the torch.compile cache. There's not much to do about this besides faster storage (we're on a virtual disk, not NVMe) or preloading weights into tmpfs.

What I'd do differently

The biggest gap isn't performance — it's operational maturity. The firewall is still off (ufw inactive). TLS is self-signed (no Let's Encrypt because the server is behind a government network with non-standard DNS). There's no Prometheus/Grafana stack — we're relying on the custom gateway's built-in metrics and manual nvidia-smi checks. There's no fail2ban, no encryption at rest, no SSO/LDAP for the admin dashboard. The production readiness score I gave the final state was 82/100 — the missing 18 points are all security and observability, not performance.

On the STT side, the biggest throughput unlock would be Whisper's BatchedInferencePipeline from faster-whisper. Right now CTranslate2 processes one audio file at a time on the GPU — the encoder runs on a single spectrogram, and the next request waits. Batched inference would pack multiple spectrograms into a single GPU forward pass, potentially pushing throughput from 3.7 RPS to 10+ RPS. I didn't have time to integrate it, but it's the most obvious next step.

Adding uvicorn workers (currently running with 1 worker) would multiply IndicConformer's effective throughput almost linearly — each worker gets its own ONNX Runtime session and thread pool. And speaker diarization (pyannote-audio or equivalent) is the elephant in the room: the moment someone tries to transcribe a multi-speaker meeting, the absence of "who said what" will be immediately obvious.

What this project actually taught me

Before this fellowship, "inference" to me was model.generate(). Now it's a systems problem that spans memory hierarchy design (KV cache sizing, FP8 vs FP16 trade-offs, CTranslate2's static allocation vs PyTorch's dynamic pools), kernel-level CPU optimization (ONNX thread affinity, CTC vs RNNT decoder economics, AVX-512 VNNI dispatch on Sapphire Rapids), real-time streaming architecture (VAD state machines, partial transcription timing, WebSocket backpressure), and production reliability engineering (systemd chains, watchdogs, live-restore).

The final state of the server: 6 containers, 4 AI services (LLM, small LLM, OCR, embeddings) plus a dual streaming STT pipeline, all behind an authenticated API gateway, serving 128 concurrent users with zero errors, at 33°C and 23% of the GPU's power budget. The 140 GB of HBM3e that was 89% wasted on Ollama containers is now 68% utilized with 45 GB free for whatever comes next.

Inference engineering isn't about one clever trick. It's about understanding the full stack — from the AVX-512 instruction set to the WebSocket frame boundary — and systematically removing the things that stop your hardware from doing its job. Most of those things are defaults that made sense for someone else's workload.