Taming an H200 — Part 1: The Inference Engine

If you've read the last two posts, you know I've been building the AI layer inside DIALOG at the Centre for Smart Governance (CSG), Karnataka. It's an agentic system that generates and edits Form.io schemas from natural language, with the queue-driven architecture I wrote about in Queue the Context. All of that runs against self-hosted LLMs on a GPU server and the models were fine, the problem was how they were being served.

For weeks I'd been hitting the same wall: every request processed sequentially. My agentic pipeline fires dozens of small, focused LLM calls per user interaction: draft a section schema, infer validations, classify an edit, repair a component. With the old setup, each of those calls queued behind every other user's requests. A single "generate this form" action that needed 15 micro-tasks at ~500 ms each would take 7+ seconds if nobody else was using the server. If two people were using it simultaneously? One of them just waited. Latency wasn't the model thinking, it was just waiting in line. My multi-form workflows would sometimes time out entirely. All the work I'd done to squeeze everything under 16k tokens was being wasted because the inference stack couldn't serve two users at once.

The machine itself was serious hardware. CSG had received GPU compute through the IndiaAI Mission, the central government initiative that's allocating high-end GPU infrastructure to state departments and public institutions for AI workloads. SSH in and run nvidia-smi: an NVIDIA H200 SXM with 140 GB of HBM3e, 4.8 TB/s memory bandwidth, 989 Tensor Cores, a 32-core Intel Xeon Sapphire Rapids, 188 GB of DDR5 RAM. A machine built for the kind of inference workloads I'd only read about. And most of its capability was going unused.

If your department has also received GPU compute through IndiaAI and it's currently running Ollama with default settings, this two-part blog is basically the playbook for what to do next. Configs, kernel params, architectural decisions, all documented here. The hardware is great, the gap is usually in how it's being served.

docker ps -a told the story: seven Ollama containers. Four models loaded with KEEP_ALIVE=-1 (which means "never unload, ever"). 124 GB of the 140 GB VRAM permanently locked. Seven more stopped containers just sitting there. Around 90 GB of dangling Docker images. Over 200 GB of dead model weights scattered across /data. The firewall was off. TLS was self-signed. No API key management, no rate limiting, no monitoring. The whole thing could serve exactly one user at a time per model.

So yeah, the context window problem from the last blog was only half the story. The other half was the infrastructure.

The book that rewired my brain

Around the same time I stumbled on the Inference Engineering book by Baseten Labs. I knew "serving a model" was more than ollama run but I hadn't internalized how much more. The book goes through the whole stack: how attention gets computed on Tensor Cores, why KV caching matters, continuous batching, PagedAttention, speculative decoding, quantization tradeoffs, what a production serving engine actually does under the hood. Dense, specific, doesn't handwave.

Reading it while looking at our Ollama setup was eye-opening. Every chapter covered something our setup wasn't doing, and in most cases could have been doing with the hardware we already had.

What Ollama was actually doing (and not doing)

Ollama is great for local experimentation. ollama pull, ollama run, chat. It uses llama.cpp under the hood with GGUF-quantized models and it works. But "works" and "actually uses the hardware well" are not the same thing.

Ollama on our H200 was

• Not using Tensor Cores. llama.cpp primarily uses CUDA cores with its own kernel implementations. The H200's 989 Tensor Cores, purpose-built for FP8/FP16/BF16 matrix multiplication, were sitting idle. The most capable part of the chip wasn't being used at all.
• Not doing continuous batching. One request at a time, per model. While User A's 500-token generation was running, Users B through Z were queued. On a chip that can do 48+ concurrent requests with proper batching.
• Using f16 KV cache exclusively. Every token in every layer's key-value cache was stored in 16-bit float. On an H200 with native FP8 Tensor Core support, this is 2x the memory waste for the KV cache. With 40 transformer layers, that adds up fast.
• No PagedAttention. No dynamic memory management for the KV cache. Memory was allocated in contiguous blocks, leading to fragmentation and wasted VRAM.
• No speculative decoding. No CUDA graph optimization. No prefix caching. No torch.compile kernel fusion. No chunked prefill.

Ollama is right for quick setup. But when you move to production loads you need to utilise the compute more efficiently. Can't have a 700W GPU sitting there serving one person at a time.

Picking the engine: SGLang vs vLLM

The Baseten book and everyone in the inference community kept pointing at two engines: vLLM and SGLang. Both do continuous batching, PagedAttention, CUDA graphs, FP8, all the stuff Ollama doesn't. Question was which one fits our workload.

The primary model I wanted to serve was Qwen3.5-35B-A3B-FP8, a Mixture-of-Experts model with 256 total experts and 8 active per token. So it's a 35B model that computes like a 3B (only 8 of 256 experts fire per token) but knows as much as the full 35B. The FP8 pre-quantized variant weighs about 20 GB on disk, versus 70 GB for the BF16 version.

I went with SGLang for a few concrete reasons.

• Native Qwen3.5 MTP support. Qwen3.5 ships with a Multi-Token Prediction head, which is essentially a built-in draft model for speculative decoding. SGLang had first-class support for this via the EAGLE algorithm. vLLM supported it too, but SGLang's implementation was more mature at the time and had an official Qwen3.5 recipe.
• RadixAttention. SGLang's prefix caching uses a tree-structured LRU that shares common prefixes across requests. If ten users all start with the same system prompt (which in a government API gateway, they absolutely do), RadixAttention caches the KV for that prefix once and reuses it. Published benchmarks showed 10–20% better cache hit rates than standard prefix caching.
• Raw throughput. On H100 benchmarks (the closest public comparison to our H200), SGLang showed 29% higher throughput than vLLM for similar MoE workloads.

Not gonna pretend this was a rigorous A/B test. I read the benchmarks, read the Qwen3.5 cookbook, looked at what the community was converging on for Hopper GPUs, and made a call. Sometimes engineering is just reading a lot and then picking something you can justify.

The SGLang configuration: what actually got deployed

This is where the Baseten book paid for itself. Every flag below maps to a concept from the inference stack. If you actually understand why each one matters you can debug things when they break instead of just blindly copying configs.

Final SGLang config for Qwen3.5-35B-A3B-FP8

# Memory: 55% of 140 GB = ~77 GB for KV cache pool
# Supports ~48 concurrent 32K-context requests
mem_fraction_static: 0.55
max_running_requests: 48

# Kernel optimization
enable_torch_compile: true    # JIT kernel fusion
attention_backend: fa3        # FlashAttention3 (Hopper-native)
chunked_prefill_size: 8192    # Process long prompts in 8K chunks

# Speculative decoding via EAGLE
speculative_algorithm: EAGLE
speculative_num_steps: 3      # 3 draft steps
speculative_num_draft_tokens: 4

# Quantization
# FP8 weights (pre-quantized on disk)
# FP8 KV cache (halves memory vs f16)
kv_cache_dtype: fp8

# Qwen3.5-specific
reasoning_parser: qwen3       # Parses <think> tokens
tool_call_parser: qwen3_coder # Native function calling

The ones that matter most.

mem_fraction_static: 0.55 allocates 55% of the 140 GB VRAM (~77 GB) as a static KV cache pool. Model weights eat ~20 GB (FP8), activation memory takes some, rest is KV cache. With FP8 KV cache and 40 transformer layers, 77 GB supports roughly 48 concurrent contexts at 32K tokens each. The Baseten book hammers on this: your maximum concurrency is gated by how much KV cache memory you have, not compute throughput. On an H200 compute is almost never the bottleneck, memory is.

attention_backend: fa3 selects FlashAttention3, optimized specifically for Hopper architecture. It exploits the H200's async copy engines and warp specialization to overlap GEMM computation with softmax and memory ops. H200 has dedicated hardware for this that didn't exist on Ampere. Using FA2 on an H200 is like using SSE on a chip that supports AVX-512. Works, but you're leaving perf on the table.

EAGLE speculative decoding (3 steps, 4 draft tokens) was the single biggest per-request latency win. Qwen3.5's MTP head acts as a lightweight draft model that predicts the next few tokens in parallel, main model verifies all 4 in a single forward pass (verification is almost free, you're just checking not generating). In practice this gave us 2.75x decode speedup. Model generates tokens nearly 3x faster than autoregressive decoding alone.

The core idea from the book: speculative decoding exploits the fact that verification is cheaper than generation. One forward pass through the full model can verify N draft tokens in the same time it takes to generate 1 token autoregressively. If the draft model is accurate enough (Qwen3.5's MTP head is, it was trained alongside the main model) you get near-linear speedup for free.

enable_torch_compile: true runs torch.compile on the model's forward pass, fusing multiple small CUDA kernels into larger ones. Instead of launching a separate kernel for each operation in a transformer layer (attention projection, FFN, normalization, etc.), the compiler fuses them. Fewer kernel launches, less overhead, better GPU utilization. The first few requests are slower because of compilation, but after warmup every subsequent request benefits.

The system-level tuning nobody talks about

Something the Baseten book touches on but most inference tutorials skip entirely. The OS between your serving engine and your GPU matters a lot. We were running Ubuntu 22.04 with entirely default kernel settings which meant

• vm.swappiness = 60 — the kernel was happily swapping out CUDA-related memory to disk
• vm.max_map_count = 65530 — SGLang loading 14 safetensor shards was hitting mmap limits
• vm.nr_hugepages = 0 — no hugepage support for large contiguous allocations
• TCP congestion control: cubic instead of BBR
• TCP buffers: 208 KB max (on a server that needs to stream multi-megabyte responses)
• vm.overcommit_memory = 0 — CUDA's large virtual memory mappings were getting rejected

I wrote a /etc/sysctl.d/99-llm-serving.conf with 37 kernel parameter changes. Highlights

# Let CUDA mmap whatever it wants
vm.overcommit_memory = 1

# Stop swapping GPU-adjacent memory to disk
vm.swappiness = 10

# SGLang loads 14 safetensor shards via mmap
vm.max_map_count = 1048576

# 2 GB of hugepages for large allocations
vm.nr_hugepages = 1024

# BBR congestion control (way better than cubic for streaming)
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

# TCP buffers: 208 KB → 16 MB
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 1048576 16777216
net.ipv4.tcp_wmem = 4096 1048576 16777216

# TCP Fast Open (client + server)
net.ipv4.tcp_fastopen = 3

# Keep connections warm
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_keepalive_time = 300

The BBR switch alone was noticeable on streaming responses. cubic is built for bulk throughput. BBR is built for latency and maintaining high utilization without filling buffers. When you're streaming tokens via Server-Sent Events to a client 50ms away, BBR's pacing algorithm means the client sees tokens as fast as the model generates them instead of in bursty chunks.

GPU hardware tuning

I also locked the GPU clocks. Never thought to do this before but it makes inference latency way more predictable. By default, the H200 dynamically scales its SM and memory clocks based on load. Good for power efficiency, bad for latency. GPU spends the first few hundred microseconds of every burst request just ramping up clocks.

# Lock SM clock at 1980 MHz (max sustained)
nvidia-smi -lgc 1980,1980

# Lock memory clock at 3201 MHz (max)
nvidia-smi -lmc 3201,3201

# Enable persistence mode (keep driver loaded)
nvidia-smi -pm 1

Also NVIDIA kernel module options: NVreg_UsePageAttributeTable=1 for direct page table mapping and NVreg_InitializeSystemMemoryAllocations=0 to skip unnecessary memory init. Small wins on their own but together they shave tens of milliseconds off every cold request.

Wrapped all of this in a gpu-init.service that runs at boot before Docker starts. Also turned on Receive Packet Steering (RPS) across all 32 CPU cores and Receive Flow Steering (RFS) with 32768 flow entries so streaming responses from 48 concurrent users don't bottleneck on a single CPU core handling all the NIC interrupts.

The great cleanup: 495 GB freed

Before the new stack could even fit I had to deal with the debris. Not as exciting as configuring EAGLE but had to be done.

• 13 old Ollama model directories: 146 GB
• Leftover HuggingFace weights (Llama-70B, Gemma-12B, LFM-2.5): 65 GB
• Unused Docker images: ~193 GB
• Docker dangling images + build cache: ~88 GB
• Stale Open WebUI, OpenClaw, and various caches: ~3 GB

Total: ~495 GB freed. The /data disk went from 572 GB used (57%) to 308 GB (33%). A good 40% of our storage had been consumed by models that were downloaded once, tried once, and never cleaned up. Pretty common when multiple people experiment on the same server over time.

The new stack: 6 containers, 4 AI services

After the migration, the final running setup.

1. Qwen3.5-35B container — SGLang serving Qwen3.5-35B-A3B-FP8 (84 GB VRAM)
2. Qwen3.5-0.8B container — SGLang serving Qwen3.5-0.8B for lightweight tasks (4.8 GB VRAM)
3. OCR container — vLLM 0.18.0 serving GLM-OCR 0.9B for document OCR (7.1 GB VRAM)
4. Embeddings container — Text Embeddings Inference serving nomic-embed-text-v1.5 (1.0 GB VRAM)
5. API gateway — Custom FastAPI gateway (auth, rate limiting, routing, dashboard)
6. Reverse proxy — nginx for TLS termination

Total VRAM: 97.6 GB used (68%), 45.6 GB free. GPU idling at 30°C and 111W of its 700W TDP. Side note on the OCR model: GLM-OCR 0.9B was #1 on OmniDocBench at the time and at 0.9B params it replaced a 43.8 GB PaddleOCR Docker image someone had tried setting up before. Smaller model, better results, way less VRAM.

The numbers: before vs after

Ran identical prompts through both setups and measured throughput, latency, and concurrency.

┌───────────────────────┬─────────────────┬────────────────┬──────────────┐
│ Metric                │ Ollama (Before) │ SGLang (After) │ Improvement  │
├───────────────────────┼─────────────────┼────────────────┼──────────────┤
│ 35B tok/s             │ ~20–30          │ 256            │ 8–12x         │
│ 35B TTFT              │ 1–5 seconds     │ 184 ms         │ 5–27x        │
│ Concurrent users      │ 1               │ 48+            │ 48x          │
│ Free VRAM (compute)   │ ~16 GB          │ 45.6 GB        │ 3x more      │
│ Gateway overhead      │ N/A             │ <5 ms          │ —            │
│ Production readiness  │ 35/100          │ 82/100         │ —            │
└───────────────────────┴─────────────────┴────────────────┴──────────────┘

SGLang numbers by response length

Qwen3.5-35B (SGLang, FP8, EAGLE, FA3):
  Short  (20 tokens):   184 ms,  109 tok/s
  Medium (200 tokens):  803 ms,  249 tok/s
  Long   (500 tokens): 1955 ms,  256 tok/s

Qwen3.5-0.8B (SGLang):
  Short  (11 tokens):   80 ms,  138 tok/s
  Medium (64 tokens):  190 ms,  337 tok/s
  Long   (253 tokens): 667 ms,  379 tok/s

To sanity-check I compared against Millstone AI's benchmark for the same model (Qwen3.5-35B-A3B-FP8) on the same GPU (1x H200 SXM) running vLLM. Their single-user generation at 1K context: 212.5 tok/s. Ours: 256 tok/s. About 20% faster per-user, which makes sense since EAGLE speculative decoding gives us that edge (their benchmark is vanilla vLLM without it). Their TTFT at 1K context is 77 ms vs our 184 ms, they win there, probably because our measurement includes the API gateway overhead and a longer prompt. Their peak system throughput hits 1,479 tok/s at 15 concurrent requests, that's the benefit of vLLM's batching at higher concurrency and a number I want to chase next.

Point being we're in the right ballpark. This isn't a tuned benchmark, it's a production govt server with an API gateway and auth middleware and real workloads. Within 20% of published benchmarks on per-user speed and beating them on single-stream decode thanks to EAGLE.

184 ms TTFT means you type a query, hit enter, and the first word is already there. The 0.8B model at 379 tok/s is fast enough for autocomplete, classification, routing, anything that needs sub-200ms end-to-end.

The API gateway: because a naked model endpoint is not a product

You can't just expose SGLang's /v1/chat/completions endpoint to government departments and call it a day. You need authentication, rate limiting, usage tracking, and the ability to answer "who used how many tokens last month" without spelunking through nginx logs.

So I built a FastAPI gateway in front of all four AI backends. What it does

• API key auth with SHA-256 hashed keys (never stored raw), per-key rate limits, daily token budgets, IP restrictions, and model restrictions
• Auto-routing so you send a request with model: "qwen3.5-35b" and the gateway routes to the right backend. No need for clients to know internal ports or container names
• Streaming proxy with SSE passthrough with connection pooling (100 max connections, 20 keepalive). The gateway adds <5 ms overhead on streaming responses
• Usage logging where every request is logged to SQLite (WAL mode): tokens in, tokens out, latency, model, IP, API key ID
• Admin dashboard with 9 pages covering GPU stats, backend health, API key management, usage analytics, audit logs, and a chat playground for testing

Yeah it's SQLite WAL for the database. For a single-server setup with a few hundred concurrent users it's solid and zero-maintenance. PostgreSQL would've been overkill, it's basically a key-value store with some counters. Will switch when the workload demands it.

What I actually learned

Biggest takeaway wasn't any single config flag. Before this project "serving a model" to me meant "load it and call it." Now I think of it as a systems engineering problem. Memory hierarchy, kernel scheduling, network tuning, quantization, batching, hardware-specific stuff, it all interacts and the differences are orders of magnitude not percentages.

The 8-12x throughput improvement wasn't one clever trick. It was the compound effect of FP8 quantization (2x less memory, native Tensor Core path), FlashAttention3 (Hopper-native attention), EAGLE speculative decoding (2.75x decode speed), continuous batching (48 concurrent vs 1), FP8 KV cache (2x more cache capacity), torch.compile (fused kernels), locked GPU clocks (consistent latency), BBR (smoother streaming), and 37 kernel params that stopped the OS from sabotaging the GPU.

None of this is secret, it's all documented. Hard part is just knowing these things exist and having the patience to measure before and after.

The playbook: if your department just got a GPU through IndiaAI

Dozens of state departments and public institutions are getting H100/H200-class GPU allocations through IndiaAI right now. If you're in that position and your server is running Ollama with defaults, here's the step-by-step. Everything below is what we did, in order, configs and tools documented above.

1. Audit what's running. docker ps -a, nvidia-smi, df -h. Know your VRAM usage, how many containers are running vs stopped, and how much disk is consumed. We found 124 GB of VRAM locked by idle models and 495 GB of dead weight on disk.
2. Clean up. Remove stopped containers, dangling images (docker image prune), orphaned model weights, stale HuggingFace caches. If your server has had multiple people experimenting over months, expect to recover hundreds of gigabytes.
3. Replace Ollama with SGLang or vLLM. This is the single highest-impact change. You go from one user at a time to 48+ concurrent, from ~20 tok/s to 250+ tok/s, and from zero Tensor Core usage to native FP8 compute. Use FP8 pre-quantized model weights — they halve VRAM and run natively on H100/H200 Tensor Cores with near-zero quality loss.
4. Enable speculative decoding. If your model supports MTP (Qwen3.5 does), turn on EAGLE in SGLang. This gave us 2.75x faster per-request decode with no accuracy loss. It's a config flag, not a code change.
5. Tune the kernel. Copy the sysctl config above into /etc/sysctl.d/99-llm-serving.conf and run sysctl --system. The critical ones: vm.swappiness=10, vm.max_map_count=1048576, vm.overcommit_memory=1, tcp_congestion_control=bbr, TCP buffers to 16 MB. Takes 5 minutes, measurable impact on every request.
6. Lock GPU clocks. Three commands: nvidia-smi -pm 1, nvidia-smi -lgc <max_clock>,<max_clock>, nvidia-smi -lmc <max_mem_clock>,<max_mem_clock>. Eliminates cold-start clock ramp latency. Wrap in a systemd service so it persists across reboots.
7. Build an API gateway. Don't expose raw model endpoints. You need SHA-256 hashed API keys, per-key rate limits, usage logging, and an admin dashboard. We built ours in FastAPI with SQLite. Zero external dependencies, <5 ms overhead. Without this, you can't answer "who used how many tokens last month" and you have no access control.
8. Automate recovery. Create a systemd boot chain (gpu-init then docker then ai-stack), a cron-based health watchdog, and enable Docker live-restore. Government servers get rebooted. Your stack needs to come back without manual intervention.
9. Benchmark and compare. Run your own latency and throughput tests. Compare against published benchmarks like Millstone AI for your specific model and GPU. If you're within 20% of published numbers on a production server with auth middleware, you're doing well.

IndiaAI gave departments the hardware. What's missing is knowing how to make it work at capacity. Everything above is open-source, configs documented with reasoning, numbers reproducible. 20 tok/s to 256 tok/s is not a hardware upgrade, it's just configuring things properly.

In Part 2 (coming soon) I'll cover the other half of the project. Building a dual speech-to-text pipeline (Whisper + IndicConformer for 22 Indian languages), the WebSocket streaming ASR pipeline with a Silero VAD state machine and partial transcriptions, optimizing IndicConformer from 650 ms to 107 ms per utterance by switching from RNNT to CTC-only decoding and tuning ONNX Runtime threads for Sapphire Rapids, and stress-testing the whole stack at 128 concurrent users with zero errors.