// Services / Inference Infrastructure
Private inference.
Production throughput.
We size, configure, and operate model inference inside your environment - selecting the right serving stack, quantisation strategy, and hardware layout so regulated workloads get deterministic latency without shipping data to a third-party API.
// Serving engine selection
Choose the right stack before you build.
The serving stack sets hardware cost, update cadence, and latency profile. vLLM is the right default for most regulated workloads. TensorRT-LLM fits deterministic latency SLAs with a planned engine build on each model update.
| vLLM | TensorRT-LLM | Triton + backend | |
|---|---|---|---|
| Best for | High-throughput regulated workflows, OpenAI-compatible HTTP drop-in | Sub-50ms latency SLAs, NVIDIA-only, fixed model cadence | Multi-model ensembles, heterogeneous backends, complex pipelines |
| Batching | Continuous batching (PagedAttention) - best utilisation at scale | In-flight batching, highly optimised CUDA kernels | Routing layer; backend handles batching |
| Hardware | NVIDIA, AMD (ROCm), AWS Inferentia | NVIDIA only - A100, H100, H200 | Any - backend handles HW; Triton is the routing layer |
| Model updates | Hot-swap model weights; no recompilation | Engine build per model update (~30-90 min) | Depends on backend; model repo reload is fast |
| Quantisation | AWQ, GPTQ, FP8, INT4 out of the box | FP8, INT8, INT4 - best raw performance at low precision | Passthrough - backend handles quantisation |
| Observability | Prometheus metrics, OpenTelemetry, structured logs | Custom perf counters; less native observability | Strong - dedicated stats endpoint per model |
| Our verdict | Default choice | Latency-critical only | Pipeline orchestration |
// Configuration & deployment
From cluster to serving endpoint.
We handle the full path: GPU provisioning, driver and CUDA stack, engine build or model loading, and the OpenAI-compatible API / gRPC layer that drops into your existing workflow.
--enable-chunked-prefill on
vLLM substantially improves P99 latency under mixed prompt-length traffic by preventing long prompts from
blocking short ones. It should be on by default for regulated intake workflows where document length
varies unpredictably.
- Multi-GPU tensor parallelism configured for your node layout
- KV cache sizing tuned to your context window and concurrent sessions
- Health checks, autorestart, and Prometheus scrape endpoints wired in
- PII redaction layer upstream of inference, with sensitive data filtered before model input
// Reference benchmarks
What to expect from a tuned private deployment.
| Model | Backend | Hardware | Quant | Throughput (tok/s) | P99 TTFT (ms) | Est. infra cost / 1M tok |
|---|---|---|---|---|---|---|
| Gemma 4 12B IT | vLLM / SGLang | 1x L40S 48GB | AWQ 4-bit / FP8 | 6,000-10,000 | 60-140 | $0.08-$0.25 |
| Gemma 4 31B IT | vLLM / SGLang | 1x L40S 48GB or RTX PRO 6000 96GB | AWQ 4-bit / FP8 | 3,000-5,500 | 90-220 | $0.15-$0.45 |
| Qwen3.5 27B | vLLM / SGLang | 1x RTX PRO 6000 96GB or L40S 48GB | AWQ 4-bit / FP8 | 2,500-5,000 | 120-300 | $0.20-$0.70 |
| gpt-oss-120b (high) | vLLM | 1x H100 80GB or MI300X 192GB | MXFP4 | 1,800-3,000 | 150-400 | $0.80-$2.50 |
| Mistral Medium 3.5 | vLLM | 2x H100 80GB | AWQ 4-bit / FP8 | 1,500-2,500 | 160-400 | $2.00-$5.00 |
| Nemotron 3 Ultra | TensorRT-LLM | 4x RTX PRO 6000 96GB or 4x B200 180GB | NVFP4 / FP8 | 6,000-9,000 | 180-380 | $1.50-$4.00 |
| Qwen3.5 397B A17B | SGLang | 8x B200 180GB | FP8 / NVFP4 | 6,500-9,000 | 240-520 | $4.00-$8.00 |
| MiniMax-M3 | TensorRT-LLM / SGLang | 4-8x B200 180GB | FP8 / NVFP4 | 7,000-10,000 | 220-500 | $3.00-$7.00 |
| DeepSeek V4 Pro (Max) | SGLang | 8x B200 180GB | FP8 / NVFP4 | 6,500-9,000 | 280-600 | $4.00-$9.00 |
| GLM-5.2 (max) | TensorRT-LLM / SGLang | 8x B200 180GB | FP8 / NVFP4 | 8,000-12,000 | 260-650 | $3.50-$8.00 |
Note: TTFT = time to first token. Figures are planning ranges for the inference server only, measured at batch size 32 under sustained load. Cost estimates are cloud-equivalent infrastructure costs at high utilisation and exclude customer-specific HA, private networking, storage, monitoring, support, security review and reserved-capacity terms. For low-volume or highly redundant secure Azure/AWS deployments, effective cost per million tokens can be materially higher. June 2026 rates.
// Quantisation strategy
Precision vs. performance trade-offs.
The right quantisation method depends on your accuracy floor, model size, update cadence, serving stack, and hardware.
Activation-aware weight quantisation protects the weight channels that matter most to model output, usually preserving instruction-following quality better than naive INT4.
Post-training, layer-wise quantisation with error compensation. Comparable to AWQ in many deployments, with broad model support and useful fallback coverage.
Hardware-accelerated precision on H100, H200, and B200-class GPUs. Strong balance of throughput, memory use, and answer quality when calibrated correctly.
Aggressive compression for fitting larger models onto fewer GPUs or unlocking Blackwell-class FP4 throughput. Treat as a benchmarked deployment choice.
// Deployment patterns
Runtime architectures we deploy and operate.
Each pattern is matched to a specific operational constraint: single-node simplicity, multi-GPU scale, edge deployment, or ensemble pipelines where different models handle different workflow steps.
Single-node, multi-GPU
The fastest path to a production inference endpoint. One server, two to eight GPUs, tensor parallelism across them. Right for most regulated workloads under ~5k requests/hour. Simpler to operate, easier to audit.
Multi-node pipeline parallel
For models larger than a single node, 405B+ parameters, or long-context workloads with 256k+ tokens. Tensor and pipeline parallelism across nodes via NVLink or InfiniBand.
Quantized edge / on-prem
For physical-site data boundaries. llama.cpp with CPU inference or a single consumer GPU at INT4/Q4_K_M gives local execution with lean infrastructure.
Triton ensemble pipeline
Multiple specialised models wired into a single inference pipeline: a small classifier routes to a larger reasoning model; a dedicated embedding model feeds a retrieval step. Triton handles the routing and batching across backends.
Kubernetes autoscaling
For bursty regulated workloads such as batch processing windows, end-of-day compliance runs, or variable intake volumes. KEDA-based GPU autoscaling uses warm-pool pods for scale-up latency control.
PII-filtered inference proxy
A Rust proxy upstream of the model server that detects, redacts, and reversibly tokenises PII before it reaches the inference engine. Output is de-tokenised before returning to the caller. Full audit log at both layers.
// Infrastructure review
Already running inference? Get a concrete optimisation plan.
Book a one-hour infrastructure review. We'll look at your current serving configuration, quantisation choice, hardware layout, and cost-per-token, then return a concrete optimisation plan.