AI Model Serving on K8s: vLLM vs Triton vs NIM (2026)

The Serving Stack Decision

Every production LLM deployment needs a serving framework. The three leaders:

vLLM: Maximum Open-Source Throughput

vLLM pioneered PagedAttention — treating KV-cache like virtual memory pages for near-optimal GPU utilization.

Deployment on Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama70b
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.8.4
          args:
            - "--model"
            - "meta-llama/Llama-3.1-70B-Instruct"
            - "--tensor-parallel-size"
            - "2"
            - "--max-model-len"
            - "8192"
            - "--gpu-memory-utilization"
            - "0.92"
            - "--enable-prefix-caching"
            - "--max-num-seqs"
            - "64"
            - "--dtype"
            - "auto"
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: "2"
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120
            periodSeconds: 10

vLLM Strengths

• PagedAttention — 24x higher throughput than HuggingFace Transformers
• Continuous batching — dynamic batch formation for optimal GPU use
• Tensor parallelism — split models across GPUs seamlessly
• OpenAI-compatible API — drop-in replacement
• Speculative decoding — use draft model for faster generation
• Prefix caching — share KV-cache for common system prompts
• Quantization — AWQ, GPTQ, FP8 native support

vLLM Limitations

• Single-framework (PyTorch only)
• No built-in model management
• No multi-model serving (one model per process)
• Limited ensemble support

Triton Inference Server: Multi-Model Platform

Triton serves any model from any framework with a unified API. It’s an inference platform, not just an LLM server.

Architecture

┌────────────────────────────────────────────┐
│            Triton Inference Server          │
│                                            │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐  │
│  │  Model A │ │  Model B │ │  Model C │  │
│  │(PyTorch) │ │(TensorRT)│ │ (ONNX)   │  │
│  └──────────┘ └──────────┘ └──────────┘  │
│                                            │
│  ┌──────────────────────────────────────┐  │
│  │     Dynamic Batching Scheduler       │  │
│  └──────────────────────────────────────┘  │
│  ┌──────────────────────────────────────┐  │
│  │     Model Repository (S3/GCS/local)  │  │
│  └──────────────────────────────────────┘  │
└────────────────────────────────────────────┘

Deployment with vLLM Backend

apiVersion: apps/v1
kind: Deployment
metadata:
  name: triton-llm
spec:
  template:
    spec:
      containers:
        - name: triton
          image: nvcr.io/nvidia/tritonserver:24.05-vllm-python-py3
          args:
            - "tritonserver"
            - "--model-repository=/models"
            - "--model-control-mode=poll"
            - "--repository-poll-secs=30"
          volumeMounts:
            - name: model-repo
              mountPath: /models
          resources:
            limits:
              nvidia.com/gpu: "2"

Model Repository Structure

models/
├── llama-70b/
│   ├── config.pbtxt
│   ├── 1/
│   │   └── model.json
│   └── llama-70b-config.yaml
├── embedding-model/
│   ├── config.pbtxt
│   └── 1/
│       └── model.onnx
└── reranker/
    ├── config.pbtxt
    └── 1/
        └── model.pt

Triton Strengths

• Multi-model — serve LLM + embeddings + reranker in one server
• Multi-framework — PyTorch, TensorRT, ONNX, TensorFlow, vLLM
• Model ensembles — chain models (embed → retrieve → rerank → generate)
• Dynamic batching — framework-agnostic request batching
• Model versioning — hot-swap models without downtime
• Metrics — Prometheus metrics out of the box

Triton Limitations

• Complex configuration — config.pbtxt files for every model
• Steeper learning curve — model repository structure is rigid
• LLM throughput — slightly lower than native vLLM (abstraction overhead)
• Resource overhead — heavier base footprint

NVIDIA NIM: Enterprise Turnkey

NIM packages optimized models as containerized microservices. Pull, run, serve.

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nim-llama70b
spec:
  template:
    spec:
      containers:
        - name: nim
          image: nvcr.io/nim/meta/llama-3.1-70b-instruct:latest
          env:
            - name: NGC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ngc-secret
                  key: api-key
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: "2"

That’s it. No model download, no config files, no optimization tuning.

NIM Strengths

• Zero configuration — pre-optimized for each GPU type
• TensorRT-LLM backend — maximum inference performance
• Model profiles — automatic GPU memory/throughput optimization
• Enterprise support — NVIDIA AI Enterprise license includes SLA
• Security — signed containers, vulnerability scanning

NIM Limitations

• Proprietary — requires NVIDIA AI Enterprise license ($$$)
• Limited model selection — only NGC catalog models
• No customization — can’t modify serving behavior
• Vendor lock-in — NVIDIA GPUs only

Performance Benchmarks

Llama 3.1 70B on 2x A100 80GB, 32 concurrent requests:

NIM wins on raw performance (TensorRT-LLM optimization), vLLM wins on flexibility and cost.

Decision Framework

Choose vLLM when:

• ✅ Open-source models (Llama, Mistral, Qwen)
• ✅ Maximum flexibility and customization
• ✅ Budget-conscious (no license fees)
• ✅ Need speculative decoding or experimental features
• ✅ Self-hosted, full control

Choose Triton when:

• ✅ Serving multiple model types (LLM + embedding + classifier)
• ✅ Model ensemble pipelines (RAG, reranking)
• ✅ Mixed framework environment (ONNX + PyTorch + TensorRT)
• ✅ Need model versioning and A/B testing
• ✅ Existing NVIDIA infrastructure

Choose NIM when:

• ✅ Enterprise with NVIDIA AI Enterprise license
• ✅ Fastest time-to-production
• ✅ Team lacks ML infrastructure expertise
• ✅ Need vendor support and SLAs
• ✅ Compliance requires signed, scanned containers

Cost Comparison (Monthly, 2x A100 80GB)

For teams with ML platform engineers, vLLM is the clear cost winner. For teams without, NIM’s premium pays for itself in engineering time savings.

Related Articles

• NVIDIA NIM Support Matrix 2026
• NVIDIA Dynamo: Triton Successor for LLM Inference
• LLM Inference Cost Optimization on Kubernetes
• Autoscale AI Inference with KEDA + GPUs