Cut LLM Inference Costs 70% on Kubernetes (Proven Tactics)

The Cost Problem

Running LLMs in production is expensive. A single A100 80GB costs $2-4/hour on cloud. At scale:

The good news: most teams overspend by 50-70%. Here’s how to fix that.

1. Spot/Preemptible GPUs (Save 60-70%)

AWS Spot Instances with Karpenter

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: gpu-spot
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["g5.xlarge", "g5.2xlarge", "p4d.24xlarge"]
        - key: nvidia.com/gpu
          operator: Exists
      nodeClassRef:
        name: gpu-nodes
  limits:
    nvidia.com/gpu: "16"
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 5m

Handling Spot Interruptions

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 120
      containers:
        - name: vllm
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 90"]  # Drain in-flight requests

Key insight: Inference workloads are stateless — perfect for spot. Unlike training, a killed inference pod loses nothing (no checkpoints needed). Just route traffic to remaining pods.

2. Model Quantization (Save 50% GPU Memory)

GPTQ (4-bit) vs AWQ vs GGUF

vLLM with AWQ Quantized Model

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama70b-awq
spec:
  template:
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args:
            - "--model"
            - "TheBloke/Llama-3.1-70B-AWQ"
            - "--quantization"
            - "awq"
            - "--max-model-len"
            - "4096"
            - "--gpu-memory-utilization"
            - "0.95"
          resources:
            limits:
              nvidia.com/gpu: "1"  # 70B on SINGLE A100 with AWQ!

Cost impact: Llama 70B goes from 2x A100 ($6.50/hr) to 1x A100 ($3.25/hr) = 50% savings with negligible quality loss.

3. KV-Cache Optimization

KV-cache is the #1 memory consumer during inference. Optimizing it directly reduces GPU count.

PagedAttention (vLLM)

vLLM’s PagedAttention manages KV-cache like virtual memory pages:

args:
  - "--model"
  - "meta-llama/Llama-3.1-70B"
  - "--gpu-memory-utilization"
  - "0.92"           # Use 92% of GPU memory for KV-cache
  - "--max-num-seqs"
  - "256"            # Max concurrent sequences
  - "--enable-prefix-caching"  # Reuse KV for shared prefixes

Prefix Caching (System Prompt Sharing)

If all requests share the same system prompt (common in production):

args:
  - "--enable-prefix-caching"
  # Saves 20-40% KV-cache memory when system prompts are shared

KV-Cache Compression (llm-d / Dynamo)

The latest approach: disaggregated KV-cache with compression:

┌──────────┐    ┌──────────────┐    ┌──────────┐
│ Prefill  │───▶│  KV Store    │◀───│  Decode  │
│  Node    │    │ (Compressed) │    │  Nodes   │
└──────────┘    └──────────────┘    └──────────┘

4. Request Batching (Double Throughput)

Continuous Batching

vLLM and TGI use continuous batching — new requests join the batch as soon as any sequence finishes:

args:
  - "--max-num-seqs"
  - "64"              # Batch up to 64 concurrent requests
  - "--max-num-batched-tokens"
  - "32768"           # Total tokens in a batch

Batch Size vs Latency Trade-off

Sweet spot: 32-64 batch size balances throughput and latency for most production workloads.

5. Intelligent Routing (Match Model to Task)

Not every request needs your largest model. Route by complexity:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ai-router
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      # Route based on request header
      if ($http_x_model_tier = "fast") {
        proxy_pass http://llama-8b.ai-inference:8000;
      }
      if ($http_x_model_tier = "quality") {
        proxy_pass http://llama-70b.ai-inference:8000;
      }

Cost Impact of Model Routing

Route 60% of traffic to the small model = 60% cost reduction with minimal quality impact.

6. Right-Size GPU Selection

L4 GPUs are the sweet spot for 7-8B models — 50% cheaper than A10G with better inference performance.

7. Auto-Scale to Zero (Dev/Test)

KEDA scale-to-zero eliminates idle GPU costs entirely:

spec:
  minReplicaCount: 0
  triggers:
    - type: prometheus
      metadata:
        query: sum(rate(vllm:num_requests_running[5m]))
        threshold: "1"

Dev clusters: Scale to zero outside working hours = 70% savings on GPU compute.

Total Savings Calculator

Real example: A team spending $20K/month on inference reduced to $6K/month by implementing spot + quantization + routing — 70% reduction.

Related Articles

• Autoscale AI Inference with KEDA + GPUs
• GPU Sharing on Kubernetes: MIG vs MPS vs Time-Slicing
• AI Model Serving: vLLM vs Triton vs NIM
• The Inference Gold Rush: Economics of AI Serving