Enterprise LLM Deployment: On-Premises

“Can we use ChatGPT?” is the question. “No, but we can deploy our own LLM” is the answer that satisfies both innovation and compliance teams.

Banking, healthcare, defense, and government organizations face a fundamental constraint: sensitive data cannot leave the network perimeter. This does not mean they cannot use AI — it means they need on-premises LLM infrastructure.

Why On-Premises

Architecture Pattern: The Enterprise LLM Stack

┌─────────────────────────────────────────────────┐
│                  Application Layer                │
│  Chatbot │ Doc Search │ Code Assistant │ Agent    │
├─────────────────────────────────────────────────┤
│                  API Gateway                      │
│  Rate limiting │ Auth │ Audit logging │ Routing   │
├─────────────────────────────────────────────────┤
│                  Inference Layer                   │
│  vLLM / NIM / TGI                                │
│  Model: Llama 3.1 70B (8× H100)                 │
│  Autoscaling: KEDA + custom metrics              │
├─────────────────────────────────────────────────┤
│                  Platform Layer                    │
│  Kubernetes │ GPU Operator │ Network Operator     │
├─────────────────────────────────────────────────┤
│                  Infrastructure                    │
│  Bare metal │ 8× H100 per node │ InfiniBand      │
│  Shared storage │ Redundant power │ HSM           │
└─────────────────────────────────────────────────┘

Model Selection for Enterprises

Not every model works on-premises. Key criteria:

Recommendation: Start with Llama 3.1 70B. Best quality-to-cost ratio, broad community support, and Meta’s community license is acceptable for most enterprise use.

Air-Gapped Deployment

True air-gapped environments have no internet access. Every dependency must be pre-staged:

Pre-Stage Model Weights

# On an internet-connected machine
huggingface-cli download meta-llama/Llama-3.1-70B-Instruct \
  --local-dir ./llama-70b-instruct

# Transfer to air-gapped environment
rsync -av ./llama-70b-instruct/ airgap-server:/models/llama-70b-instruct/

Pre-Stage Container Images

# Pull and save images
docker pull vllm/vllm-openai:latest
docker save vllm/vllm-openai:latest | gzip > vllm-openai.tar.gz

# Transfer and load on air-gapped nodes
docker load < vllm-openai.tar.gz

Deploy with Local Registry

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: vllm
          image: registry.internal:5000/vllm-openai:latest  # Local registry
          args:
            - "--model=/models/llama-70b-instruct"
            - "--tensor-parallel-size=8"
            - "--max-model-len=8192"
            - "--gpu-memory-utilization=0.9"
          resources:
            limits:
              nvidia.com/gpu: 8
          volumeMounts:
            - name: models
              mountPath: /models
              readOnly: true
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: model-storage

Security Architecture

Network Segmentation

Internet ──X── (air gap) ──X── LLM Cluster
                                    │
                                    ├── API Gateway (mTLS)
                                    │     └── Rate limiting
                                    │     └── Audit logging
                                    │     └── Token auth
                                    │
                                    ├── Inference pods (isolated VLAN)
                                    │
                                    └── Model storage (encrypted, HSM keys)

Audit Trail

Every request to the LLM must be logged:

{
  "timestamp": "2026-04-08T14:30:00Z",
  "user_id": "jsmith@corp.com",
  "model": "llama-3.1-70b-instruct",
  "prompt_tokens": 150,
  "completion_tokens": 89,
  "latency_ms": 1200,
  "classification": "internal",
  "department": "legal",
  "pii_detected": false
}

Do not log prompt content by default — that creates a secondary data liability. Log metadata for operational monitoring, and content only when explicitly required by compliance policy.

Cost Model

On-premises LLM infrastructure is a capital expenditure:

Compare to API costs: at $15/million tokens (GPT-4o), an enterprise processing 10M tokens/day spends $150/day = $54,750/year. On-premises breaks even at ~20M tokens/day.

For more detailed modeling, use the GPU Cost Calculator.

Related Resources

• NVIDIA NIM Multinode Inference
• NVIDIA Run:ai Distributed Inference
• NVIDIA GPU Operator
• FinOps for AI
• RHEL AI Tutorial
• InstructLab Fine-Tuning
• M365 Enhanced Data Encryption

About the Author

I am Luca Berton, AI and Cloud Advisor. I design on-premises AI infrastructure for regulated enterprises. Book a consultation.