Confidential Computing for AI

AI workloads process some of the most sensitive data in an organization — training data, model weights, inference inputs, and predictions. Confidential computing protects this data while it is being processed, not just at rest and in transit.

The Three States of Data Protection

The “in use” gap means that anyone with access to the host machine — cloud provider employees, compromised hypervisors, malicious co-tenants — can potentially read your data while it is being processed.

Trusted Execution Environments (TEEs)

TEEs create hardware-isolated enclaves where code and data are protected from everything outside — including the operating system, hypervisor, and cloud provider.

Available TEE Technologies

Confidential AI Architecture

┌──────────────── Confidential VM (TEE) ────────────────┐
│                                                        │
│  ┌─────────────┐  ┌─────────────┐  ┌──────────────┐  │
│  │ Encrypted   │  │ AI Model    │  │ Encrypted    │  │
│  │ Training    │→ │ (in secure  │→ │ Predictions  │  │
│  │ Data        │  │  memory)    │  │              │  │
│  └─────────────┘  └─────────────┘  └──────────────┘  │
│                                                        │
│  Memory encrypted with hardware keys                   │
│  Attestation proves code integrity                     │
└────────────────────────────────────────────────────────┘
│
│ Cloud provider / hypervisor CANNOT see inside

Kubernetes Integration

Running confidential AI workloads on Kubernetes:

apiVersion: v1
kind: Pod
metadata:
  name: confidential-inference
spec:
  runtimeClassName: kata-cc  # Confidential Containers
  containers:
    - name: inference
      image: registry.example.com/ai-model:v1
      resources:
        limits:
          nvidia.com/gpu: 1
      env:
        - name: NVIDIA_CC_MODE
          value: "on"  # Enable GPU confidential computing
      volumeMounts:
        - name: encrypted-model
          mountPath: /models
          readOnly: true
  volumes:
    - name: encrypted-model
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true

Confidential Containers (CoCo)

The CNCF Confidential Containers project brings TEE support to Kubernetes:

1. Kata Containers: Provides VM-level isolation per pod
2. Attestation: Cryptographic proof that the workload runs in a genuine TEE
3. Encrypted images: Container images decrypted only inside the TEE
4. Sealed secrets: Secrets released only to attested workloads

Use Cases for Confidential AI

1. Multi-party ML: Train models on combined datasets without any party seeing the other’s data
2. Regulated inference: Healthcare/financial AI where data must never be exposed to cloud provider
3. Model protection: Prevent model theft — weights are never accessible outside the TEE
4. Edge AI: Protect models running on untrusted hardware at the edge

Performance Considerations

Confidential computing adds overhead:

• CPU TEE: 5-15% performance overhead for memory encryption
• GPU TEE (H100): 2-5% overhead for GPU memory encryption
• Attestation: One-time cost at workload startup (seconds)
• Memory limit: TEE size limits maximum model/data size

For inference workloads, the overhead is often acceptable. For large-scale training, plan for 10-15% longer training times.

Related Reading

• Confidential Computing: Encrypted VMs and Enclaves
• Enterprise AI Security Hardening on RHEL AI
• GPU Cluster Architecture