Skip to main content
🎓 Claude Code Masterclass Learn AI-assisted development on Udemy — plus the companion book on Leanpub & Amazon. Start Learning

GPU Platform Architecture

OpenShift AI Multi-Tenant GPU Architecture

GPUs are the most expensive, most contended resource on any AI platform. Most teams either over-isolate — dedicating a GPU per team and eating the idle capacity — or under-isolate, throwing every workload into one shared pool and living with noisy neighbors, security exposure, and no way to attribute cost.

Book a Discovery Call Try the GPU Cost Calculator

Multi-tenant GPU architecture on OpenShift AI is the practice of letting several teams or applications safely share a pool of physical GPUs instead of each getting dedicated hardware. It combines the NVIDIA GPU Operator for driver and device lifecycle management, MIG partitioning or time-slicing to divide GPUs at the hardware or software level, namespace-scoped isolation and scheduling to enforce fair access, and chargeback to attribute usage back to cost centers. Luca Berton, who spoke on this exact topic at KubeCon + CloudNativeCon Europe 2026 in Amsterdam ("Lessons Learned Orchestrating Multi-Tenant GPUs on OpenShift AI with NVIDIA KAI (G/H200)"), designs these platforms for enterprises moving AI workloads into production.

Why multi-tenancy matters

A single H100 costs more per hour than most of the rest of a Kubernetes node combined, and GPU supply is almost always the tightest constraint on an AI platform — tighter than CPU, memory, or storage. That scarcity forces a choice. Give every team its own dedicated GPUs and you get predictable performance but pay for capacity that sits idle outside business hours, during off-peak traffic, or between training runs. Put every workload into one shared pool with no isolation and utilization goes up, but so does risk: one tenant's runaway batch job starves another's production inference, a memory leak in one container can corrupt another's compute context, and finance has no reliable way to say which team's usage drove the bill.

Without multi-tenancy

  • ❌ Dedicated GPUs per team sit idle 40-70% of the time
  • ❌ Shared pools with no isolation create noisy-neighbor incidents
  • ❌ No security boundary between tenant memory and compute
  • ❌ Cost is a single line item — no way to attribute spend by team

With a multi-tenant platform

  • ✓ MIG and time-slicing raise utilization without buying more GPUs
  • ✓ Namespace isolation and quotas remove noisy-neighbor risk
  • ✓ Hardware-level memory isolation where it matters (MIG)
  • ✓ Chargeback turns raw usage into a per-team cost report

Core building blocks

The pieces that make GPU multi-tenancy work in practice on OpenShift AI.

⚙️

NVIDIA GPU Operator

Manages the full driver, CUDA toolkit, and device plugin lifecycle as Kubernetes-native resources. It's what lets OpenShift AI expose MIG profiles and time-sliced GPUs as schedulable objects instead of hand-configured host state.

🔪

MIG partitioning

Hardware-level slicing available on A100, H100, and newer data-center GPUs. Splits one physical card into up to seven instances with dedicated compute and memory — true isolation, not just scheduling.

🕐

Time-slicing

Software-level sharing that lets multiple pods take turns on the same GPU's streaming multiprocessors. No memory isolation, but works on any GPU generation and is ideal for bursty or low-utilization dev workloads.

🚧

Namespace isolation & scheduling

ResourceQuota and LimitRange constrain how many GPU resources a namespace can request. Combined with node selectors and taints, this keeps tenant workloads on the right MIG or time-sliced node pool.

💳

Chargeback / showback

Tags GPU usage by namespace or label and rolls it up into a cost-per-team report. Turns "the GPU bill went up" into "team X's inference workload drove 60% of last month's spend."

📈

Monitoring

The DCGM exporter feeds per-GPU, per-MIG-instance metrics into Prometheus, visualized in Grafana. This is the data layer that makes both isolation problems and chargeback numbers visible.

MIG vs time-slicing vs full-GPU passthrough

ApproachIsolation levelGranularityBest forOverhead
MIG partitioningHardware-level (separate compute + memory slices)Fixed profiles (e.g. 1g.5gb up to 7g.40gb on A100)Predictable multi-team production workloadsNone (native silicon feature, A100/H100 class only)
Time-slicingSoftware-level (shared SM, no memory isolation)Configurable time-slice count per GPUDev/test, bursty or low-utilization workloadsContext-switch overhead, no hard isolation
Full-GPU passthroughFull isolation (one tenant = one GPU)Whole GPU onlyLatency-sensitive production inference, trainingLowest utilization, highest cost per tenant

Most production platforms combine all three — MIG for steady-state serving, time-slicing for dev namespaces, and full passthrough for training jobs that need every streaming multiprocessor on the card.

Isolation and security considerations

  • Namespace-scoped ResourceQuota and device plugin limits. Every tenant namespace should have an explicit cap on GPU, MIG-profile, and time-slice resource requests — without it, one team's deployment can silently consume capacity another team was counting on.
  • PodSecurity and SCC constraints. GPU workloads often want elevated privileges for driver access; OpenShift's Security Context Constraints should scope that access as narrowly as possible per tenant namespace, not grant it cluster-wide.
  • Network policies between tenant namespaces. GPU isolation solves compute and memory boundaries, not network exposure — tenant namespaces still need explicit NetworkPolicy rules to stop lateral movement between workloads sharing a node.
  • "Isolation" means more than a Kubernetes namespace. A namespace boundary controls scheduling and RBAC, but it says nothing about what happens on the GPU silicon itself. Two pods in separate namespaces, time-sliced onto the same physical GPU, can still share device memory address space at the hardware level.
  • Memory isolation is only guaranteed with MIG. Time-slicing shares the same GPU memory across all tenants on that device — a workload with a memory leak, or one deliberately probing for leftover data, can potentially observe another tenant's memory contents. If tenants have real security requirements (regulated data, competitive workloads, untrusted code), MIG or full-GPU passthrough are the only options that provide a hardware-enforced boundary.
  • Treat GPU isolation policy as a data-sensitivity decision, not just a capacity one. Decide per workload whether the requirement is "don't starve each other" (time-slicing is fine) or "must not be able to see each other's data" (MIG or passthrough required) before allocating.

Illustrative scenario

This is an illustrative example to show the reasoning process — not a specific client engagement.

Picture an 8-node H100 cluster on OpenShift AI, shared by 5 internal teams: two production inference teams, two fine-tuning/experimentation teams, and one shared dev/test group. A reasonable allocation might look like this:

  • Production inference teams (2): MIG profiles on a dedicated set of nodes, sized to each team's expected request volume — for example a 3g.40gb slice for the higher-traffic team and a 2g.20gb slice for the other. Hardware isolation gives both teams a predictable SLA regardless of what else is running on the card.
  • Shared dev/test namespace: Time-sliced GPUs on a separate node pool, configured for 4-way sharing. Dev workloads are bursty and rarely latency-sensitive, so the context-switch overhead is an acceptable tradeoff for far higher utilization than dedicating hardware to a namespace that's idle most of the day.
  • Fine-tuning / experimentation teams (2): Full-GPU access on a rotating schedule — each team gets scheduled windows with exclusive access to a subset of nodes for training runs that need every SM on the card, then releases the nodes back to the pool when the run completes.

The result: five teams sharing eight nodes instead of each needing dedicated hardware, with isolation guarantees matched to what each workload actually requires.

Monitoring and chargeback

None of this works without visibility. The DCGM exporter runs alongside the GPU Operator and exposes per-GPU and per-MIG-instance metrics — utilization, memory usage, temperature, ECC errors — into Prometheus. Grafana dashboards built on top turn that into per-namespace, per-team views: which tenant is actually using their allocated MIG slice, which time-sliced namespace is starving its neighbors, which full-GPU allocation sat idle overnight.

Chargeback (or showback, if you're not billing internal teams directly) takes those raw utilization metrics and rolls them into a cost-per-team report — GPU-hours consumed, tagged by namespace or label, priced against the platform's actual per-GPU cost. That's the mechanism that closes the loop between "we bought this hardware" and "here's what each team's usage cost."

This is also the lever behind the 30% average cloud cost reduction we typically see on platforms we help design: utilization visibility is what lets teams right-size their allocation instead of over-provisioning "just in case." Without per-tenant metrics, most organizations default to buying more GPUs; with them, the same workloads often fit on less hardware.

Frequently asked questions

What is MIG partitioning?
Multi-Instance GPU (MIG) is a hardware feature on NVIDIA A100, H100, and newer data-center GPUs that splits a single physical GPU into up to seven isolated instances, each with dedicated compute cores, memory, and cache bandwidth. Because the isolation happens in silicon, one MIG instance cannot see or interfere with another's memory or compute, which makes it the only GPU-sharing approach that gives true hardware-level tenant isolation on OpenShift AI.
Can you mix MIG and time-slicing on the same cluster?
Yes. The NVIDIA GPU Operator lets you apply different configurations per node or per node pool, so it's common to run MIG-partitioned nodes for production inference and time-sliced nodes for development namespaces within the same OpenShift AI cluster. What you cannot do is mix MIG and time-slicing on the exact same physical GPU at the exact same time — a given GPU is configured one way or the other.
How many teams can share one GPU?
With MIG on an A100 or H100, up to seven teams can share one physical GPU with hardware isolation, using profiles like 1g.5gb through 7g.40gb. With time-slicing there is no hardware limit — the device plugin can expose a GPU as an arbitrary number of shareable replicas — but every additional slice increases context-switch overhead and reduces the effective throughput available to each tenant, so most platforms cap replicas at 2-4 for meaningful workloads.
Does OpenShift AI support NVIDIA multi-instance GPU natively?
Yes. Red Hat OpenShift AI integrates the NVIDIA GPU Operator, which manages MIG configuration, drivers, and the device plugin as first-class cluster resources. MIG profiles are exposed as schedulable Kubernetes resources, so workloads request them the same way they would request a full GPU, just with a MIG-specific resource name and profile size.
What is the difference between GPU scheduling and GPU isolation?
GPU scheduling is about which pod gets access to a GPU resource and when — handled by Kubernetes' scheduler, resource quotas, and the device plugin. GPU isolation is about what happens once two workloads are running on the same physical card — whether one tenant's process can see, starve, or corrupt another's memory and compute. Namespaces and ResourceQuota solve scheduling; only MIG (not time-slicing, and not namespaces alone) solves isolation at the memory level.

Related pillar pages

GPU Cost Optimization

Turn utilization visibility into a lower monthly GPU bill.

AI Infrastructure for Regulated Enterprises

Governance and compliance patterns for AI platforms in finance, healthcare, and public sector.

Enterprise MLOps Governance

Model registry, approval workflow, and audit-ready evidence for production AI.

Related Articles

Platform Engineering

PlatformCon 2026: Multi-Tenant GPUs on OpenShift AI

Lessons orchestrating multi-tenant GPUs on OpenShift AI with NVIDIA KAI: GPU sharing, workload isolation, scheduling efficiency, and cost control.

AI

Run:ai Topology-Aware Scheduling for GPU Workloads (2026)

Run:ai topology-aware scheduling places GPU workloads on nodes close in the network hierarchy. Configure topology labels and preferred or required modes.

AI

NVIDIA Dynamo: Why It Replaces Triton for LLM Serving

NVIDIA Dynamo is the open-source successor to Triton. Disaggregated prefill/decode, KV-cache-aware routing, and 2-3x throughput gains for vLLM deployments.

Written by Luca Berton — AI & Cloud Advisor, Docker Captain, former Red Hat engineer. Speaker at KubeCon + CloudNativeCon Europe 2026 on multi-tenant GPUs on OpenShift AI, and Red Hat Summit 2026. More about Luca →

Ready to design your multi-tenant GPU platform?

30-minute discovery call. We look at your GPU utilization, tenant isolation requirements, and identify the fastest path to a platform teams can safely share.

Book a Free Call

Free 30-min AI & Cloud consultation

Book Now