GPU Hardware Selection Guide for RHEL AI

📘 Book Reference: This article is based on Chapter 2: Setup and Chapter 4: Advanced Features of Practical RHEL AI, covering GPU hardware requirements and optimization strategies.

Introduction

Choosing the right GPU for your RHEL AI deployment is one of the most consequential infrastructure decisions you’ll make. The wrong choice can mean wasted budget, performance bottlenecks, or inability to run your target models.

Practical RHEL AI provides detailed hardware guidance that I’ll summarize here, helping you match GPU capabilities to your specific workload requirements.

Supported GPU Hardware

RHEL AI officially supports:

NVIDIA A100: The Proven Workhorse

Specifications

Architecture: Ampere
CUDA Cores: 6,912
Tensor Cores: 432 (3rd Gen)
Memory: 40GB or 80GB HBM2e
Memory Bandwidth: 2 TB/s
TDP: 400W

When to Choose A100

✅ Ideal for:

• Mixed training and inference workloads
• Models up to 13B parameters (single GPU)
• Budget-conscious deployments
• Established software ecosystem

❌ Limitations:

• Older FP8 support (limited)
• Lower memory bandwidth than H100
• Cannot run 70B+ models efficiently

RHEL AI Configuration

# Verify A100 detection
nvidia-smi --query-gpu=name,memory.total,compute_cap --format=csv

# Expected output:
# NVIDIA A100-SXM4-80GB, 81920 MiB, 8.0

# Optimal vLLM settings for A100
python -m vllm.entrypoints.openai.api_server \
  --model granite-7b-instruct \
  --dtype float16 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 8192

NVIDIA H100: Maximum Performance

Specifications

Architecture: Hopper
CUDA Cores: 16,896
Tensor Cores: 528 (4th Gen)
Memory: 80GB HBM3
Memory Bandwidth: 3.35 TB/s
TDP: 700W
FP8 Tensor Core Performance: 3,958 TFLOPS

When to Choose H100

✅ Ideal for:

• Large model training (34B+ parameters)
• Latency-critical inference
• FP8 inference optimization
• Multi-node distributed training

❌ Limitations:

• Higher cost (~2.5x A100 pricing)
• Higher power requirements
• May be overkill for smaller models

Performance Comparison

H100 with FP8 Inference

from vllm import LLM, SamplingParams

# H100 with FP8 quantization
llm = LLM(
    model="granite-34b-instruct",
    dtype="float8_e4m3fn",
    quantization="fp8",
    tensor_parallel_size=2,
    enforce_eager=False  # Use CUDA graphs
)

# 50% memory reduction, minimal accuracy loss

AMD MI300X: Memory Champion

Specifications

Architecture: CDNA 3
Compute Units: 304
Memory: 192GB HBM3
Memory Bandwidth: 5.3 TB/s
TDP: 750W

When to Choose MI300X

✅ Ideal for:

• Memory-bound workloads
• Running 70B+ models on single GPU
• Avoiding tensor parallelism complexity
• ROCm-compatible workflows

❌ Limitations:

• Smaller software ecosystem than NVIDIA
• Some CUDA libraries need porting
• Fewer cloud availability options

RHEL AI with ROCm

# Install AMD GPU support
sudo dnf install -y rocm-hip-runtime rocm-hip-sdk

# Verify MI300X detection
rocm-smi --showproductname

# vLLM with ROCm backend
python -m vllm.entrypoints.openai.api_server \
  --model granite-70b-instruct \
  --dtype float16 \
  --device rocm

Memory Requirements by Model Size

Memory Calculation Formula

def estimate_gpu_memory(params_billions, dtype_bytes=2, overhead=1.2):
    """
    Estimate GPU memory required for inference.
    
    Args:
        params_billions: Model parameters in billions
        dtype_bytes: 2 for FP16, 1 for INT8, 0.5 for INT4
        overhead: KV cache and activation memory (1.2 = 20% overhead)
    
    Returns:
        Required GPU memory in GB
    """
    base_memory = params_billions * dtype_bytes
    return base_memory * overhead

# Examples:
# 7B FP16: 7 * 2 * 1.2 = 16.8 GB
# 70B FP16: 70 * 2 * 1.2 = 168 GB
# 70B INT8: 70 * 1 * 1.2 = 84 GB

Multi-GPU Configurations

Training Configurations

NVLink vs PCIe

NVLink 4.0 (H100): 900 GB/s bidirectional
NVLink 3.0 (A100): 600 GB/s bidirectional
PCIe 5.0: 128 GB/s bidirectional
PCIe 4.0: 64 GB/s bidirectional

Recommendation: Always use NVLink for multi-GPU training

Cost Analysis

On-Premises (3-Year TCO)

Cloud Hourly Rates (Approximate)

Decision Framework

START
  │
  ├─ Budget constrained?
  │   YES → A100 80GB
  │   NO ↓
  │
  ├─ Model size > 34B parameters?
  │   YES → H100 or MI300X
  │   NO ↓
  │
  ├─ Training large models?
  │   YES → H100 (speed matters)
  │   NO ↓
  │
  ├─ Inference-only workload?
  │   YES → A100 (cost-effective)
  │   NO ↓
  │
  └─ Mixed workload → H100 (future-proof)

RHEL AI Hardware Validation

# Run RHEL AI hardware validation
rhel-ai-validate --check hardware

# Expected output:
# GPU detected: NVIDIA H100 80GB
# Driver version: 535.129.03
# CUDA version: 12.4
# Memory: 80GB HBM3
# NVLink: Connected (4 links)
# PCIe: Gen5 x16
# 
# Hardware validation: PASSED

Related Book Content

This article covers material from:

• Chapter 2: Setup - Hardware requirements and validation
• Chapter 4: Advanced Features - GPU optimization techniques
• Chapter 7: Use Cases - Workload-specific recommendations

Complete Hardware Planning Guide

Need help sizing your RHEL AI infrastructure?

Practical RHEL AI includes comprehensive hardware planning resources:

• ✅ Detailed benchmark data for all supported GPUs
• ✅ Workload sizing calculators
• ✅ Cloud vs on-premises cost models
• ✅ Power and cooling requirements
• ✅ Procurement checklists and vendor guidance