Somewhere between the second and third day of Red Hat Summit 2026, I stopped grabbing quotes about products and started grabbing quotes about people who build the plumbing everyone else depends on. Sally O’Malley is one of those people. She is a maintainer of llm-d, the CNCF sandbox project for distributed LLM inference, and I pulled her aside on the show floor in Atlanta to talk about what her week actually looked like.
The short version: while most of the hall was talking about which model to use, Sally was talking about how to run any model, on any accelerator, reliably, at production scale — with genuine excitement in her voice about how fast that conversation is moving.
Maintaining a Project That Won’t Sit Still
llm-d sits on top of inference engines like vLLM and SGLang, adding what those engines do not provide on their own: multi-node serving, LLM-aware load balancing, distributed KV caching, and autoscaling. I covered the technical depth of that cache-aware routing layer in my write-up of the Community Central Theater talk — the 10x cost swing between cached and uncached tokens is the number that sticks. Sitting across from a maintainer gives a different picture than a stage talk does: the day-to-day reality of keeping a CNCF sandbox project coherent while IBM Research, Google, Red Hat, CoreWeave, and NVIDIA all land code in the same repos at once. That is a genuinely hard job — not because any contributor is difficult, but because the underlying inference engines never slow down. vLLM ships changes, SGLang ships changes, new accelerators show up, and the orchestration layer has to absorb it all without breaking what already-adopted teams depend on.
Enterprise Was the Whole Point of Her Week
Sally was blunt about what she was there to do: her focus for Summit was enterprise use cases, full stop. Not the next feature branch, not a roadmap slide — how llm-d actually behaves wired into a real production AI platform carrying real traffic. That is a different job than writing the code — it means sitting with platform teams who care less about an elegant scheduler design and more about whether P99 latency holds under multi-tenant load and whether the GPU bill goes down. It tracked with something else I heard on the floor this week from Pete Cheslock: a year ago, most enterprise teams were still doing RAG experiments; this year, the same teams are running their own inference and training their own models. Sally’s conversations were the natural next step in that curve — teams already committed to running inference in-house now need the orchestration layer mature enough to trust with real production traffic.
Why Vendor Neutrality Is a Strategic Bet, Not a Checkbox
The booth signage behind Sally made the point plainly: llm-d is vendor-neutral and engine-agnostic, spanning NVIDIA, AMD, Google TPU, and Intel HPU backends. That reads like a routine interoperability checkbox. It is not. The accelerator market is genuinely unsettled — NVIDIA’s dominance is real, but AMD’s Instinct line, Google’s TPUs, and Intel’s HPU are all fighting for the same inference workloads, with pricing, availability, and performance-per-dollar shifting underfoot. A platform team that builds its inference stack directly against one vendor’s SDK is making a multi-year bet on that vendor’s roadmap and supply chain. Building on llm-d instead keeps the hard, expensive-to-rebuild part — orchestration and routing logic — constant while the hardware underneath gets swapped or renegotiated as the market moves. That is the same hybrid-cloud logic Red Hat has sold for two decades, applied one layer down, to the GPU itself. Turkish Airlines scaling 60+ models under the banner “keep your options open” and llm-d’s cross-vendor story are, structurally, the same pitch.
Kubernetes-Native by Design, Not by Afterthought
The other detail worth sitting with is how deliberately llm-d leans on existing Kubernetes primitives rather than inventing its own control plane: Gateway API for routing, Custom Resource Definitions for declaring inference topology, and standard Horizontal Pod Autoscaler hooks for scaling. That is a real operator-facing decision, not a footnote. A platform team that already knows how to manage Gateway API resources, write CRDs, and tune HPA policies for other workloads does not need a second, parallel skill set just to run inference — the observability, RBAC, GitOps, and incident tooling they already trust also understands llm-d’s objects. Paired with its composable architecture, where teams can adopt cache-aware routing without also adopting disaggregated prefill/decode serving, the design lowers the bar for a first production deployment: adopt the piece that solves your current bottleneck, grow from there. For a sandbox project barely past its first year, that incremental adoption path is exactly what turns curious platform teams into committed contributors — which, from a maintainer’s chair, is the whole game.
Related Reading
- Red Hat Summit 2026 in Atlanta: Open Source Meets AI
- llm-d at Red Hat Summit 2026: KV-Cache Aware Routing for vLLM
- llm-d Joins the CNCF: Kubernetes-Native Distributed LLM Inference
- vLLM Inference Optimization on RHEL AI
- OpenBao Founder on Secrets Management for AI Agents
About the Author
I am Luca Berton, AI and Cloud Advisor. I work at the intersection of platform engineering, cloud security, and enterprise AI deployments. Book a consultation.
