At KubeCon + CloudNativeCon Japan 2026 in Yokohama, “AI on Kubernetes” was no longer a headline — it was the baseline. The interesting conversations were one level down: not whether AI runs on Kubernetes, but how you tune a cluster that was designed for stateless, CPU-bound microservices to carry workloads it was never built for. Three short interviews on the show floor captured the real trade-offs better than any keynote.
Not every AI workload fits as-is
One speaker made a point that sounds obvious only after someone says it out loud: some AI workloads are a great fit for the Kubernetes patterns we already have, but many are not. They need adjustments. They need new capabilities. You end up fine-tuning the platform, not just the model — because AI workloads bring three things traditional workloads don’t:
- Differentiated hardware — not every node is a vanilla container host; some carry GPUs, accelerators, or specialized silicon.
- Different traffic patterns — inference is bursty and latency-sensitive in ways batch jobs are not.
- Different security requirements — the blast radius of a compromised model server is not the same as a compromised web frontend.
The takeaway was that the CNCF project itself has a role here. The work to support AI workloads better is happening at every layer of the stack — scheduler, runtime, networking, policy — and it is being coordinated on the community channels, not inside any single vendor’s product.
Caching and routing beat blind load balancing
A second interview highlighted LLMD (llm-d), and the framing was sharp. Kubernetes plus the framework around it is a solid foundation, but the gain comes from what you build on top of it. With LLMD, because of the way it does caching and intelligent routing, you can get many times better performance out of the same models compared to just load-balancing requests independently across GPUs.
That is a bigger deal than it sounds. Most teams scaling inference think first about “how many GPUs,” then “how do I split traffic.” LLMD’s lesson is that what you cache and where you route a request matters as much as raw accelerator count. For a cloud-native stack that already prides itself on smart scheduling, extending that intelligence to model serving is exactly where the technology is headed.
The true cost drivers: GPU, memory, electricity
The third conversation was the most grounded of the three. The AI platform is changing every single day, and the speaker framed the core constraint as sorting cost — the resources an AI platform actually consumes:
- GPU cost — the obvious one.
- Memory cost — models, KV caches, and intermediate state add up fast.
- Electricity cost — the bill that surprises everyone once utilization climbs.
Under those constraints, the platform is expected to squeeze the maximum out of the resources a team actually has. The strategic shift the speaker described is moving from CPU-based infrastructure to GPU-centric infrastructure — not because GPUs are fashionable, but because AI platforms require it. The economics only make sense once you stop treating the accelerator as an afterthought and start designing the cluster around it.
What this means for platform teams
If you run AI on Kubernetes today, the throughline from Yokohama is clear:
- Stop assuming the defaults fit. The container-era assumptions — CPU-first, stateless, best-effort — quietly break under AI. Audit where your hardware, traffic, and security profile actually diverge.
- Treat routing and caching as a first-class tuning surface. Projects like LLMD show that intelligent placement can multiply throughput without adding a single GPU.
- Cost-model the whole stack. GPU, memory, and electricity are one system. Optimizing one while ignoring the others just moves the bill.
The hype says “put your model on Kubernetes.” The practitioners at KubeCon Japan 2026 were solving the harder, more useful problem: making Kubernetes good at AI, layer by layer.
My full talk and sponsor picks for the event are in the KubeCon Japan 2026 media-partner preview.