At KubeCon + CloudNativeCon Japan 2026 in Yokohama, “AI on Kubernetes” was no longer a headline — it was the baseline. The interesting conversations were one level down: not whether AI runs on Kubernetes, but how you tune a cluster that was designed for stateless, CPU-bound microservices to carry workloads it was never built for. Three short clips I recorded during the keynotes captured the real trade-offs.
Not every AI workload fits as-is
One speaker made a point that sounds obvious only after someone says it out loud: some AI workloads are a great fit for the Kubernetes patterns we already have, but many are not. They need adjustments. They need new capabilities. You end up fine-tuning the platform, not just the model — because AI workloads bring three things traditional workloads don’t:
- Differentiated hardware — not every node is a vanilla container host; some carry GPUs, accelerators, or specialized silicon.
- Different traffic patterns — inference is bursty and latency-sensitive in ways batch jobs are not.
- Different security requirements — the blast radius of a compromised model server is not the same as a compromised web frontend.
The takeaway was that the CNCF project itself has a role here. The work to support AI workloads better is happening at every layer of the stack — scheduler, runtime, networking, policy — and it is being coordinated on the community channels, not inside any single vendor’s product.
A slide from the opening keynote, “What Running Inference at Scale Requires”: distributed systems, orchestration, networking, observability and scheduling.
Caching and routing beat blind load balancing
A second clip, also from the opening keynote, highlighted LLMD (llm-d), and the framing was sharp. Kubernetes plus the framework around it is a solid foundation, but the gain comes from what you build on top of it. With LLMD, because of the way it does caching and intelligent routing, you can get many times better performance out of the same models compared to just load-balancing requests independently across GPUs.
That is a bigger deal than it sounds. Most teams scaling inference think first about “how many GPUs,” then “how do I split traffic.” LLMD’s lesson is that what you cache and where you route a request matters as much as raw accelerator count. For a cloud-native stack that already prides itself on smart scheduling, extending that intelligence to model serving is exactly where the technology is headed.
The keynote slide “New Infrastructure Emerging”: llm-d for distributed LLM workloads on Kubernetes.
The true cost drivers: GPU, memory, electricity
The third clip, from the Fujitsu keynote, was the most grounded of the three. The AI platform is changing every single day, and the speaker framed the core constraint as resource cost — the resources an AI platform actually consumes:
- GPU cost — the obvious one.
- Memory cost — models, KV caches, and intermediate state add up fast.
- Electricity cost — the bill that surprises everyone once utilization climbs.
Under those constraints, the platform is expected to squeeze the maximum out of the resources a team actually has. The strategic shift the speaker described is moving from CPU-based infrastructure to GPU-centric infrastructure — not because GPUs are fashionable, but because AI platforms require it. The economics only make sense once you stop treating the accelerator as an afterthought and start designing the cluster around it.
The Fujitsu keynote on the main stage: “The Next Evolution of Kubernetes: GPU-Centric Infrastructure for AI Workloads”, by Takao Indoh.
What this means for platform teams
If you run AI on Kubernetes today, the throughline from Yokohama is clear:
- Stop assuming the defaults fit. The container-era assumptions — CPU-first, stateless, best-effort — quietly break under AI. Audit where your hardware, traffic, and security profile actually diverge.
- Treat routing and caching as a first-class tuning surface. Projects like LLMD show that intelligent placement can multiply throughput without adding a single GPU.
- Cost-model the whole stack. GPU, memory, and electricity are one system. Optimizing one while ignoring the others just moves the bill.
The hype says “put your model on Kubernetes.” The practitioners at KubeCon Japan 2026 were solving the harder, more useful problem: making Kubernetes good at AI, layer by layer.
My session and sponsor picks for the event are in the KubeCon Japan 2026 media-partner preview.
