A local Kubernetes cluster can carry a lot more history than the minikube binary on your laptop suggests.
I hit exactly that while starting Minikube v1.39.0 on Apple Silicon macOS. An existing QEMU-backed profile still came up with Kubernetes v1.28.3, but trying to move it to Kubernetes v1.37.0 failed during kubeadm init.
The final fix was simple once the logs were read in the right order: delete the stale QEMU cluster and recreate Minikube with the Docker driver.
Here is the full troubleshooting path and why that worked.
The First Clue: New Minikube Binary, Old VM
The original cluster could still start:
minikube v1.39.0 on Darwin arm64
Using the qemu2 driver based on existing profile
Image was not built for the current minikube version.
Expected minikube version: v1.32.0
Actual minikube version: v1.39.0
Preparing Kubernetes v1.28.3 on Docker 24.0.7
Done! kubectl is now configured to use "minikube" clusterThat warning is easy to dismiss because Minikube continues starting.
The node even looked healthy:
NAME STATUS ROLES VERSION
minikube Ready control-plane v1.28.3But the important detail was this:
Expected minikube version: v1.32.0 -> Actual minikube version: v1.39.0The executable on macOS was current. The VM behind the profile was not.
Updating Minikube does not automatically rebuild an existing VM profile, so the cluster was still carrying an older guest kernel, container runtime, and cgroup setup.
Forcing Kubernetes 1.37 Exposed the Real Problem
I stopped the old cluster and explicitly requested Kubernetes 1.37:
minikube stop
minikube start --kubernetes-version=v1.37.0Minikube downloaded the new preload and tried to initialize the control plane, but kubeadm failed its preflight checks.
The guest reported:
CONFIG_OVERLAY_FS: enabled (as module)
CONFIG_AUFS_FS: not set - Required for aufs.
CONFIG_BLK_DEV_DM: enabled (as module)
CONFIG_CFS_BANDWIDTH: enabled
CONFIG_SECCOMP: enabled
CONFIG_SECCOMP_FILTER: enabled
OS: LinuxThe cgroup controllers were also clearly the cgroup v1 layout:
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_PIDS: enabled
CGROUPS_HUGETLB: enabled
CGROUPS_BLKIO: enabledThen came the fatal line:
[ERROR SystemVerification]: cgroups v1 support is deprecated and will be removed in a future release.
Please migrate to cgroups v2.
To explicitly enable cgroups v1 support for kubelet v1.35 or newer,
you must set the kubelet configuration option 'FailCgroupV1' to 'false'.
You must also explicitly skip this validation.That was the actual blocker.
Kubernetes has been moving away from cgroup v1, and the kubelet’s failCgroupV1 setting defaults to true for Kubernetes 1.35 and newer. Kubernetes 1.37 therefore refuses to initialize on this old cgroup v1 guest unless you deliberately override the protection.
For a disposable development cluster, carrying that override forward was not the direction I wanted.
See the official Kubernetes 1.37 release notes for the current cgroup v1 guidance.
The CRI RuntimeConfig Warning Was Not the Fatal Error
The same failed startup included another warning:
RuntimeConfig from runtime service failed
rpc error: code = Unimplemented desc = unknown method RuntimeConfig
for service runtime.v1.RuntimeServiceFollowed by:
You must update your container runtime to a version that supports the CRI method RuntimeConfig.
Falling back to using cgroupDriver from kubelet config will be removed in 1.38.There was also:
[WARNING Service-kubelet]: kubelet service is not enabledThose warnings matter, but neither was the line that stopped this run.
The first fatal preflight error was the cgroup v1 SystemVerification failure.
That distinction is useful when a Minikube start produces hundreds of lines: find the first fatal kubeadm error before chasing every warning.
Root Cause: A Stale Cluster Environment
The failure was really a version-skew problem across several layers:
- The host was running Minikube v1.39.0.
- The existing QEMU VM had been created for Minikube v1.32.0.
- That guest was still using cgroup v1.
- Kubernetes v1.37.0 rejected that cgroup configuration during
kubeadm init. - The old container runtime also lacked the newer CRI
RuntimeConfigmethod expected by current Kubernetes tooling.
This is why repeatedly adding --kubernetes-version=v1.37.0 could not solve the real problem. The control plane version was only one piece of the environment.
One nuance is worth making explicit: QEMU itself is not inherently limited to cgroup v1. The problem here was the old Minikube VM being reused.
Check the Host Before Choosing a Replacement Driver
Before deleting the cluster, I checked which container or VM backends were actually available on the Mac.
A simple probe is enough:
echo "--- docker ---"
command -v docker >/dev/null && docker info >/dev/null 2>&1 \
&& echo "docker OK" || echo "no working docker"
echo "--- colima ---"
command -v colima >/dev/null && colima status 2>&1 || echo "no colima"
echo "--- podman ---"
command -v podman >/dev/null && podman info >/dev/null 2>&1 \
&& echo "podman OK" || echo "no working podman"
echo "--- vfkit ---"
command -v vfkit >/dev/null && echo "vfkit installed" || echo "no vfkit"On this machine, Docker was already working.
That made the cleanest fix obvious: stop trying to repair the stale QEMU guest and recreate the local cluster using the Docker driver.
The Fix That Worked: Delete QEMU and Recreate with Docker
Because this was a disposable lab cluster, I removed the old profile completely:
minikube deletedestroys the local cluster and workloads stored inside it. Back up anything you need first.
minikube deleteThen I created a fresh Kubernetes 1.37 cluster using Docker:
minikube start \
--driver=docker \
--kubernetes-version=v1.37.0This time the cluster came up cleanly.
The important difference is that Minikube was no longer trying to boot the stale VM image built for v1.32.0. It created the cluster against the currently working Docker environment instead.
If Docker is going to be the default Minikube backend on the workstation, you can persist that choice:
minikube config set driver dockerSee the official Minikube Docker driver documentation.
Verify the Node Before Reinstalling Addons
After recreating the cluster, I verified the node first:
kubectl get nodesThe node reported Ready.
I also checked Minikube itself:
minikube statusFor deeper verification, inspect the cgroup filesystem inside the new node:
minikube ssh -- 'stat -fc %T /sys/fs/cgroup'A cgroup v2 environment reports:
cgroup2fsYou can also inspect the mounts directly:
minikube ssh -- 'mount | grep cgroup'This is a better validation than treating a successful minikube start as proof that every compatibility issue is gone.
Re-enable Ingress
With the new cluster healthy, I restored the ingress addon:
minikube addons enable ingressThen checked the controller:
kubectl get pods -n ingress-nginxThe ingress controller pod reached Running. It was still completing its startup probe at first, which is normal immediately after enabling the addon.
At this point the local cluster was fixed and ready for the hello-ingress.yaml workload from the ingress lab.
Important: Docker Driver Networking on macOS
Switching from a VM driver to Docker changes how the host reaches the cluster.
With the Docker driver on macOS, the Minikube node IP is not directly reachable from the host in the same way as a traditional VM-backed Minikube node.
For the standard Ingress workflow on Docker Desktop, run this in another terminal:
minikube tunnelThen access the Ingress through 127.0.0.1 rather than assuming minikube ip is directly routable from macOS.
The current Minikube getting-started documentation explicitly calls this out for Docker Desktop users: Minikube start / Ingress example.
For a service exposed as NodePort, Minikube also provides:
minikube service <service-name> --urlThat command creates a host-accessible tunnel and prints the URL. See Accessing apps in Minikube.
So the mental model after switching drivers is:
- Ingress on Docker Desktop/macOS: use
minikube tunneland typically127.0.0.1. - NodePort service: use
minikube service <service-name> --urlwhen you want a convenient host URL. - Do not assume the Docker driver’s
minikube ipbehaves like the old VM driver’s IP from the macOS host.
What About failCgroupV1: false?
Kubernetes 1.37 still allows an explicit temporary override:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failCgroupV1: falseI did not use it.
For a local Minikube cluster that can be destroyed and recreated, rebuilding on a current environment is cleaner than extending the life of cgroup v1.
The override makes more sense when you have a real migration constraint and need a short-lived bridge. It is not a great default for a development cluster whose underlying VM is already stale.
The Complete Recovery Sequence
The successful path on this Mac boiled down to:
# Confirm Docker is healthy
docker info
# Remove the stale qemu2-backed profile
minikube delete
# Recreate Kubernetes 1.37 using Docker
minikube start \
--driver=docker \
--kubernetes-version=v1.37.0
# Verify the node
kubectl get nodes
minikube status
# Restore ingress
minikube addons enable ingress
kubectl get pods -n ingress-nginx
# Keep this running in another terminal for Ingress access on macOS
minikube tunnelThat removed the stale VM image, avoided the old cgroup v1 guest, and restored a working Kubernetes 1.37 local environment.
The Practical Lesson
The most valuable line in the original log was not the final Minikube failure banner. It was this earlier warning:
Image was not built for the current minikube version.
Expected minikube version: v1.32.0 -> Actual minikube version: v1.39.0A local Kubernetes cluster is still infrastructure. Its VM image, kernel, cgroup hierarchy, container runtime, kubelet, kubeadm, and Kubernetes version all have a compatibility relationship.
When a profile has lived through years of upgrades, updating only the host CLI can leave you in a misleading state: the command on your Mac is current, but the environment it controls is not.
My troubleshooting order for this class of failure is now:
- Read the first fatal kubeadm error.
- Look for warnings that the Minikube profile or VM was created by an older release.
- Check the node’s cgroup version.
- Separate fatal errors from secondary runtime warnings.
- Check which current Minikube driver actually works on the host.
- For a disposable cluster, recreate it instead of forcing a new Kubernetes version into stale infrastructure.
- After switching drivers, re-check how services and Ingress are reached from the host.
In this case, Docker was already healthy, so deleting the old QEMU cluster and starting fresh with --driver=docker was the shortest route from a broken Kubernetes 1.37 upgrade to a Ready node and a running ingress controller.