I am trying to provision a cluster in VSphere but it stalls in Scaling up and WaitingForAvailableMachines

[hatef94]

• clusterctl version: v1.9.6
• kind version: v0.26.0
• Vsphere ESXI: v8.0.x

Hi,
I have a Kind cluster in my workstation. I want to provision a cluster in my Vsphere cluster that contains 3 nodes.

First I installed the prerequisites such as

• Kind
• clusterctl
• clusterctl init --infrastructure=vsphere
• set up environment variables such as VSPHERE_SERVER, VSPHERE_USER and VSPHERE_PASSWORD
• I've got NSX enabled

Then I create a manifest using the following command:

clusterctl --config /home/hatef/.config/cluster-api/clusterctl.yaml  generate cluster my-cluster \
  --infrastructure vsphere \
  --control-plane-machine-count=1\
  --worker-machine-count=1\
  > my-cluster.yaml

And my configuration in /home/hatef/.config/cluster-api/clusterctl.yaml

CONTROL_PLANE_ENDPOINT_IP: "192.168.230.25"
VSPHERE_DATACENTER: "Armanlab-Datacenter"
VSPHERE_SSH_AUTHORIZED_KEY: "ssh-ed25519 AApassss hatef@hatef-ASUS-EXPER"
KUBERNETES_VERSION: "v1.32.0"
VIP_NETWORK_INTERFACE: "eth0"
VSPHERE_FOLDER: ""
VSPHERE_DATASTORE: "H35-DS-01"
VSPHERE_NETWORK: "VM Network"
VSPHERE_PASSWORD: "password"
VSPHERE_RESOURCE_POOL: ""
VSPHERE_SERVER: "192.168.230.40"
VSPHERE_USERNAME: "[email protected]"
VSPHERE_TEMPLATE: "ubuntu-2404-kube-v1.32.0"
CONTROL_PLANE_MACHINE_COUNT: "1"
WORKER_MACHINE_COUNT: "1"
EXP_CLUSTER_RESOURCE_SET: "true"
EXP_MACHINE_POOL: "true"
VSPHERE_STORAGE_POLICY: ""
VSPHERE_TLS_THUMBPRINT: ""
"CPI_IMAGE_K8S_VERSION": "v1.32.0"  

Next I applied the my-cluster.yaml using kubectl

here is some output:

kubectl get cluster my-cluster 
NAME         CLUSTERCLASS   PHASE         AGE   VERSION
my-cluster                  Provisioned   61m

The status of cluster doesn't change:

clusterctl describe cluster my-cluster
NAME                                                 READY  SEVERITY  REASON                           SINCE  MESSAGE                                                       
Cluster/my-cluster                                   False  Info      WaitingForKubeadmInit            88s                                                                   
├─ClusterInfrastructure - VSphereCluster/my-cluster  True                                              2m48s                                                                 
├─ControlPlane - KubeadmControlPlane/my-cluster      False  Info      WaitingForKubeadmInit            88s                                                                   
│ └─Machine/my-cluster-tvz7l                         True                                              88s                                                                   
└─Workers                                                                                                                                                                    
  └─MachineDeployment/my-cluster-md-0                False  Warning   WaitingForAvailableMachines      2m48s  Minimum availability requires 1 replicas, current 0 available  
    └─Machine/my-cluster-md-0-pgxvs-2vpcd            False  Info      WaitingForControlPlaneAvailable  2m33s  0 of 2 completed

Some logs that may be useful:
kubectl logs -n capd-system pods/capd-controller-manager-59b9dff94f-drqxs

I0419 07:07:16.332543       1 cluster_accessor.go:252] "Connecting" controller="clustercache" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="default/my-cluster" namespace="default" name="my-cluster" reconcileID="160adcdb-18c6-48a9-93f1-aa36020587ad"
E0419 07:07:21.334845       1 cluster_accessor.go:262] "Connect failed" err="error creating HTTP client and mapper: cluster is not reachable: Get \"https://192.168.230.25:6443/?timeout=5s\": context deadline exceeded" controller="clustercache" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="default/my-cluster" namespace="default" name="my-cluster" reconcileID="160adcdb-18c6-48a9-93f1-aa36020587ad"

kubectl logs -n capi-kubeadm-bootstrap-system pods/capi-kubeadm-bootstrap-controller-manager-688fd8c888-4zm5p

I0419 07:11:56.360209       1 cluster_accessor.go:252] "Connecting" controller="clustercache" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="default/my-cluster" namespace="default" name="my-cluster" reconcileID="97523224-679c-40d9-a5e2-01747a2c8121"
E0419 07:12:01.362687       1 cluster_accessor.go:262] "Connect failed" err="error creating HTTP client and mapper: cluster is not reachable: Get \"https://192.168.230.25:6443/?timeout=5s\": context deadline exceeded" controller="clustercache" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="default/my-cluster" namespace="default" name="my-cluster" reconcileID="97523224-679c-40d9-a5e2-01747a2c8121"

If I ssh into my-cluster-tvz7l it couldn't run cloud-init script successfully
{ "boot_status_code": "enabled-by-generator", "datasource": "vmware", "detail": "DataSourceVMware [seed=guestinfo]", "errors": [ "('scripts_user', RuntimeError('Runparts: 1 failures (runcmd) in 1 attempted commands'))" ], "extended_status": "error - done", "init": { "errors": [], "finished": 10.47, "recoverable_errors": {}, "start": 7.97 }, "init-local": { "errors": [], "finished": 6.1, "recoverable_errors": {}, "start": 5.35 }, "last_update": "Thu, 01 Jan 1970 00:04:17 +0000", "modules-config": { "errors": [], "finished": 12.57, "recoverable_errors": {}, "start": 12.37 }, "modules-final": { "errors": [ "('scripts_user', RuntimeError('Runparts: 1 failures (runcmd) in 1 attempted commands'))" ], "finished": 257.37, "recoverable_errors": { "WARNING": [ "Failed to run module scripts_user (scripts in /var/lib/cloud/instance/scripts)", "Running module scripts_user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed" ] }, "start": 12.91 }, "recoverable_errors": { "WARNING": [ "Failed to run module scripts_user (scripts in /var/lib/cloud/instance/scripts)", "Running module scripts_user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed" ] }, "stage": null, "status": "error" }

[tgquan67]

In this case I would recommend you to set VIP_NETWORK_INTERFACE to empty instead. The ubuntu 24.04 image has some changes compare to ubuntu 22.04 image where kube-vip does not recognize/set the interface altname correctly, which makes kube-vip and apiserver unable to start.

[chrischdi]

Please investigate in the other logs from cloud-init which should show better what exactly failed.

For questions to the community, feel free to reach out to Slack https://kubernetes.slack.com/archives/CKFGK3SSD or open an issue with the description.

We normally don't use github discussions. Thanks.