Cluster Creation Error -- panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x128 pc=0x2bc83ea]

[paramasivam1406]

What happened: Error while creating cluster in Vsphere using admin machine with CentOS 9 and CentOS 9 Template
Error:
2024-05-31T07:18:27.826-0400 V6 Executing command {"cmd": "/usr/bin/docker version --format {{.Client.Version}}"}
2024-05-31T07:18:27.844-0400 V6 Executing command {"cmd": "/usr/bin/docker info --format '{{json .MemTotal}}'"}
2024-05-31T07:18:27.879-0400 V4 Reading release manifest {"url": "https://anywhere-assets.eks.amazonaws.com/releases/eks-a/manifest.yaml"}
2024-05-31T07:18:27.961-0400 V4 Reading bundles manifest {"url": "https://anywhere-assets.eks.amazonaws.com/releases/bundles/66/manifest.yaml"}
2024-05-31T07:18:27.979-0400 V4 Using CAPI provider versions {"Core Cluster API": "v1.6.1+b6a617d", "Kubeadm Bootstrap": "v1.6.1+33fe18b", "Kubeadm Control Plane": "v1.6.1+9da846d", "External etcd Bootstrap": "v1.0.10+c04b9b9", "External etcd Controller": "v1.0.17+60120d2", "Cluster API Provider VSphere": "v1.8.5+ba1ba79"}
2024-05-31T07:18:28.018-0400 V4 Reading release manifest {"url": "https://anywhere-assets.eks.amazonaws.com/releases/eks-a/manifest.yaml"}
2024-05-31T07:18:28.033-0400 V5 Retrier: {"timeout": "2562047h47m16.854775807s", "backoffFactor": null}
2024-05-31T07:18:28.033-0400 V2 Pulling docker image {"image": "public.ecr.aws/eks-anywhere/cli-tools:v0.19.6-eks-a-66"}
2024-05-31T07:18:28.033-0400 V6 Executing command {"cmd": "/usr/bin/docker pull public.ecr.aws/eks-anywhere/cli-tools:v0.19.6-eks-a-66"}
2024-05-31T07:18:28.310-0400 V5 Retry execution successful {"retries": 1, "duration": "277.364978ms"}
2024-05-31T07:18:28.310-0400 V3 Initializing long running container {"name": "eksa_1717154308033448772", "image": "public.ecr.aws/eks-anywhere/cli-tools:v0.19.6-eks-a-66"}
2024-05-31T07:18:28.310-0400 V6 Executing command {"cmd": "/usr/bin/docker run -d --name eksa_1717154308033448772 --network host -w /home/odh_admin_pilot -v /var/run/docker.sock:/var/run/docker.sock -v /home/odh_admin_pilot:/home/odh_admin_pilot -v /home/odh_admin_pilot:/home/odh_admin_pilot --entrypoint sleep public.ecr.aws/eks-anywhere/cli-tools:v0.19.6-eks-a-66 infinity"}
2024-05-31T07:18:28.401-0400 V0 Using the new workflow using the controller for management cluster create
2024-05-31T07:18:28.401-0400 V4 Task start {"task_name": "setup-validate"}
2024-05-31T07:18:28.401-0400 V0 Performing setup and validations
2024-05-31T07:18:28.401-0400 V4 Tasks completed {"duration": "58.094µs"}
2024-05-31T07:18:28.401-0400 V3 Logging out from current govc session
2024-05-31T07:18:28.401-0400 V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_PASSWORD=***** -e GOVC_URL=https://norwvcsa01.moderna.local/ -e GOVC_INSECURE=true -e GOVC_DATACENTER=Norwood MTC-East -e GOVC_USERNAME=***** eksa_1717154308033448772 govc session.logout"}
2024-05-31T07:18:28.904-0400 V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=https://norwvcsa01.moderna.local/ -e GOVC_INSECURE=true -e GOVC_DATACENTER=Norwood MTC-East eksa_1717154308033448772 govc session.logout -k"}
2024-05-31T07:18:29.366-0400 V3 Cleaning up long running container {"name": "eksa_1717154308033448772"}
2024-05-31T07:18:29.366-0400 V6 Executing command {"cmd": "/usr/bin/docker rm -f -v eksa_1717154308033448772"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x128 pc=0x2bc83ea]

goroutine 1 [running]:
github.com/aws/eks-anywhere/pkg/providers/vsphere.SetupEnvVars(0x0)
github.com/aws/eks-anywhere/pkg/providers/vsphere/envars.go:32 +0x14a
github.com/aws/eks-anywhere/pkg/providers/vsphere.(*vsphereProvider).SetupAndValidateCreateCluster(0xc0007c2d00, {0x3c0f9d8, 0x55d3700}, 0xc000cb3f00)
github.com/aws/eks-anywhere/pkg/providers/vsphere/vsphere.go:302 +0x49
github.com/aws/eks-anywhere/pkg/workflows/management.(*setupAndValidateCreate).Run.(*setupAndValidateCreate).providerValidation.func1()
github.com/aws/eks-anywhere/pkg/workflows/management/validate.go:37 +0xac
github.com/aws/eks-anywhere/pkg/validations.(*Runner).Run(0xc0006c89e8?)
github.com/aws/eks-anywhere/pkg/validations/runner.go:29 +0x84
github.com/aws/eks-anywhere/pkg/workflows/management.(*setupAndValidateCreate).Run(0xc000778870?, {0x3c0f9d8?, 0x55d3700}, 0xc000523e40)
github.com/aws/eks-anywhere/pkg/workflows/management/validate.go:23 +0x37c
github.com/aws/eks-anywhere/pkg/task.(*taskRunner).RunTask(0xc0006f26c0, {0x3c0f9d8, 0x55d3700}, 0xc000523e40)
github.com/aws/eks-anywhere/pkg/task/task.go:163 +0x4c3
github.com/aws/eks-anywhere/pkg/workflows/management.(*Create).Run(0xc00099bb58, {0x3c0f9d8, 0x55d3700}, 0xc000cb3f00, {0x3bd3640?, 0xc0006c89e0})
github.com/aws/eks-anywhere/pkg/workflows/management/create.go:67 +0x2a5
github.com/aws/eks-anywhere/cmd/eksctl-anywhere/cmd.(*createClusterOptions).createCluster(0x555e3a0, 0x5567b80, {0x0?, 0x0?, 0x0?})
github.com/aws/eks-anywhere/cmd/eksctl-anywhere/cmd/createcluster.go:281 +0x10c5
github.com/spf13/cobra.(*Command).execute(0x5567b80, {0xc000251bc0, 0x4, 0x4})
github.com/spf13/[email protected]/command.go:983 +0xabc
github.com/spf13/cobra.(*Command).ExecuteC(0x556d1c0)
github.com/spf13/[email protected]/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
github.com/spf13/[email protected]/command.go:1039
github.com/spf13/cobra.(*Command).ExecuteContext(...)
github.com/spf13/[email protected]/command.go:1032
github.com/aws/eks-anywhere/cmd/eksctl-anywhere/cmd.Execute()
github.com/aws/eks-anywhere/cmd/eksctl-anywhere/cmd/root.go:65 +0x4f
main.main()
github.com/aws/eks-anywhere/cmd/eksctl-anywhere/main.go:29 +0x117
What you expected to happen:

Expected to create a cluster

How to reproduce it (as minimally and precisely as possible):

Created Admin machine with CentOS 9 with recommended Configs

Anything else we need to know?:

Environment: vSphere

• EKS Anywhere Release:0.19.6
• EKS Distro Release:1.29
• Template: tried with CentOS 9 Template and Bottle rocket

[abhay-krishna]

Hello @paramasivam1406, thank you for opening the issue.

From the stack trace, the offending line of code in SetupEnvVars seems to be

if err := os.Setenv(vSphereServerKey, datacenterConfig.Spec.Server); err != nil {

Are you setting the server field in the VSphereDatacenterConfig section of your cluster config?

[saiteja313]

It looks like OS CentOS is not supported. Please try with Ubuntu or Redhat.

Mac OS 10.15+ / Ubuntu 20.04.2 LTS or 22.04 LTS / RHEL or Rocky Linux 8.8+
https://anywhere.eks.amazonaws.com/docs/getting-started/install/

[abhay-krishna]

@paramasivam1406 was this issue solved when you switched to Ubuntu or RedHat?

[paramasivam1406]

@abhay-krishna That specific issue was resolved once we changed the OS to Ubuntu, we are running into some other issues while creating cluster, we are following up with the AWS enterprise support for resolutions.
Thanks the the response and sorry for the delay.