Failed to Upgrade Calico v3.25.1 to v3.26.0

[kimullaa]

I upgraded from calico v3.25.1 to calico v3.26.0 as described in "Upgrading an installation that uses manifests and the Kubernetes API datastore" but it failed. calico-node's status becomes Init:CrashLoopBackOff.

calico-cni-plugin is not created when execute kubectl replace. (calico-cni-plugin was added in v3.26.0 projectcalico/calico#7106 )so I think we need to change kubectl replace to kubectl apply.https://github.com/tigera/docs/blob/main/calico/operations/upgrading/kubernetes-upgrade.mdx?plain=1#L136

# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:20:07Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:13:28Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}

# calicoctl version
Client Version:    v3.25.1
Git commit:        82dadbce1
Cluster Version:   v3.25.1
Cluster Type:      k8s,bgp,kubeadm,kdd

# kubectl replace -f 3.26.0/calico.yaml
poddisruptionbudget.policy/calico-kube-controllers replaced
serviceaccount/calico-kube-controllers replaced
serviceaccount/calico-node replaced
configmap/calico-config replaced
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org replaced
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org replaced
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org replaced
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org replaced
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org replaced
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org replaced
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org replaced
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org replaced
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org replaced
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org replaced
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org replaced
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org replaced
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org replaced
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org replaced
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org replaced
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org replaced
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org replaced
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers replaced
clusterrole.rbac.authorization.k8s.io/calico-node replaced
clusterrole.rbac.authorization.k8s.io/calico-cni-plugin replaced
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers replaced
clusterrolebinding.rbac.authorization.k8s.io/calico-node replaced
clusterrolebinding.rbac.authorization.k8s.io/calico-cni-plugin replaced
daemonset.apps/calico-node replaced
deployment.apps/calico-kube-controllers replaced
Error from server (NotFound): error when replacing "3.26.0/calico.yaml": serviceaccounts "calico-cni-plugin" not found
Error from server (NotFound): error when replacing "3.26.0/calico.yaml": customresourcedefinitions.apiextensions.k8s.io "bgpfilters.crd.projectcalico.org" not found


# kubectl get pods -n kube-system
NAME                                       READY   STATUS                  RESTARTS      AGE
calico-kube-controllers-786b679988-d9qdv   0/1     Running                 1 (7s ago)    73s
calico-node-l42zf                          0/1     Init:CrashLoopBackOff   3 (21s ago)   75s
calico-node-tjkh5                          0/1     Init:CrashLoopBackOff   2 (20s ago)   45s
coredns-5d78c9869d-l6rd2                   1/1     Running                 0             6m16s
coredns-5d78c9869d-v5jd7                   1/1     Running                 0             6m16s
etcd-k8s-master                            1/1     Running                 16            6m29s
kube-apiserver-k8s-master                  1/1     Running                 16            6m29s
kube-controller-manager-k8s-master         1/1     Running                 12            6m29s
kube-proxy-kl5gx                           1/1     Running                 0             5m43s
kube-proxy-tvxkh                           1/1     Running                 0             6m16s
kube-scheduler-k8s-master                  1/1     Running                 17            6m29s


# kubectl logs -n kube-system calico-node-l42zf -c install-cni
2023-06-07 09:45:50.047 [INFO][1] cni-installer/<nil> <nil>: Running as a Kubernetes pod
2023-06-07 09:45:50.213 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/bandwidth"
2023-06-07 09:45:50.213 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/bandwidth
2023-06-07 09:45:50.297 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/calico"
2023-06-07 09:45:50.297 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/calico
2023-06-07 09:45:50.366 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/calico-ipam"
2023-06-07 09:45:50.366 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/calico-ipam
2023-06-07 09:45:50.368 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/flannel"
2023-06-07 09:45:50.368 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/flannel
2023-06-07 09:45:50.372 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/host-local"
2023-06-07 09:45:50.372 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/host-local
2023-06-07 09:45:50.434 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/install"
2023-06-07 09:45:50.434 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/install
2023-06-07 09:45:50.437 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/loopback"
2023-06-07 09:45:50.437 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/loopback
2023-06-07 09:45:50.441 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/portmap"
2023-06-07 09:45:50.441 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/portmap
2023-06-07 09:45:50.444 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/tuning"
2023-06-07 09:45:50.444 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/tuning
2023-06-07 09:45:50.444 [INFO][1] cni-installer/<nil> <nil>: Wrote Calico CNI binaries to /host/opt/cni/bin

2023-06-07 09:45:50.461 [INFO][1] cni-installer/<nil> <nil>: CNI plugin version: v3.26.0

2023-06-07 09:45:50.461 [INFO][1] cni-installer/<nil> <nil>: /host/secondary-bin-dir is not writeable, skipping
W0607 09:45:50.461386       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2023-06-07 09:45:50.874 [ERROR][1] cni-installer/<nil> <nil>: Unable to create token for CNI kubeconfig error=serviceaccounts "calico-cni-plugin" not found
2023-06-07 09:45:50.874 [FATAL][1] cni-installer/<nil> <nil>: Unable to create token for CNI kubeconfig error=serviceaccounts "calico-cni-plugin" not found

[ctauchen]

Hi @kimullaa! Sorry to hear you're having trouble with the upgrade. Do you have a specific suggestion for how we can improve the docs here? This repo is just about documentation.

Before we can begin to address this as a documentation problem, I suggest you take your problem to the Calico Users Slack channel. If it turns out that the failure can be fixed through documentation, we'll be happy to do what we can to help.

[kimullaa]

Thanks.I posted this on Slack:https://calicousers.slack.com/archives/CPTH1KS00/p1686184117110149

[kimullaa]

@ctauchen
We discussed this in slack and it is probably a bug in the doc.(Upgrade from v3.25.1 to v3.26.0 fails when using kubectl replace)

We also found that kubectl apply --server-side --force-conflicts -f is preferable to my suggestion kubectl apply. (client side apply may not be able to handle huge manifests.)

[ctauchen]

@kimullaa Thanks for those details. I'll open a bug and we'll get this fixed.

[bmckercher123]

Fixed in this PR: #796. Closing out.