How to upgrade kubefed v0.3.0 to v0.6.0

[swiftslee]

/kind support
Kubefed v0.3.0 was installed by helm2, now v0.6.0 is installed by helm3. According to the READEME, it's recommended to use helm 2to3, but the command didn't work and I got errors(kubefed-admission-webhook pod) as follows:
E1229 06:20:55.428412 1 controller.go:267] Could not update status fields of the CRD: "kube-federation-system/namespaces": Internal error occurred: failed calling webhook "federatedtypeconfigs.core.kubefed.io": Post https://kubefed-admission-webhook.kube-federation-system.svc:443/apis/validation.core.kubefed.io/v1beta1/federatedtypeconfigs?timeout=30s: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefed-admission-webhook-ca")
It seems that there are some problems with tls. The value of the kubefed-admission-webhook-serving-cert secret didn't change. Why would I get the error? Thanks, guys.

[k8s-ci-robot]

@yuswift: The label(s) triage/support cannot be applied, because the repository doesn't have them

In response to this:

/triage support
Kubefed v0.3.0 was installed by helm2, now v0.6.0 is installed by helm3. According to the READEME, it's recommended to use helm 2to3, but the command didn't work and I got errors(kubefed-admission-webhook pod) as follows:
E1229 06:20:55.428412 1 controller.go:267] Could not update status fields of the CRD: "kube-federation-system/namespaces": Internal error occurred: failed calling webhook "federatedtypeconfigs.core.kubefed.io": Post https://kubefed-admission-webhook.kube-federation-system.svc:443/apis/validation.core.kubefed.io/v1beta1/federatedtypeconfigs?timeout=30s: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefed-admission-webhook-ca")
It seems that there are some problems with tls. The value of the kubefed-admission-webhook-serving-cert secret didn't change. Why would I get the error? Thanks, guys.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[swiftslee]

After I executed helm rollback -n kube-federation-system kubefed 1, the error disappeared. I guess the secret was changed by helm but the upgrade was failed, so I got the error.

[swiftslee]

I tried the upgrade again using the following command:
helm --namespace kube-federation-system upgrade -i kubefed kubefed-charts/kubefed --version=v0.6.0
Then I got an error:
Error: UPGRADE FAILED: cannot patch "kubefed" with kind KubeFedConfig: admission webhook "kubefedconfigs.core.kubefed.io" denied the request: spec.featureGates.name: Unsupported value: "RawResourceStatusCollection": supported values: "PushReconciler", "SchedulerPreferences", "CrossClusterServiceDiscovery", "FederatedIngress"
I found that the image of the deployment has been replaced, which means that I have upgraded the kubefed.
Then I executed the command again:
helm --namespace kube-federation-system upgrade -i kubefed kubefed-charts/kubefed --version=v0.6.0
I got a message:
Release "kubefed" has been upgraded. Happy Helming!.
Is there a way of upgrading kubefed only using helm upgrade command once?

[naikn]

Facing the same problem upgrading from 0.4.0 to 0.6.0. I did workaround the problem to perform helm upgrade in one shot without rollback, but by updating the webhook image to 0.6.0 and then upgrading to 0.6.0.
kubectl set image deploy kubefed-admission-webhook -n kube-federation-system admission-webhook=quay.io/kubernetes-multicluster/kubefed:v0.6.0
helm3 --namespace kube-federation-system upgrade -i kubefed kubefed-charts/kubefed --version=0.6.0 --create-namespace

[cebernardi]

Following, same problem

[hectorj2f]

I'll test it myself and investigate the issue. But RawResourceStatusCollection feature flag was added in v0.6.0 so it is unclear to me why you are facing this issue.

[hectorj2f]

I believe we need to add a test to validate the upgrade path in the future.

[naikn]

I did try to enable the feature flag, while upgrading.

[cebernardi]

But it's happening even without enabling the feature flag (always upgrade path)

[swiftslee]

But it's happening even without enabling the feature flag (always upgrade path)
Try to downgrade the kubefed version.

[cebernardi]

But it's happening even without enabling the feature flag (always upgrade path)
Try to downgrade the kubefed version.

Hi Yuswift, thank you for your reply.

What I did was:
Upgrade from 0.4 to 0.6 (fail)
Upgrade from 0.4 to 0.5 (success), and from 0.5 to 0.6 (fail)

That said, could you please expand a bit on your suggestion? To which version are you suggesting to downgrade?
Thank you in advance!

I might try to add more context here:
we are using --atomic flag to prevent partial or incomplete upgrades, so we're not in the situation where some of the components are upgraded and some others are not..

[swiftslee]

But it's happening even without enabling the feature flag (always upgrade path)
Try to downgrade the kubefed version.

Hi Yuswift, thank you for your reply.

What I did was:
Upgrade from 0.4 to 0.6 (fail)
Upgrade from 0.4 to 0.5 (success), and from 0.5 to 0.6 (fail)

That said, could you please expand a bit on your suggestion? To which version are you suggesting to downgrade?
Thank you in advance!

I might try to add more context here:
we are using --atomic flag to prevent partial or incomplete upgrades, so we're not in the situation where some of the components are upgraded and some others are not..

The reason I upgraded to v0.6.0 was to try out the remote status collection feature. I just downgraded to the original v0.3.0 version I used. If you are not interested in the features after v0.4.0, I suggest you downgrading to v0.4.0 for safety. For more features, please take a look at the release page.

[hectorj2f]

I am gonna work on this issue regarding the upgrading path today. I'll share whatever I discover or the PR with the bugfixes.

[cebernardi]

The reason I upgraded to v0.6.0 was to try out the remote status collection feature. I just downgraded to the original v0.3.0 version I used. If you are not interested in the features after v0.4.0, I suggest you downgrading to v0.4.0 for safety. For more features, please take a look at the release page.

Oh, I see.. I also need the new feature, of course.
I was just sharing some more data I collected trying to troubleshoot the issue.

[naikn]

@hectorj2f I looked at the changes webhook component had between 0.4.0 and 0.6.0, and there not many changes, so I upgraded the webhook deployment to v0.6.0 version before upgrading the helm chart to 0.6.0. I just wanted to check with you if it would cause any issue, something I may have missed.
Initially when I had faced the issue, I let helm upgrade fail for the first attempt and then upgraded again to success. But that does not help as the controller does not pick up the kubefedconfig changes and does not enable the feature gate correctly. But after I restarted the controller deployments (rolling restart), it picks up the changes. Then the raw remote status feature starts to work. But since I wanted to install without rollback, so I upgraded the webhook image id first, then the helm charts.

[hectorj2f]

Alright! I managed to reproduce the error mentioned in the thread. So I got a working upgrade from v0.5.0 to v0.6.0 after adding some helm hooks to the resources.
After that, I did a full upgrade path from v0.3.0 till v0.6.0 without issues. I will create a PR with the fix in the following hours.

Note that I am using all the time Helm v3.