Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutabl #508

Closed
barnettZQG opened this issue Nov 27, 2018 · 4 comments
Closed

Comments

@barnettZQG
Copy link

I want patch deployment with:

{"metadata":{"creationTimestamp":null,"labels":{"creater_id":"1543299964865665410","version":"20181127142603"}},"spec":{"template":{"metadata":{"labels":{"version":"20181127142603","creater_id":"1543299964865665410"}},"spec":{"volumes":[{"name":"slug","hostPath":{"path":"/grdata/build/tenant/e8539a9c33fd418db11cce26d2bca431/slug/2bc58cb783992dff584ee7454360b0c1/20181127142603.tgz"}}],"containers":[{"name":"2bc58cb783992dff584ee7454360b0c1","image":"goodrain.me/runner","args":["start","web"],"ports":[{"containerPort":5000}],"env":[{"name":"LOGGER_DRIVER_NAME","value":"streamlog"},{"name":"PORT","value":"5000"},{"name":"PROTOCOL","value":"http"},{"name":"DEFAULT_DOMAIN_5000","value":"5000.gr60b0c1.0enb7gyx.37e53f.grapps.cn"},{"name":"MONITOR_PORT","value":"5000"},{"name":"CUR_NET","value":"calico"},{"name":"TENANT_ID","value":"e8539a9c33fd418db11cce26d2bca431"},{"name":"SERVICE_ID","value":"2bc58cb783992dff584ee7454360b0c1"},{"name":"MEMORY_SIZE","value":"micro"},{"name":"SERVICE_NAME","value":"gr60b0c1"},{"name":"SERVICE_POD_NUM","value":"1"},{"name":"HOST_IP","valueFrom":{"fieldRef":{"fieldPath":"status.hostIP"}}},{"name":"POD_IP","valueFrom":{"fieldRef":{"fieldPath":"status.podIP"}}}],"resources":{"limits":{"cpu":"80m","memory":"128Mi"},"requests":{"cpu":"30m","memory":"128Mi"}},"volumeMounts":[{"name":"slug","readOnly":true,"mountPath":"/tmp/slug/slug.tgz"}]}]}},"strategy":{},"selector":{"matchLabels":{"version":"20181127142603"}}}}

if I use method:

_, err := s.manager.client.AppsV1().RESTClient().Patch(types.MergePatchType).Namespace(deployment.Namespace).Resource("deployments").Name(deployment.Name).Body(app.UpgradePatch["deployment"]).Do().Get()

It will fail. The error message is as follows:

Deployment.apps "2bc58cb783992dff584ee7454360b0c1-deployment" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"version":"20181127141244", "name":"gr60b0c1"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

but if I use method:

_, err := s.manager.client.ExtensionsV1beta1().RESTClient().Patch(types.MergePatchType).Namespace(deployment.Namespace).Resource("deployments").Name(deployment.Name).Body(app.UpgradePatch["deployment"]).Do().Get()

it can be successful.

kube-api version is : v1.10.9

@liggitt
Copy link
Member

liggitt commented Nov 27, 2018

the selector in Deployment, ReplicaSet, and DaemonSet objects is immutable as of the apps/v1beta2 API. See kubernetes/kubernetes#50808

/close

@k8s-ci-robot
Copy link
Contributor

@liggitt: Closing this issue.

In response to this:

the selector in Deployment, ReplicaSet, and DaemonSet objects is immutable as of the apps/v1beta2 API. See kubernetes/kubernetes#50808

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@vishrantgupta
Copy link

This could be because of the kubernetes service and you are trying to do a rolling upgrade, once deployed the label selectors of kubernetes service can not be updated until you decide to delete the existing deployment so that the services have new label selectors to point to new pods with the updated label.

fghaas added a commit to fghaas/tutor that referenced this issue Nov 22, 2021
…n Tutor version changes

Through the commonLabels directive in kustomization.yml, all resources
get a label named "app.kubernetes.io/version", which is being set to
the Tutor version at the time of initial deployment.

When the user then subsequently progresses to a new Tutor version,
Kubernetes attempts to update this label — but for Deployment,
ReplicaSet, and DaemonSet resources, this is no longer allowed as of
kubernetes/kubernetes#50808. This causes
"tutor k8s start" (at the "kubectl apply --kustomize" step) to break
with errors such as:

Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Simply removing the app.kubernetes.io/version label from
kustomization.yml will permanently fix this issue for newly created
Kubernetes deployments, which will "survive" any future Tutor version
changes thereafter.

However, *existing* production Open edX deployments will need to throw
the affected Deployments away, and re-create them.

Also, add the Tutor version as a resource annotation instead, using
the `commonAnnotations` directive.

See also:
kubernetes/client-go#508
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/
fghaas added a commit to fghaas/tutor that referenced this issue Nov 22, 2021
…n Tutor version changes

Through the commonLabels directive in kustomization.yml, all resources
get a label named "app.kubernetes.io/version", which is being set to
the Tutor version at the time of initial deployment.

When the user then subsequently progresses to a new Tutor version,
Kubernetes attempts to update this label — but for Deployment,
ReplicaSet, and DaemonSet resources, this is no longer allowed as of
kubernetes/kubernetes#50808. This causes
"tutor k8s start" (at the "kubectl apply --kustomize" step) to break
with errors such as:

Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Simply removing the app.kubernetes.io/version label from
kustomization.yml will permanently fix this issue for newly created
Kubernetes deployments, which will "survive" any future Tutor version
changes thereafter.

However, *existing* production Open edX deployments will need to throw
the affected Deployments away, and re-create them.

Also, add the Tutor version as a resource annotation instead, using
the commonAnnotations directive.

See also:
kubernetes/client-go#508
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/

Fixes overhangio#531.
fghaas added a commit to fghaas/tutor that referenced this issue Nov 23, 2021
…n Tutor version changes

Through the commonLabels directive in kustomization.yml, all resources
get a label named "app.kubernetes.io/version", which is being set to
the Tutor version at the time of initial deployment.

When the user then subsequently progresses to a new Tutor version,
Kubernetes attempts to update this label — but for Deployment,
ReplicaSet, and DaemonSet resources, this is no longer allowed as of
kubernetes/kubernetes#50808. This causes
"tutor k8s start" (at the "kubectl apply --kustomize" step) to break
with errors such as:

Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Simply removing the app.kubernetes.io/version label from
kustomization.yml will permanently fix this issue for newly created
Kubernetes deployments, which will "survive" any future Tutor version
changes thereafter.

However, *existing* production Open edX deployments will need to throw
the affected Deployments away, and re-create them.

Also, add the Tutor version as a resource annotation instead, using
the commonAnnotations directive.

See also:
kubernetes/client-go#508
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/

Fixes overhangio#531.
regisb pushed a commit to overhangio/tutor that referenced this issue Nov 25, 2021
…n Tutor version changes

Through the commonLabels directive in kustomization.yml, all resources
get a label named "app.kubernetes.io/version", which is being set to
the Tutor version at the time of initial deployment.

When the user then subsequently progresses to a new Tutor version,
Kubernetes attempts to update this label — but for Deployment,
ReplicaSet, and DaemonSet resources, this is no longer allowed as of
kubernetes/kubernetes#50808. This causes
"tutor k8s start" (at the "kubectl apply --kustomize" step) to break
with errors such as:

Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Simply removing the app.kubernetes.io/version label from
kustomization.yml will permanently fix this issue for newly created
Kubernetes deployments, which will "survive" any future Tutor version
changes thereafter.

However, *existing* production Open edX deployments will need to throw
the affected Deployments away, and re-create them.

Also, add the Tutor version as a resource annotation instead, using
the commonAnnotations directive.

See also:
kubernetes/client-go#508
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/

Fixes #531.
regisb pushed a commit to overhangio/tutor that referenced this issue Dec 20, 2021
…n Tutor version changes

Through the commonLabels directive in kustomization.yml, all resources
get a label named "app.kubernetes.io/version", which is being set to
the Tutor version at the time of initial deployment.

When the user then subsequently progresses to a new Tutor version,
Kubernetes attempts to update this label — but for Deployment,
ReplicaSet, and DaemonSet resources, this is no longer allowed as of
kubernetes/kubernetes#50808. This causes
"tutor k8s start" (at the "kubectl apply --kustomize" step) to break
with errors such as:

Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Simply removing the app.kubernetes.io/version label from
kustomization.yml will permanently fix this issue for newly created
Kubernetes deployments, which will "survive" any future Tutor version
changes thereafter.

However, *existing* production Open edX deployments will need to throw
the affected Deployments away, and re-create them.

Also, add the Tutor version as a resource annotation instead, using
the commonAnnotations directive.

See also:
kubernetes/client-go#508
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/

Fixes #531.
regisb pushed a commit to overhangio/tutor that referenced this issue Dec 20, 2021
…n Tutor version changes

Through the commonLabels directive in kustomization.yml, all resources
get a label named "app.kubernetes.io/version", which is being set to
the Tutor version at the time of initial deployment.

When the user then subsequently progresses to a new Tutor version,
Kubernetes attempts to update this label — but for Deployment,
ReplicaSet, and DaemonSet resources, this is no longer allowed as of
kubernetes/kubernetes#50808. This causes
"tutor k8s start" (at the "kubectl apply --kustomize" step) to break
with errors such as:

Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Simply removing the app.kubernetes.io/version label from
kustomization.yml will permanently fix this issue for newly created
Kubernetes deployments, which will "survive" any future Tutor version
changes thereafter.

However, *existing* production Open edX deployments will need to throw
the affected Deployments away, and re-create them.

Also, add the Tutor version as a resource annotation instead, using
the commonAnnotations directive.

See also:
kubernetes/client-go#508
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/

Fixes #531.
@blaskovicz
Copy link

blaskovicz commented Mar 16, 2024

This could be because of the kubernetes service and you are trying to do a rolling upgrade, once deployed the label selectors of kubernetes service can not be updated until you decide to delete the existing deployment so that the services have new label selectors to point to new pods with the updated label.

To clarify for others that find this post later, check that your spec.selector.matchLabels doesn't have labels that change each deployment like app.kubernetes.io, pod-hash, etc. If so, delete the Deployment and make a more generic matchLabels.

Thanks @vishrantgupta

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants