Fixed "NeedsUpdate" status of nodes in mixedinstancegroups after rolling update #7445
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… in the "NeedsUpdate" state even after the rolling update
|
Welcome @hippolin! |
|
Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). 📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA. It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
k8s-ci-robot
added
cncf-cla: no
|
Hi @hippolin. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
k8s-ci-robot
added
the
size/S
|
I signed it |
k8s-ci-robot
added
cncf-cla: yes
|
/assign @KashifSaadat |
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
|
Hi @zetaab / @granular-ryanbonham, could you review this PR, please? |
zetaab
requested changes
Sep 6, 2019
| request := &ec2.DescribeLaunchTemplateVersionsInput{ | ||
| LaunchTemplateName: &name, | ||
| } | ||
| versions, _ := c.EC2().DescribeLaunchTemplateVersions(request) |
There was a problem hiding this comment.
ignoring error is not good thing, maybe we should even print that?
There was a problem hiding this comment.
Thanks for the reminder. I already add print error message. Could you review again, please?
|
/retest |
|
/test pull-kops-e2e-kubernetes-aws |
k8s-ci-robot
added
the
lgtm
|
@zetaab Thank you! |
|
/retest |
|
@KashifSaadat Could you review or approve this PR, please? |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: chrisz100, hippolin The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
|
Please please, merge this PR into 1.14.0. It solves our problem, which is currently keeping us from automating cluster updates for spot instance groups. We built it locally and it worked fine for us, but working with an in-official build is kind of no go... |
|
I think the plan right now is to 100% cherry-pick this into 1.15 but since there are few tests for this area we will discuss at office hours whether to cherry pick into 1.14. |
|
If it doesn't make the initial 1.14, we can consider cutting a 1.14.1 with this, I just don't want to danger any 1.14 release. |
k8s-ci-robot
added a commit
that referenced
this pull request
Sep 21, 2019
…45-origin-release-1.15 Automated cherry pick of #7445: Fix that the node of the instance group of the mixed
|
Any plan to backport it to 1.13 (current latest release)? These commits can be applied w/o conflict and we no longer have NeedsUpdate issue with them. |
k8s-ci-robot
added a commit
that referenced
this pull request
Sep 25, 2019
…45-origin-release-1.14 Automated cherry pick of #7445: Fix that the node of the instance group of the mixed
|
This MR caused our Cluster to always role nodes. I have a MR to address this, i believe it handles all the ways you can specify the version, anyone who was seeing issues before, that this PR addressed, if you can test the PR #8038 to ensure we do not break you that would be great. |
kaldorn
pushed a commit
to getoutreach/kops
that referenced
this pull request
Jul 28, 2020
jaredallard
pushed a commit
to getoutreach/kops
that referenced
this pull request
Apr 14, 2021
Cache LaunchConfigurations On any given read operation for LCs, warm a thread-safe cache if needed. Continue to use this cache until a write operation is performed. Cache AMIs AMIs can often be the same across different ASGs. Cache on each fetch for faster lookup later. Cache autoscaling groups On any given read operation for ASGs, warm a thread-safe cache if needed. Continue to use this cache until a write operation is performed. Don't default adding MIMEBOUNDARY headers when a mixed instances policy is set Fixed "NeedsUpdate" status of nodes in mixedinstancegroups after rolling update kubernetes#7445 https://github.com/kubernetes/kops/pull/7445/files Upgrading k8s-srcdst to v.0.2.2 https://github.com/kubernetes/kops/pull/7388/files Align AWS and kops validation for spot allocation strategy https://github.com/kubernetes/kops/pull/7660/files add our calico changes calico-kube-controllers is required: https://github.com/kubernetes/kops/pull/7517/files calico-node patch: https://github.com/getoutreach/kube_factory/blob/master/patches/calico/calico-node.yaml calico-config patch: https://github.com/getoutreach/kube_factory/blob/master/patches/calico/calico-config.yaml calico-typha: https://github.com/getoutreach/kube_factory/blob/master/addons/calico/calico-typha.yaml calico-kube-controllers: https://github.com/getoutreach/kube_factory/blob/master/addons/calico/calico-kube-controllers.yaml Update aws_cloud.go Patching in capacity-optimized spot allocation strategy and updating AWS SDK Fix Handling of LaunchTemplate Versions for MixedInstancePolicy according to kubernetes#8047 Automated cherry pick of kubernetes#8261: Fix RollingUpdate behaviour when using LaunchTemplates for kubernetes#8567: Treat nil of LaunchTemplateSpecification.Version as from kubernetes#8808 Machine types update from - kubernetes#7947 A4-935 Make CircleCI build pipeline for kops fork [A4-935](https://outreach-io.atlassian.net/browse/A4-935) Adds a `.circleci/config.yml` to allow us to reproducibly build and upload assets for our fork of kops. This is used mainly to backport fixes and features into a 0.13-based branch. The management of this fork is complicated by the fact that kops configures nodes to go load the `nodeup` binary from a well known URL managed by vanilla upstream. We need to have our own S3 bucket with our own custom built binaries ready for download onto our nodes if we are to make changes to `nodeup` behavior, which is sometimes necesasry for the features we want to backport. So this CircleCI build goes through all the effort of building those assets and uploading them to S3. Tweak `Gopkg.toml` and run `make dep-ensure` Updates `Gopkg.toml` to attempt to work-around the fact that "goautoneg" no longer lives at bitbucket.org. The update process here was very finnicky. I had to make the update and delete some old generated files to get `make dep-ensure` to run to completion. Checks in the results of `make dep-ensure`. I suspect that last time there were changes to `Gopkg.toml` in [1] the changes to generated files were not fully committed and so we've partly lost the ability to build from this particular fork of kops. [1] 0984f14 Update gitignore preventing checkin of go-bindata vendor Upload to path without a `+` Upload a duplicate copy of our assets to a path that doesn't include a `+` sign. Although the S3 issue can be worked around by referencing the path as `%2B`, it seems `kops`, via the Go `url` package, will aggressively convert it back into a `+` and not re-encode it. The kops and Go behaviors would be fine if S3 followed the spec, but it doesn't. The easiest and safest work-around to this whole mess is to just not have any + signs in our path. Expose API Server flags needed for aws pod identities This adds the fields described in the documentation here: https://github.com/aws/amazon-eks-pod-identity-webhook/blob/master/SELF_HOSTED_SETUP.md#kubernetes-api-server-configuration Update k8s-1.12.yaml.template fix: calico Merge pull request #12 from getoutreach/fix-calico fix: calico
jaredallard
pushed a commit
to getoutreach/kops
that referenced
this pull request
Apr 14, 2021
Cache LaunchConfigurations On any given read operation for LCs, warm a thread-safe cache if needed. Continue to use this cache until a write operation is performed. Cache AMIs AMIs can often be the same across different ASGs. Cache on each fetch for faster lookup later. Cache autoscaling groups On any given read operation for ASGs, warm a thread-safe cache if needed. Continue to use this cache until a write operation is performed. Don't default adding MIMEBOUNDARY headers when a mixed instances policy is set Fixed "NeedsUpdate" status of nodes in mixedinstancegroups after rolling update kubernetes#7445 https://github.com/kubernetes/kops/pull/7445/files Upgrading k8s-srcdst to v.0.2.2 https://github.com/kubernetes/kops/pull/7388/files Align AWS and kops validation for spot allocation strategy https://github.com/kubernetes/kops/pull/7660/files add our calico changes calico-kube-controllers is required: https://github.com/kubernetes/kops/pull/7517/files calico-node patch: https://github.com/getoutreach/kube_factory/blob/master/patches/calico/calico-node.yaml calico-config patch: https://github.com/getoutreach/kube_factory/blob/master/patches/calico/calico-config.yaml calico-typha: https://github.com/getoutreach/kube_factory/blob/master/addons/calico/calico-typha.yaml calico-kube-controllers: https://github.com/getoutreach/kube_factory/blob/master/addons/calico/calico-kube-controllers.yaml Update aws_cloud.go Patching in capacity-optimized spot allocation strategy and updating AWS SDK Fix Handling of LaunchTemplate Versions for MixedInstancePolicy according to kubernetes#8047 Automated cherry pick of kubernetes#8261: Fix RollingUpdate behaviour when using LaunchTemplates for kubernetes#8567: Treat nil of LaunchTemplateSpecification.Version as from kubernetes#8808 Machine types update from - kubernetes#7947 A4-935 Make CircleCI build pipeline for kops fork [A4-935](https://outreach-io.atlassian.net/browse/A4-935) Adds a `.circleci/config.yml` to allow us to reproducibly build and upload assets for our fork of kops. This is used mainly to backport fixes and features into a 0.13-based branch. The management of this fork is complicated by the fact that kops configures nodes to go load the `nodeup` binary from a well known URL managed by vanilla upstream. We need to have our own S3 bucket with our own custom built binaries ready for download onto our nodes if we are to make changes to `nodeup` behavior, which is sometimes necesasry for the features we want to backport. So this CircleCI build goes through all the effort of building those assets and uploading them to S3. Tweak `Gopkg.toml` and run `make dep-ensure` Updates `Gopkg.toml` to attempt to work-around the fact that "goautoneg" no longer lives at bitbucket.org. The update process here was very finnicky. I had to make the update and delete some old generated files to get `make dep-ensure` to run to completion. Checks in the results of `make dep-ensure`. I suspect that last time there were changes to `Gopkg.toml` in [1] the changes to generated files were not fully committed and so we've partly lost the ability to build from this particular fork of kops. [1] 0984f14 Update gitignore preventing checkin of go-bindata vendor Upload to path without a `+` Upload a duplicate copy of our assets to a path that doesn't include a `+` sign. Although the S3 issue can be worked around by referencing the path as `%2B`, it seems `kops`, via the Go `url` package, will aggressively convert it back into a `+` and not re-encode it. The kops and Go behaviors would be fine if S3 followed the spec, but it doesn't. The easiest and safest work-around to this whole mess is to just not have any + signs in our path. Expose API Server flags needed for aws pod identities This adds the fields described in the documentation here: https://github.com/aws/amazon-eks-pod-identity-webhook/blob/master/SELF_HOSTED_SETUP.md#kubernetes-api-server-configuration Update k8s-1.12.yaml.template fix: calico Merge pull request #12 from getoutreach/fix-calico fix: calico
jaredallard
pushed a commit
to getoutreach/kops
that referenced
this pull request
Apr 19, 2021
Cache LaunchConfigurations On any given read operation for LCs, warm a thread-safe cache if needed. Continue to use this cache until a write operation is performed. Cache AMIs AMIs can often be the same across different ASGs. Cache on each fetch for faster lookup later. Cache autoscaling groups On any given read operation for ASGs, warm a thread-safe cache if needed. Continue to use this cache until a write operation is performed. Don't default adding MIMEBOUNDARY headers when a mixed instances policy is set Fixed "NeedsUpdate" status of nodes in mixedinstancegroups after rolling update kubernetes#7445 https://github.com/kubernetes/kops/pull/7445/files Upgrading k8s-srcdst to v.0.2.2 https://github.com/kubernetes/kops/pull/7388/files Align AWS and kops validation for spot allocation strategy https://github.com/kubernetes/kops/pull/7660/files add our calico changes calico-kube-controllers is required: https://github.com/kubernetes/kops/pull/7517/files calico-node patch: https://github.com/getoutreach/kube_factory/blob/master/patches/calico/calico-node.yaml calico-config patch: https://github.com/getoutreach/kube_factory/blob/master/patches/calico/calico-config.yaml calico-typha: https://github.com/getoutreach/kube_factory/blob/master/addons/calico/calico-typha.yaml calico-kube-controllers: https://github.com/getoutreach/kube_factory/blob/master/addons/calico/calico-kube-controllers.yaml Update aws_cloud.go Patching in capacity-optimized spot allocation strategy and updating AWS SDK Fix Handling of LaunchTemplate Versions for MixedInstancePolicy according to kubernetes#8047 Automated cherry pick of kubernetes#8261: Fix RollingUpdate behaviour when using LaunchTemplates for kubernetes#8567: Treat nil of LaunchTemplateSpecification.Version as from kubernetes#8808 Machine types update from - kubernetes#7947 A4-935 Make CircleCI build pipeline for kops fork [A4-935](https://outreach-io.atlassian.net/browse/A4-935) Adds a `.circleci/config.yml` to allow us to reproducibly build and upload assets for our fork of kops. This is used mainly to backport fixes and features into a 0.13-based branch. The management of this fork is complicated by the fact that kops configures nodes to go load the `nodeup` binary from a well known URL managed by vanilla upstream. We need to have our own S3 bucket with our own custom built binaries ready for download onto our nodes if we are to make changes to `nodeup` behavior, which is sometimes necesasry for the features we want to backport. So this CircleCI build goes through all the effort of building those assets and uploading them to S3. Tweak `Gopkg.toml` and run `make dep-ensure` Updates `Gopkg.toml` to attempt to work-around the fact that "goautoneg" no longer lives at bitbucket.org. The update process here was very finnicky. I had to make the update and delete some old generated files to get `make dep-ensure` to run to completion. Checks in the results of `make dep-ensure`. I suspect that last time there were changes to `Gopkg.toml` in [1] the changes to generated files were not fully committed and so we've partly lost the ability to build from this particular fork of kops. [1] 0984f14 Update gitignore preventing checkin of go-bindata vendor Upload to path without a `+` Upload a duplicate copy of our assets to a path that doesn't include a `+` sign. Although the S3 issue can be worked around by referencing the path as `%2B`, it seems `kops`, via the Go `url` package, will aggressively convert it back into a `+` and not re-encode it. The kops and Go behaviors would be fine if S3 followed the spec, but it doesn't. The easiest and safest work-around to this whole mess is to just not have any + signs in our path. Expose API Server flags needed for aws pod identities This adds the fields described in the documentation here: https://github.com/aws/amazon-eks-pod-identity-webhook/blob/master/SELF_HOSTED_SETUP.md#kubernetes-api-server-configuration Update k8s-1.12.yaml.template fix: calico Merge pull request #12 from getoutreach/fix-calico fix: calico
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixed issue #6796, that "NeedsUpdate" status of nodes in mixedinstancegroups after rolling update.