Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: update vendor, ignore some errors in eks delete #191

Merged
merged 11 commits into from
Dec 7, 2020
Merged

Conversation

gyuho
Copy link
Contributor

@gyuho gyuho commented Dec 4, 2020

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@natherz97
Copy link
Contributor

Thanks for doing this Gyuho! I ran this against an aws-k8s-tester cluster that failed to delete its vpc and role stacks and it successfully cleaned up those two stacks. I checked out the config from S3 on a different machine than what executed the original test.

./aws-k8s-tester-v20201207.180242-linux-amd64 eks delete cluster -p aws-k8s-tester-eks.config.yaml
...
*********************************
deleteClusterRole ("/home/natherz/aws-k8s-tester-eks.config.yaml")
{"level":"info","ts":"2020-12-07T18:16:30.831Z","caller":"cluster/role.go:354","msg":"deleting role CFN stack","cluster-role-cfn-stack-id":"arn:aws:cloudformation:us-west-2:617930562442:stack/vertical-autoscaling-up-1331508242177593347-role/1f4a7290-2ef5-11eb-8459-02310ad85abb"}
{"level":"info","ts":"2020-12-07T18:16:30.948Z","caller":"cfn/cfn.go:43","msg":"polling stack","stack-id":"arn:aws:cloudformation:us-west-2:617930562442:stack/vertical-autoscaling-up-1331508242177593347-role/1f4a7290-2ef5-11eb-8459-02310ad85abb","want":"DELETE_COMPLETE","initial-wait":"25s","poll-interval":"10s","ctx-time-left":"9m59.999257509s"}
{"level":"info","ts":"2020-12-07T18:16:30.973Z","caller":"cfn/cfn.go:119","msg":"poll","name":"vertical-autoscaling-up-1331508242177593347-role","desired":"DELETE_COMPLETE","current":"DELETE_IN_PROGRESS","current-reason":"User Initiated","started":"now","ctx-time-left":"9m59.974724299s"}
{"level":"info","ts":"2020-12-07T18:16:30.973Z","caller":"cfn/cfn.go:164","msg":"sleeping","initial-wait":"25s"}


🏊 🚣 ⛵ 🌒   ⚓ Waiting for CFN stack DELETE_COMPLETE

{"level":"info","ts":"2020-12-07T18:17:06.011Z","caller":"cfn/cfn.go:119","msg":"poll","name":"vertical-autoscaling-up-1331508242177593347-role","desired":"DELETE_COMPLETE","current":"DELETE_COMPLETE","current-reason":"","started":"35 seconds ago","ctx-time-left":"9m24.936393209s"}
{"level":"info","ts":"2020-12-07T18:17:06.011Z","caller":"cfn/cfn.go:158","msg":"desired stack status; done","current-stack-status":"DELETE_COMPLETE"}
{"level":"info","ts":"2020-12-07T18:17:06.011Z","caller":"cluster/role.go:385","msg":"deleted a role","cluster-role-cfn-stack-id":"arn:aws:cloudformation:us-west-2:617930562442:stack/vertical-autoscaling-up-1331508242177593347-role/1f4a7290-2ef5-11eb-8459-02310ad85abb","cluster-role-arn":"arn:aws:iam::617930562442:role/vertical-autoscaling-up-1331508242177593347-role","cluster-role-name":"vertical-autoscaling-up-1331508242177593347-role"}


*********************************
deleteVPC ("/home/natherz/aws-k8s-tester-eks.config.yaml")
{"level":"info","ts":"2020-12-07T18:17:06.017Z","caller":"cluster/vpc.go:967","msg":"sleeping before VPC deletion","wait":"30s"}
{"level":"info","ts":"2020-12-07T18:17:36.017Z","caller":"cluster/vpc.go:970","msg":"deleting VPC","vpc-id":"vpc-067ca7bf96803721f","vpc-cfn-stack-id":"arn:aws:cloudformation:us-west-2:617930562442:stack/vertical-autoscaling-up-1331508242177593347-vpc/34715580-2ef5-11eb-9293-06491fb55b7d"}
{"level":"info","ts":"2020-12-07T18:17:36.017Z","caller":"cluster/vpc.go:1061","msg":"deleting ELBv2 for the VPC","vpc-id":"vpc-067ca7bf96803721f"}
{"level":"info","ts":"2020-12-07T18:17:36.072Z","caller":"cluster/vpc.go:1085","msg":"found ELBv2 for other VPCs","vpc-id":"vpc-00fdcea7e239c23ee","elb-arn":"arn:aws:elasticloadbalancing:us-west-2:617930562442:loadbalancer/app/310a5049-default-deck-f5c5/02b54dfab3473983"}
{"level":"info","ts":"2020-12-07T18:17:36.072Z","caller":"cluster/vpc.go:1085","msg":"found ELBv2 for other VPCs","vpc-id":"vpc-0aa66db65b54ae383","elb-arn":"arn:aws:elasticloadbalancing:us-west-2:617930562442:loadbalancer/app/dd3c4763-default-deck-f5c5/e13c9bf78ddd609f"}
{"level":"info","ts":"2020-12-07T18:17:36.414Z","caller":"cfn/cfn.go:43","msg":"polling stack","stack-id":"arn:aws:cloudformation:us-west-2:617930562442:stack/vertical-autoscaling-up-1331508242177593347-vpc/34715580-2ef5-11eb-9293-06491fb55b7d","want":"DELETE_COMPLETE","initial-wait":"2m0s","poll-interval":"20s","ctx-time-left":"9m59.999215707s"}
{"level":"info","ts":"2020-12-07T18:17:36.446Z","caller":"cfn/cfn.go:119","msg":"poll","name":"vertical-autoscaling-up-1331508242177593347-vpc","desired":"DELETE_COMPLETE","current":"DELETE_IN_PROGRESS","current-reason":"User Initiated","started":"now","ctx-time-left":"9m59.967297322s"}
{"level":"info","ts":"2020-12-07T18:17:36.446Z","caller":"cfn/cfn.go:164","msg":"sleeping","initial-wait":"2m0s"}


🏊 🚣 ⛵ 🌘   ⚓ Waiting for CFN stack DELETE_COMPLETE

{"level":"info","ts":"2020-12-07T18:19:56.480Z","caller":"cfn/cfn.go:119","msg":"poll","name":"vertical-autoscaling-up-1331508242177593347-vpc","desired":"DELETE_COMPLETE","current":"DELETE_COMPLETE","current-reason":"","started":"2 minutes ago","ctx-time-left":"7m39.933013848s"}
{"level":"info","ts":"2020-12-07T18:19:56.480Z","caller":"cfn/cfn.go:158","msg":"desired stack status; done","current-stack-status":"DELETE_COMPLETE"}
{"level":"info","ts":"2020-12-07T18:19:56.481Z","caller":"cluster/vpc.go:1053","msg":"deleted a VPC","vpc-cfn-stack-id":"arn:aws:cloudformation:us-west-2:617930562442:stack/vertical-autoscaling-up-1331508242177593347-vpc/34715580-2ef5-11eb-9293-06491fb55b7d","vpc-id":"vpc-067ca7bf96803721f"}
{"level":"warn","ts":"2020-12-07T18:19:56.487Z","caller":"eks/eks.go:1521","msg":"failed clusterTester.Delete","error":"failed to write RemoteAccessCommandsOutputPath \"/build/test.ssh.sh\" (open /build/test.ssh.sh: no such file or directory), failed to write RemoteAccessCommandsOutputPath \"/build/test.ssh.sh\" (open /build/test.ssh.sh: no such file or directory)"}

*********************************
aws-k8s-tester eks delete cluster FAIL (failed to write RemoteAccessCommandsOutputPath "/build/test.ssh.sh" (open /build/test.ssh.sh: no such file or directory), failed to write RemoteAccessCommandsOutputPath "/build/test.ssh.sh" (open /build/test.ssh.sh: no such file or directory), failed to write RemoteAccessCommandsOutputPath "/build/test.ssh.sh" (open /build/test.ssh.sh: no such file or directory), failed to write RemoteAccessCommandsOutputPath "/build/test.ssh.sh" (open /build/test.ssh.sh: no such file or directory), failed to write RemoteAccessCommandsOutputPath "/build/test.ssh.sh" (open /build/test.ssh.sh: no such file or directory), failed to write RemoteAccessCommandsOutputPath "/build/test.ssh.sh" (open /build/test.ssh.sh: no such file or directory))

It looks like the overall command fails due to the missing file even though everything is successfully cleaned up. Is it possible for the command to have a successful exit in this case?

I also ran into an issue building the Mac binaries:

a483e7119c66:aws-k8s-tester natherz$ ./hack/build.sh
GIT_COMMIT: c601508e496c
RELEASE_VERSION: v20201207.180242
BUILD_TIME: 2020-12-07_18:02:42

Usage: `make TARGET='linux' WHAT='aws-k8s-tester cw-utils'`
DEFAULT_TARGETS=linux darwin
DEFAULT_WHAT=aws-k8s-tester cw-utils ec2-utils ecr-utils eks-utils etcd-utils s3-utils sts-utils

=== Building target=aws-k8s-tester, os=linux ===
github.com/aws/aws-k8s-tester/eks
github.com/aws/aws-k8s-tester/cmd/aws-k8s-tester/eks
github.com/aws/aws-k8s-tester/cmd/aws-k8s-tester
=== Building target=cw-utils, os=linux ===
=== Building target=ec2-utils, os=linux ===
=== Building target=ecr-utils, os=linux ===
=== Building target=eks-utils, os=linux ===
=== Building target=etcd-utils, os=linux ===
=== Building target=s3-utils, os=linux ===
=== Building target=sts-utils, os=linux ===
=== Building target=aws-k8s-tester, os=darwin ===
github.com/docker/docker/pkg/term
# github.com/docker/docker/pkg/term
vendor/github.com/docker/docker/pkg/term/tc.go:13:28: undefined: unix.SYS_IOCTL
vendor/github.com/docker/docker/pkg/term/tc.go:18:28: undefined: unix.SYS_IOCTL
vendor/github.com/docker/docker/pkg/term/termios_bsd.go:24:31: undefined: unix.SYS_IOCTL
vendor/github.com/docker/docker/pkg/term/termios_bsd.go:37:31: undefined: unix.SYS_IOCTL

Is this an issue with my local environment or is something broken?

@gyuho
Copy link
Contributor Author

gyuho commented Dec 7, 2020

@natherz97 Try

TARGETS=darwin ./hack/build.sh

The build script builds both linux and darwin by default.

Never mind, getting same error.

gyuho added 4 commits December 7, 2020 11:57
ory/dockertest#212 (comment)

=== Building target=aws-k8s-tester, os=darwin ===
github.com/docker/docker/pkg/term
vendor/github.com/docker/docker/pkg/term/tc.go:13:28: undefined: unix.SYS_IOCTL
vendor/github.com/docker/docker/pkg/term/tc.go:18:28: undefined: unix.SYS_IOCTL
vendor/github.com/docker/docker/pkg/term/termios_bsd.go:24:31: undefined: unix.SYS_IOCTL
vendor/github.com/docker/docker/pkg/term/termios_bsd.go:37:31: undefined: unix.SYS_IOCTL

Signed-off-by: Gyuho Lee <[email protected]>
Signed-off-by: Gyuho Lee <[email protected]>
Signed-off-by: Gyuho Lee <[email protected]>
@natherz97
Copy link
Contributor

Ran the command again and deleting had a successful status after cleaning up leftover role and vpc stacks:

 ./aws-k8s-tester-v20201207.204822-linux-amd64 eks delete cluster -p aws-k8s-tester-eks.config.yaml
...
*********************************
DOWN DEFER START ("/home/natherz/aws-k8s-tester-eks.config.yaml")


💯 😁 👍 :) DOWN SUCCESS


{"level":"info","ts":"2020-12-07T21:36:37.434Z","caller":"eks/eks.go:1421","msg":"successfully finished Down","started":"3 minutes ago"}

Trying against a test which has a vpc stack stuck in DELETED_FAILED:

 ./aws-k8s-tester-v20201207.204822-linux-amd64 eks delete cluster -p aws-k8s-tester-eks.config.yaml
...
DOWN DEFER START ("/home/natherz/aws-k8s-tester-eks.config.yaml")
🔥 💀 👽 😱 😡 ⛈   (-_-) DOWN FAIL
{"level":"info","ts":"2020-12-07T21:41:57.226Z","caller":"eks/eks.go:1430","msg":"failed Down","error":"failed to delete stack (previous status reason \"The following resource(s) failed to delete: [VPCCIDRBlock2, VPCCIDRBlock3, VPCCIDRBlock4]. \", current stack status \"DELETE_FAILED\", current status reason \"The following resource(s) failed to delete: [VPCCIDRBlock2, VPCCIDRBlock3, VPCCIDRBlock4]. \")","started":"3 minutes ago"}



*********************************
aws-k8s-tester eks delete cluster FAIL (failed to delete stack (previous status reason "The following resource(s) failed to delete: [VPCCIDRBlock2, VPCCIDRBlock3, VPCCIDRBlock4]. ", current stack status "DELETE_FAILED", current status reason "The following resource(s) failed to delete: [VPCCIDRBlock2, VPCCIDRBlock3, VPCCIDRBlock4]. "))

@gyuho gyuho merged commit f4e4032 into aws:master Dec 7, 2020
@gyuho gyuho deleted the update branch December 7, 2020 21:45
mmerkes pushed a commit to mmerkes/aws-k8s-tester that referenced this pull request Dec 21, 2020
* *: update CHANGELOG, vendor

Signed-off-by: Gyuho Lee <[email protected]>

* eks/conformance: fmt

Signed-off-by: Gyuho Lee <[email protected]>

* eksconfig: ignore/log error in validate config file sync

Signed-off-by: Gyuho Lee <[email protected]>

* pkg/logutil: ignore writer error for log file

Signed-off-by: Gyuho Lee <[email protected]>

* CHANGELOG: update

Signed-off-by: Gyuho Lee <[email protected]>

* eks: ignore more sync errors

Signed-off-by: Gyuho Lee <[email protected]>

* eks: do no fail delete on missing file

Signed-off-by: Gyuho Lee <[email protected]>

* *: fix darwin build with docker dependency

ory/dockertest#212 (comment)

=== Building target=aws-k8s-tester, os=darwin ===
github.com/docker/docker/pkg/term
vendor/github.com/docker/docker/pkg/term/tc.go:13:28: undefined: unix.SYS_IOCTL
vendor/github.com/docker/docker/pkg/term/tc.go:18:28: undefined: unix.SYS_IOCTL
vendor/github.com/docker/docker/pkg/term/termios_bsd.go:24:31: undefined: unix.SYS_IOCTL
vendor/github.com/docker/docker/pkg/term/termios_bsd.go:37:31: undefined: unix.SYS_IOCTL

Signed-off-by: Gyuho Lee <[email protected]>

* vendor: update golang/sys/unix

Signed-off-by: Gyuho Lee <[email protected]>

* CHANGELOG: update

Signed-off-by: Gyuho Lee <[email protected]>

* eks: ignore Sync errors

Signed-off-by: Gyuho Lee <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants