Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to parsed scheduled event start time #147

Closed
yzargari opened this issue Apr 30, 2020 · 10 comments
Closed

Unable to parsed scheduled event start time #147

yzargari opened this issue Apr 30, 2020 · 10 comments
Assignees
Labels
Type: Bug Something isn't working

Comments

@yzargari
Copy link

Hello,
I recently deployed aws-node-termination-handler in my cluster and now I have my first scheduled termination. The node hasn't drain nor cordoned and the application logs show the following message repeatedly every 2-3 seconds:

2020/04/30 12:19:25 There was a problem monitoring for Scheduled Maintenance events: Unable to parsed scheduled event start time: parsing time "8 May 2020 16:00:00 GMT" as "02 Jan 2006 15:04:05 GMT": cannot parse "8 May 2020 16:00:00 GMT" as "02"
2020/04/30 12:19:27 There was a problem monitoring for Scheduled Maintenance events: Unable to parsed scheduled event start time: parsing time "8 May 2020 16:00:00 GMT" as "02 Jan 2006 15:04:05 GMT": cannot parse "8 May 2020 16:00:00 GMT" as "02"
2020/04/30 12:19:29 There was a problem monitoring for Scheduled Maintenance events: Unable to parsed scheduled event start time: parsing time "8 May 2020 16:00:00 GMT" as "02 Jan 2006 15:04:05 GMT": cannot parse "8 May 2020 16:00:00 GMT" as "02"
2020/04/30 12:19:32 There was a problem monitoring for Scheduled Maintenance events: Unable to parsed scheduled event start time: parsing time "8 May 2020 16:00:00 GMT" as "02 Jan 2006 15:04:05 GMT": cannot parse "8 May 2020 16:00:00 GMT" as "02"
2020/04/30 12:19:34 There was a problem monitoring for Scheduled Maintenance events: Unable to parsed scheduled event start time: parsing time "8 May 2020 16:00:00 GMT" as "02 Jan 2006 15:04:05 GMT": cannot parse "8 May 2020 16:00:00 GMT" as "02"
2020/04/30 12:19:35 There was a problem monitoring for Scheduled Maintenance events: Unable to parsed scheduled event start time: parsing time "8 May 2020 16:00:00 GMT" as "02 Jan 2006 15:04:05 GMT": cannot parse "8 May 2020 16:00:00 GMT" as "02"
2020/04/30 12:19:37 There was a problem monitoring for Scheduled Maintenance events: Unable to parsed scheduled event start time: parsing time "8 May 2020 16:00:00 GMT" as "02 Jan 2006 15:04:05 GMT": cannot parse "8 May 2020 16:00:00 GMT" as "02"
2020/04/30 12:19:39 There was a problem monitoring for Scheduled Maintenance events: Unable to parsed scheduled event start time: parsing time "8 May 2020 16:00:00 GMT" as "02 Jan 2006 15:04:05 GMT": cannot parse "8 May 2020 16:00:00 GMT" as "02"
2020/04/30 12:19:42 There was a problem monitoring for Scheduled Maintenance events: Unable to parsed scheduled event start time: parsing time "8 May 2020 16:00:00 GMT" as "02 Jan 2006 15:04:05 GMT": cannot parse "8 May 2020 16:00:00 GMT" as "02"

What might the problem be?

Thanks,
Yosi

@bwagner5 bwagner5 added the Type: Bug Something isn't working label Apr 30, 2020
@bwagner5
Copy link
Contributor

I've reproduced the issue. It looks like we made a bad assumption when parsing the date and assumed the day would be padded to always be two digits. I will put a PR up to fix this bug today, sorry for the inconvenience!

@bwagner5 bwagner5 self-assigned this Apr 30, 2020
@bwagner5 bwagner5 added the Pending-Release Pending an NTH or eks-charts release label Apr 30, 2020
@bwagner5
Copy link
Contributor

PR is merged, we'll be making a release soon to fix this.

@yzargari
Copy link
Author

yzargari commented May 4, 2020

Thanks for the quick resolution!

@bwagner5
Copy link
Contributor

bwagner5 commented May 8, 2020

@bwagner5 bwagner5 closed this as completed May 8, 2020
@bwagner5 bwagner5 removed the Pending-Release Pending an NTH or eks-charts release label May 8, 2020
@diegosanchez
Copy link

Using version 1.5.0 (image amazon/aws-node-termination-handler:v1.5.0) similar issue arose again.

151x

"There was a problem monitoring for Scheduled Maintenance events: Unable to parsed scheduled event end time: parsing time \"\" as \"2 Jan 2006 15:04:05 GMT\": cannot parse \"\" as \"2\""

@bwagner5 bwagner5 reopened this Jun 25, 2020
@bwagner5
Copy link
Contributor

We'll investigate and post back, thanks for bringing this up

@bwagner5
Copy link
Contributor

Is this error from an actual scheduled maintenance event or testing through ec2-metadata-mock? I've run a small test with ec2-metadata-mock below and it seems to work correctly. Can you provide some more details on your configuration? The IMDS raw output would also be helpful, but I guess we don't log raw output on a parse error currently :(

$ curl -Lo ec2-metadata-mock https://github.com/aws/amazon-ec2-metadata-mock/releases/download/v1.0.0/ec2-metadata-mock-`uname | tr '[:upper:]' '[:lower:]'`-amd64 && chmod +x ec2-metadata-mock && mv /tmp/ec2-metadata-mock

$ git clone [email protected]:aws/aws-node-termination-handler.git /tmp/nth
$ cd /tmp/nth
$ make compile

$ /tmp/ec2-metadata-mock events &

$ /tmp/nth/build/node-termination-handler --node-name test --dry-run --enable-scheduled-event-draining=true --enable-spot-interruption-draining=false --metadata-url http://localhost:1338

STDOUT:

aws-node-termination-handler arguments:
	dry-run: true,
	node-name: test,
	metadata-url: http://localhost:1338,
	kubernetes-service-host: ,
	kubernetes-service-port: ,
	delete-local-data: true,
	ignore-daemon-sets: true,
	pod-termination-grace-period: -1,
	node-termination-grace-period: 120,
	enable-scheduled-event-draining: true,
	enable-spot-interruption-draining: false,
	metadata-tries: 3,
	cordon-only: false,
	taint-node: false,
	json-logging: false,
	webhook-proxy: ,
	enable-prometheus-server: false,
	prometheus-server-port: 9092,
2020/06/25 09:32:15 Trying to get token from IMDSv2
2020/06/25 09:32:15 Unable to retrieve an IMDSv2 token, continuing with IMDSv1: IMDS v2 Token TTL header not sent in response: No token TTL header found
2020/06/25 09:32:15 Trying to get token from IMDSv2
2020/06/25 09:32:15 Unable to retrieve an IMDSv2 token, continuing with IMDSv1: IMDS v2 Token TTL header not sent in response: No token TTL header found
2020/06/25 09:32:15 Trying to get token from IMDSv2
2020/06/25 09:32:15 Unable to retrieve an IMDSv2 token, continuing with IMDSv1: IMDS v2 Token TTL header not sent in response: No token TTL header found
2020/06/25 09:32:15 Trying to get token from IMDSv2
2020/06/25 09:32:15 Unable to retrieve an IMDSv2 token, continuing with IMDSv1: IMDS v2 Token TTL header not sent in response: No token TTL header found
2020/06/25 09:32:15 Trying to get token from IMDSv2
2020/06/25 09:32:15 Unable to retrieve an IMDSv2 token, continuing with IMDSv1: IMDS v2 Token TTL header not sent in response: No token TTL header found
2020/06/25 09:32:15 Trying to get token from IMDSv2
2020/06/25 09:32:15 Unable to retrieve an IMDSv2 token, continuing with IMDSv1: IMDS v2 Token TTL header not sent in response: No token TTL header found
2020/06/25 09:32:15 Startup Metadata Retrieved: {InstanceID:i-1234567890abcdef0 InstanceType:m4.xlarge PublicHostname:ec2-192-0-2-54.compute-1.amazonaws.com PublicIP:192.0.2.54 LocalHostname:ip-172-16-34-43.ec2.internal LocalIP:172.16.34.43}
2020/06/25 09:32:15 Started watching for interruption events
2020/06/25 09:32:15 Kubernetes AWS Node Termination Handler has started successfully!
2020/06/25 09:32:15 Started watching for event cancellations
2020/06/25 09:32:15 Started monitoring for Scheduled Maintenance events
2020/06/25 09:32:17 Trying to get token from IMDSv2
2020/06/25 09:32:17 Unable to retrieve an IMDSv2 token, continuing with IMDSv1: IMDS v2 Token TTL header not sent in response: No token TTL header found
2020/06/25 09:32:17 Sending interruption events to the interruption channel
2020/06/25 09:32:17 Got interruption event from channel {InstanceID:i-1234567890abcdef0 InstanceType:m4.xlarge PublicHostname:ec2-192-0-2-54.compute-1.amazonaws.com PublicIP:192.0.2.54 LocalHostname:ip-172-16-34-43.ec2.internal LocalIP:172.16.34.43} {EventID:instance-event-1234567890abcdef0 Kind:SCHEDULED_EVENT Description:system-reboot will occur between 25 Jun 2020 09:32:08 GMT and 2 Jul 2020 09:32:08 GMT because The instance is scheduled for system-reboot
 State:active StartTime:2020-06-25 09:32:08 +0000 UTC EndTime:2020-07-02 09:32:08 +0000 UTC Drained:false PreDrainTask:0x1cd05c0}
2020/06/25 09:32:18 Would have added label (aws-node-termination-handler/event-id=instance-event-1234567890abcdef0) to node test, but dry-run flag was set
2020/06/25 09:32:18 IsUnschedulable returning false since dry-run is set
2020/06/25 09:32:18 Would have added label (aws-node-termination-handler/action=UncordonAfterReboot) to node test, but dry-run flag was set
2020/06/25 09:32:18 Would have added label (aws-node-termination-handler/action-time=1593095538) to node test, but dry-run flag was set
2020/06/25 09:32:18 Successfully applied uncordon after reboot action label to node.
2020/06/25 09:32:18 Node test would have been cordoned and drained, but dry-run flag was set
2020/06/25 09:32:18 Node "test" successfully cordoned and drained.

@diegosanchez
Copy link

@bwagner5

I was about to query the aws metadata server (AMS) http://169.254.X.X when suddenly the instance was stopped by EKS (?). Sorry, I didn't gather the metadata informed from AMS.

@bwagner5
Copy link
Contributor

no worries, I'll dig and see if one of the scheduled maintenance events does not provide an end time.

@bwagner5 bwagner5 added the Pending-Release Pending an NTH or eks-charts release label Jul 1, 2020
@bwagner5
Copy link
Contributor

bwagner5 commented Jul 8, 2020

this bug has been fixed in the v1.6.0 release.

@bwagner5 bwagner5 closed this as completed Jul 8, 2020
@snay2 snay2 removed the Pending-Release Pending an NTH or eks-charts release label Aug 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants