Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Logging multi-line klog output on cluster-autoscaler when scaling failed #7702

Open
krrrr38 opened this issue Jan 16, 2025 · 0 comments · May be fixed by #7703
Open

Logging multi-line klog output on cluster-autoscaler when scaling failed #7702

krrrr38 opened this issue Jan 16, 2025 · 0 comments · May be fixed by #7703
Labels
area/cluster-autoscaler kind/bug Categorizes issue or PR as related to a bug.

Comments

@krrrr38
Copy link

krrrr38 commented Jan 16, 2025 •
edited
Loading

Which component are you using?:

/area cluster-autoscaler

What version of the component are you using?:

Component version: v1.31.1

What k8s version are you using (kubectl version)?:

kubectl version Output
$ kubectl version
Client Version: v1.30.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.31.4-eks-2d5f260

What environment is this in?:

AWS EKS

What did you expect to happen?:

cluster-autoscaler always logs single-line klog output.

What happened instead?:

auto_scaling_groups.go:551 log contains multi line logging in klog.

I0116 07:10:42.719737       1 auto_scaling_groups.go:508] Instance group eks-batch-spot202407091400-ap-northeast-1a-72c85d24-0d0c-3317-6d99-xxxxxxxxxxxx has only 0 instances created while requested count is 1. Creating placeholder instances.
W0116 07:10:42.815402       1 auto_scaling_groups.go:551] ASG eks-batch-spot202407091400-ap-northeast-1a-72c85d24-0d0c-3317-6d99-xxxxxxxxxxxx scaling failed with {
  ActivityId: "4b6651b7-c98a-a0c3-5fe1-xxxxxxxxxxxx",
  AutoScalingGroupARN: "arn:aws:autoscaling:ap-northeast-1:xxxxxxxxxxxx:autoScalingGroup:15be2ddc-7564-4bf2-bb35-xxxxxxxxxxxx:autoScalingGroupName/eks-batch-spot202407091400-ap-northeast-1a-72c85d24-0d0c-3317-6d99-xxxxxxxxxxxx",
  AutoScalingGroupName: "eks-batch-spot202407091400-ap-northeast-1a-72c85d24-0d0c-3317-6d99-xxxxxxxxxxxx",
  Cause: "At 2025-01-16T07:09:48Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 0 to 1.",
  Description: "Launching a new EC2 instance.  Status Reason: Could not launch Spot Instances. UnfulfillableCapacity - Unable to fulfill capacity due to your request configuration. Please adjust your request and try again. Launching EC2 instance failed.",
  Details: "{\"Subnet ID\":\"subnet-xxxxxxxxxxxx\",\"Availability Zone\":\"ap-northeast-1a\"}",
  EndTime: 2025-01-16 07:09:49 +0000 UTC,
  Progress: 100,
  StartTime: 2025-01-16 07:09:49.096 +0000 UTC,
  StatusCode: "Failed",
  StatusMessage: "Could not launch Spot Instances. UnfulfillableCapacity - Unable to fulfill capacity due to your request configuration. Please adjust your request and try again. Launching EC2 instance failed."
}
W0116 07:10:42.815520       1 auto_scaling_groups.go:516] Instance group eks-batch-spot202407091400-ap-northeast-1a-72c85d24-0d0c-3317-6d99-xxxxxxxxxxxx cannot provision any more nodes!
I0116 07:10:43.011817       1 auto_scaling_groups.go:161] Updated ASG cache for eks-daemon-spot-202407091400-ap-northeast-1a-56c85d24-0d05-8da9-1e78-xxxxxxxxxxxx. min/max/current is 1/450/2

Then some services cannot parse this log as correct klog such as datadog and so on.

How to reproduce it (as minimally and precisely as possible):

just logging on

autoscaler/cluster-autoscaler/cloudprovider/aws/auto_scaling_groups.go

Line 551 in 0ddb3d3

klog.Warningf("ASG %s scaling failed with %s", asgRef.Name, *activity)

Anything else we need to know?:

N/A
@krrrr38 krrrr38 added the kind/bug Categorizes issue or PR as related to a bug. label Jan 16, 2025
@krrrr38 krrrr38 linked a pull request Jan 16, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/cluster-autoscaler kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix auto_scaling_group asg failed multi line warn log krrrr38/autoscaler
2 participants
@krrrr38 @k8s-ci-robot