Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HTTP Scaler not starting #1239

Closed
vs-dsva opened this issue Jan 24, 2025 · 3 comments
Closed

HTTP Scaler not starting #1239

vs-dsva opened this issue Jan 24, 2025 · 3 comments
Labels
bug Something isn't working

Comments

@vs-dsva
Copy link

vs-dsva commented Jan 24, 2025

Report

Immediately after installing the HTTP add-on the scaler pod enter in a restart loop.

Image

Image

Expected Behavior

Scaler pods should be up and running.

Actual Behavior

Scaler services are not starting:

2025-01-24T10:35:46Z	INFO	Scaler Version: 0.9.0
2025-01-24T10:35:46Z	INFO	Scaler Commit: 317d47406e864bac73ca56171fb7d030e95f9d42
2025-01-24T10:35:46Z	INFO	Go Version: go1.23.4
2025-01-24T10:35:46Z	INFO	Go OS/Arch: linux/amd64
2025-01-24T10:35:46Z	INFO	setup	starting the httpso informer
2025-01-24T10:35:46Z	INFO	setup	starting the grpc server
2025-01-24T10:35:46Z	INFO	starting grpc server	{"address": "0.0.0.0:9090"}
2025-01-24T10:35:46Z	INFO	setup	starting the endpoints informer
2025-01-24T10:35:46Z	INFO	setup	starting the queue pinger
2025-01-24T10:35:46Z	INFO	starting healthchecks loop
2025-01-24T10:36:17Z	ERROR	queuePinger.requestCounts	getting queue counts from interceptor	{"interceptorAddress": "http://172.16.88.134:9090", "error": "requesting the queue counts from http://172.16.88.134:9090/queue: Get \"http://172.16.88.134:9090/queue\": dial tcp 172.16.88.134:9090: i/o timeout"}
main.fetchCounts.func1
	github.com/kedacore/http-add-on/scaler/queue_pinger.go:208
golang.org/x/sync/errgroup.(*Group).Go.func1
	golang.org/x/[email protected]/errgroup/errgroup.go:78
2025-01-24T10:36:17Z	ERROR	queuePinger.requestCounts	getting queue counts from interceptor	{"interceptorAddress": "http://172.16.209.149:9090", "error": "requesting the queue counts from http://172.16.209.149:9090/queue: Get \"http://172.16.209.149:9090/queue\": dial tcp 172.16.209.149:9090: i/o timeout"}
main.fetchCounts.func1
	github.com/kedacore/http-add-on/scaler/queue_pinger.go:208
golang.org/x/sync/errgroup.(*Group).Go.func1
	golang.org/x/[email protected]/errgroup/errgroup.go:78
2025-01-24T10:36:17Z	ERROR	queuePinger.requestCounts	getting queue counts from interceptor	{"interceptorAddress": "http://172.16.18.238:9090", "error": "requesting the queue counts from http://172.16.18.238:9090/queue: Get \"http://172.16.18.238:9090/queue\": dial tcp 172.16.18.238:9090: i/o timeout"}
main.fetchCounts.func1
	github.com/kedacore/http-add-on/scaler/queue_pinger.go:208
golang.org/x/sync/errgroup.(*Group).Go.func1
	golang.org/x/[email protected]/errgroup/errgroup.go:78
2025-01-24T10:36:17Z	ERROR	queuePinger.requestCounts	fetching all counts failed	{"error": "requesting the queue counts from http://172.16.88.134:9090/queue: Get \"http://172.16.88.134:9090/queue\": dial tcp 172.16.88.134:9090: i/o timeout"}
main.fetchCounts
	github.com/kedacore/http-add-on/scaler/queue_pinger.go:237
main.(*queuePinger).fetchAndSaveCounts
	github.com/kedacore/http-add-on/scaler/queue_pinger.go:135
main.(*queuePinger).start
	github.com/kedacore/http-add-on/scaler/queue_pinger.go:106
main.main.func3
	github.com/kedacore/http-add-on/scaler/main.go:115
golang.org/x/sync/errgroup.(*Group).Go.func1
	golang.org/x/[email protected]/errgroup/errgroup.go:78
2025-01-24T10:36:17Z	ERROR	getting request counts	{"error": "requesting the queue counts from http://172.16.88.134:9090/queue: Get \"http://172.16.88.134:9090/queue\": dial tcp 172.16.88.134:9090: i/o timeout"}
main.(*queuePinger).fetchAndSaveCounts
	github.com/kedacore/http-add-on/scaler/queue_pinger.go:144
main.(*queuePinger).start
	github.com/kedacore/http-add-on/scaler/queue_pinger.go:106
main.main.func3
	github.com/kedacore/http-add-on/scaler/main.go:115
golang.org/x/sync/errgroup.(*Group).Go.func1
	golang.org/x/[email protected]/errgroup/errgroup.go:78
2025-01-24T10:36:17Z	ERROR	scaler.queuePinger.start	getting request counts	{"error": "requesting the queue counts from http://172.16.88.134:9090/queue: Get \"http://172.16.88.134:9090/queue\": dial tcp 172.16.88.134:9090: i/o timeout"}
main.(*queuePinger).start
	github.com/kedacore/http-add-on/scaler/queue_pinger.go:108
main.main.func3
	github.com/kedacore/http-add-on/scaler/main.go:115
golang.org/x/sync/errgroup.(*Group).Go.func1
	golang.org/x/[email protected]/errgroup/errgroup.go:78
2025-01-24T10:36:17Z	ERROR	scaler.queuePinger.start	context marked done. stopping queuePinger loop	{"error": "context canceled"}
main.(*queuePinger).start
	github.com/kedacore/http-add-on/scaler/queue_pinger.go:98
main.main.func3
	github.com/kedacore/http-add-on/scaler/main.go:115
golang.org/x/sync/errgroup.(*Group).Go.func1
	golang.org/x/[email protected]/errgroup/errgroup.go:78
2025-01-24T10:36:17Z	INFO	setup	Bye!

Steps to Reproduce the Problem

  1. Install KEDA 2.16.1 from kedacore helmchart in namespace pfm-keda cluster wide mode
  2. Install HTTP 0.9.0 add-on from helm chart in mamespace pfm-keda cluster wide mode
  3. Check the status of pods in namespace pfm-keda

Logs from KEDA HTTP operator

2025-01-24T12:01:54Z	ERROR	scale_handler	error getting metric spec for the scaler	{"scaledObject.Namespace": "pfm-keda", "scaledObject.Name": "keda-add-ons-http-interceptor", "scaler": "externalScaler", "error": "got empty metric spec"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScalerState
	/workspace/pkg/scaling/scale_handler.go:746
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledObjectState.func1
	/workspace/pkg/scaling/scale_handler.go:633
2025-01-24T12:01:55Z	ERROR	external_scaler	error	{"type": "ScaledObject", "namespace": "pfm-keda", "name": "keda-add-ons-http-interceptor", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 172.20.9.25:9090: connect: connection refused\""}
github.com/kedacore/keda/v2/pkg/scalers.(*externalScaler).GetMetricSpecForScaling
	/workspace/pkg/scalers/external_scaler.go:175
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetMetricSpecForScalingForScaler
	/workspace/pkg/scaling/cache/scalers_cache.go:124
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScalerState
	/workspace/pkg/scaling/scale_handler.go:743
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledObjectState.func1
	/workspace/pkg/scaling/scale_handler.go:633
2025-01-24T12:01:55Z	ERROR	external_scaler	error	{"type": "ScaledObject", "namespace": "pfm-keda", "name": "keda-add-ons-http-interceptor", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 172.20.9.25:9090: connect: connection refused\""}
github.com/kedacore/keda/v2/pkg/scalers.(*externalScaler).GetMetricSpecForScaling
	/workspace/pkg/scalers/external_scaler.go:175
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetMetricSpecForScalingForScaler
	/workspace/pkg/scaling/cache/scalers_cache.go:133
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScalerState
	/workspace/pkg/scaling/scale_handler.go:743
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledObjectState.func1
	/workspace/pkg/scaling/scale_handler.go:633
2025-01-24T12:01:55Z	ERROR	scale_handler	error getting metric spec for the scaler	{"scaledObject.Namespace": "pfm-keda", "scaledObject.Name": "keda-add-ons-http-interceptor", "scaler": "externalScaler", "error": "got empty metric spec"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScalerState
	/workspace/pkg/scaling/scale_handler.go:746
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledObjectState.func1
	/workspace/pkg/scaling/scale_handler.go:633

HTTP Add-on Version

0.9.0

Kubernetes Version

1.30

Platform

Amazon Web Services

Anything else?

No response
@vs-dsva vs-dsva added the bug Something isn't working label Jan 24, 2025
@vs-dsva
Copy link
Author

vs-dsva commented Jan 24, 2025 •
edited
Loading

Intallation log:

~ $ helm repo add kedacore https://kedacore.github.io/charts
"kedacore" already exists with the same configuration, skipping
~ $ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kedacore" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
~ $ helm install keda kedacore/keda --namespace pfm-keda --create-namespace
NAME: keda
LAST DEPLOYED: Fri Jan 24 18:26:39 2025
NAMESPACE: pfm-keda
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
:::^.     .::::^:     :::::::::::::::    .:::::::::.                   .^.
7???~   .^7????~.     7??????????????.   :?????????77!^.              .7?7.
7???~  ^7???7~.       ~!!!!!!!!!!!!!!.   :????!!!!7????7~.           .7???7.
7???~^7????~.                            :????:    :~7???7.         :7?????7.
7???7????!.           ::::::::::::.      :????:      .7???!        :7??77???7.
7????????7:           7???????????~      :????:       :????:      :???7?5????7.
7????!~????^          !77777777777^      :????:       :????:     ^???7?#P7????7.
7???~  ^????~                            :????:      :7???!     ^???7J#@J7?????7.
7???~   :7???!.                          :????:   .:~7???!.    ~???7Y&@#7777????7.
7???~    .7???7:      !!!!!!!!!!!!!!!    :????7!!77????7^     ~??775@@@GJJYJ?????7.
7???~     .!????^     7?????????????7.   :?????????7!~:      !????G@@@@@@@@5??????7:
::::.       :::::     :::::::::::::::    .::::::::..        .::::JGGGB@@@&7:::::::::
                                                                      ?@@#~
                                                                      P@B^
                                                                    :&G:
                                                                    !5.
                                                                    .Kubernetes Event-driven Autoscaling (KEDA) - Application autoscaling made simple.

Get started by deploying Scaled Objects to your cluster:
    - Information about Scaled Objects : https://keda.sh/docs/latest/concepts/
    - Samples: https://github.com/kedacore/samples

Get information about the deployed ScaledObjects:
  kubectl get scaledobject [--namespace <namespace>]

Get details about a deployed ScaledObject:
  kubectl describe scaledobject <scaled-object-name> [--namespace <namespace>]

Get information about the deployed ScaledObjects:
  kubectl get triggerauthentication [--namespace <namespace>]

Get details about a deployed ScaledObject:
  kubectl describe triggerauthentication <trigger-authentication-name> [--namespace <namespace>]

Get an overview of the Horizontal Pod Autoscalers (HPA) that KEDA is using behind the scenes:
  kubectl get hpa [--all-namespaces] [--namespace <namespace>]

Learn more about KEDA:
- Documentation: https://keda.sh/
- Support: https://keda.sh/support/
- File an issue: https://github.com/kedacore/keda/issues/new/choose
~ $ helm install http-add-on kedacore/keda-add-ons-http --namespace pfm-keda
NAME: http-add-on
LAST DEPLOYED: Fri Jan 24 18:27:25 2025
NAMESPACE: pfm-keda
STATUS: deployed
REVISION: 1
TEST SUITE: None

@dvoina
Copy link

dvoina commented Jan 25, 2025

Using a self hosted K3S I am getting the same result.

Image

│ golang.org/x/sync/errgroup.(*Group).Go.func1                                                                                                                                                                    │
│     golang.org/x/[email protected]/errgroup/errgroup.go:78                                                                                                                                                           │
│ 2025-01-25T17:18:39Z    ERROR    scaler.queuePinger.start    getting request counts    {"error": "requesting the queue counts from http://10.42.0.68:9090/queue: Get \"http://10.42.0.68:9090/queue\": dial tcp │
│ main.(*queuePinger).start                                                                                                                                                                                       │
│     github.com/kedacore/http-add-on/scaler/queue_pinger.go:108                                                                                                                                                  │
│ main.main.func3                                                                                                                                                                                                 │
│     github.com/kedacore/http-add-on/scaler/main.go:115                                                                                                                                                          │
│ golang.org/x/sync/errgroup.(*Group).Go.func1                                                                                                                                                                    │
│     golang.org/x/[email protected]/errgroup/errgroup.go:78                                                                                                                                                           │
│ 2025-01-25T17:18:39Z    ERROR    getting request counts    {"error": "pkg.k8s.EndpointsForService: client rate limiter Wait returned an error: context canceled"}                                               │
│ main.(*queuePinger).fetchAndSaveCounts                                                                                                                                                                          │
│     github.com/kedacore/http-add-on/scaler/queue_pinger.go:144                                                                                                                                                  │
│ main.(*queuePinger).start                                                                                                                                                                                       │
│     github.com/kedacore/http-add-on/scaler/queue_pinger.go:106                                                                                                                                                  │
│ main.main.func3                                                                                                                                                                                                 │
│     github.com/kedacore/http-add-on/scaler/main.go:115                                                                                                                                                          │
│ golang.org/x/sync/errgroup.(*Group).Go.func1                                                                                                                                                                    │
│     golang.org/x/[email protected]/errgroup/errgroup.go:78                                                                                                                                                           │
│ 2025-01-25T17:18:39Z    ERROR    scaler.queuePinger.start    getting request counts    {"error": "pkg.k8s.EndpointsForService: client rate limiter Wait returned an error: context canceled"}                   │
│ main.(*queuePinger).start                                                                                                                                                                                       │
│     github.com/kedacore/http-add-on/scaler/queue_pinger.go:108                                                                                                                                                  │
│ main.main.func3                                                                                                                                                                                                 │
│     github.com/kedacore/http-add-on/scaler/main.go:115                                                                                                                                                          │
│ golang.org/x/sync/errgroup.(*Group).Go.func1                                                                                                                                                                    │
│     golang.org/x/[email protected]/errgroup/errgroup.go:78                                                                                                                                                           │
│ 2025-01-25T17:18:39Z    ERROR    scaler.queuePinger.start    context marked done. stopping queuePinger loop    {"error": "context canceled"}                                                                    │
│ main.(*queuePinger).start                                                                                                                                                                                       │
│     github.com/kedacore/http-add-on/scaler/queue_pinger.go:98                                                                                                                                                   │
│ main.main.func3                                                                                                                                                                                                 │
│     github.com/kedacore/http-add-on/scaler/main.go:115                                                                                                                                                          │
│ golang.org/x/sync/errgroup.(*Group).Go.func1                                                                                                                                                                    │
│     golang.org/x/[email protected]/errgroup/errgroup.go:78                                                                                                                                                           │
│ 2025-01-25T17:18:39Z    INFO    setup    Bye!

@vs-dsva
Copy link
Author

vs-dsva commented Jan 28, 2025

The issue comes from the default NetworkPolicy. It seems that the namespace doesn't allow traffic on port 9090 so that the scaler cannot read the /queue metrics.
Setting a policy that allows port 9090 as ingress solves the issue.

@vs-dsva vs-dsva closed this as completed Jan 28, 2025
@github-project-automation github-project-automation bot moved this from To Triage to Done in Roadmap - KEDA HTTP Add-On Jan 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Roadmap - KEDA HTTP Add-On
Status: Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dvoina @vs-dsva